ROS trolley 3D reconstruction method and system based on lightweight deep learning

Abstract

The invention provides an ROS trolley 3D reconstruction method and system based on lightweight deep learning. The ROS trolley 3D reconstruction method comprises the steps of S1, data preprocessing, wherein original data of a sensor are received and preprocessed; the sensor original data comprises laser radar point cloud data and RGB-D camera data; s2, lightweight feature extraction: performing multi-view projection processing and back projection mapping on the preprocessed point cloud data to obtain a local feature descriptor of each point cloud; s3, point cloud registration and key frame management: searching historical key frame features nearest to the current frame features and obtaining feature point matching pairs for coarse registration and fine registration; judging whether the current frame feature is a key frame or not; s4, incremental map construction: updating a local map based on the obtained new key frame, managing the key frame by adopting a sliding window strategy, and executing closed-loop detection and global optimization; and S5, visualization and interaction. The method effectively balances the contradiction between precision and efficiency, and is strong in resource adaptability and high in engineering practicability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robotics, and in particular to a 3D reconstruction method and system for a ROS vehicle based on lightweight deep learning. Background Technology

[0002] In the fields of autonomous robot movement and environmental perception, various 3D reconstruction technologies already exist. These technologies can be mainly categorized as follows:

[0003] 1. Geometric Feature-Based SLAM (Simultaneous Localization and Mapping) Technology: This type of technology extracts geometric features (such as points, lines, and polygons) from the environment for matching and pose estimation, and gradually builds a map; representative open-source solutions include:

[0004] Cartographer: A graph-optimized SLAM library developed by Google that supports 2D and 3D LiDAR data and achieves high-precision mapping through submaps and loop closure detection.

[0005] ORB-SLAM series (ORB-SLAM, ORB-SLAM2, ORB-SLAM3): SLAM systems based on visual features (ORB features), supporting monocular, binocular, and RGB-D cameras, achieving localization and mapping through feature matching, tracking, local mapping, and global optimization.

[0006] 2. Point cloud processing-based methods: These methods directly process the point cloud data acquired by the sensor, using point cloud registration algorithms to estimate sensor motion and fusing the point cloud data to generate a 3D model; for example:

[0007] ICP algorithm: Iteratively finds the nearest corresponding points between the source point cloud and the target point cloud, and minimizes the distance error between these corresponding points, thereby calculating the relative transformation between the two frames of point clouds;

[0008] 3. Deep learning-based 3D reconstruction and SLAM methods: mainly reflected in feature extraction, scene understanding, and motion estimation; for example:

[0009] Feature extraction networks: Convolutional neural networks (CNNs) are used to extract more robust and semantically informative features from images or point clouds for subsequent matching and registration. For example, PointNet / PointNet++ can directly process unordered point cloud data.

[0010] 3D Convolutional Networks: Specifically designed to process sparse 3D data (such as point clouds), they perform well in tasks such as semantic segmentation and object detection, and are used to build more refined 3D semantic maps.

[0011] When applied to resource-constrained ROS vehicle platforms (such as embedded devices like NVIDIA Jetson Nano / Xavier NX), the above solutions generally suffer from the following drawbacks:

[0012] 1. Huge computational resource consumption:

[0013] Traditional SLAM systems such as Cartographer, while highly accurate, have a computationally intensive graph optimization process that demands significant CPU resources.

[0014] Deep learning-based methods, especially those using standard 3D convolutional networks (such as MinkowskiNet), have a large number of model parameters and high inference computation complexity, making them difficult to run efficiently on embedded hardware.

[0015] 2. Real-time performance bottleneck:

[0016] Many geometric methods or complex models take a long time to process a single frame when dealing with large-scale point cloud data, resulting in a low overall system operating frequency (e.g., below 5Hz), which cannot meet the needs of robot real-time navigation and obstacle avoidance.

[0017] Although the ICP algorithm and its variants are simple in principle, their iterative convergence speed is slow when the number of point clouds is large or the initial pose deviation is large, which affects real-time performance.

[0018] 3. The dilemma of balancing accuracy and efficiency:

[0019] Existing technical solutions often struggle to simultaneously achieve both reconstruction accuracy and operational efficiency. Solutions that prioritize high accuracy (such as complex graph optimization and dense point cloud processing) typically involve enormous computational demands; while simplifications aimed at improving efficiency (such as sparse features and reduced iteration counts) may sacrifice reconstruction detail and accuracy.

[0020] 4. The complexity of deployment and teaching applications:

[0021] Some advanced algorithms rely on specific hardware acceleration libraries (such as specific versions of CUDA and TensorRT) or complex software dependencies, which increases the difficulty of deployment and debugging on the embedded ROS platform.

[0022] Many research algorithms lack standardized interfaces, intuitive visualizations, and convenient parameter interaction functions, which hinders teaching demonstrations, algorithm understanding, and experimental verification.

[0023] In view of this, this invention proposes a 3D reconstruction method and system for ROS vehicles based on lightweight deep learning. A lightweight deep learning model is designed for point cloud feature extraction to reduce computational burden while maintaining feature robustness. The point cloud registration and map building processes are optimized to improve system real-time performance and reconstruction quality. A balance is struck between reconstruction accuracy, computational efficiency, and memory consumption, making the system more suitable for embedded environments. The system's usability, deployability, and pedagogical applicability are improved by providing modular design, visual interaction, and parameter adjustment functions. High-efficiency, high-precision real-time 3D reconstruction is achieved on the ROS vehicle platform, which is limited by computational resources and power consumption. Summary of the Invention

[0024] The purpose of this invention is to propose a ROS car 3D reconstruction method and system based on lightweight deep learning, which effectively balances the contradiction between accuracy and efficiency, has strong resource adaptability, and is highly practical for engineering.

[0025] To achieve the above objectives, the technical solution of the present invention is as follows:

[0026] A lightweight deep learning-based ROS car 3D reconstruction method includes the following steps:

[0027] S1. Data Preprocessing: Receive raw sensor data and perform preprocessing; the raw sensor data includes LiDAR point cloud data and RGB-D camera data; the preprocessing includes format standardization, filtering and noise reduction, and adaptive downsampling; the format standardization includes converting RGB-D camera data into point cloud data according to camera intrinsic parameters.

[0028] S2, Lightweight Feature Extraction: Perform multi-view projection processing on the preprocessed point cloud data, and perform multi-scale feature extraction and feature fusion on each obtained projection map; perform multi-view feature fusion based on the multi-scale fusion features of each projection map to obtain a global feature vector; perform back projection mapping based on the global feature vector and the 3D point cloud coordinates of the preprocessed point cloud data to obtain a local feature descriptor for each point cloud.

[0029] S3. Point cloud registration and keyframe management: Search for the nearest neighbor of the current frame feature in the historical keyframe feature library and obtain feature point matching pairs for coarse and fine registration of feature point matching pairs; and determine whether the current frame feature is a keyframe.

[0030] S4. Incremental map construction: Based on the newly acquired keyframes, update the local map, manage keyframes using a sliding window strategy, and perform loop closure detection and global optimization.

[0031] S5. Visualization and Interaction: Includes providing real-time visualization and parameter adjustment functions.

[0032] Preferably, the format standardization, filtering and noise reduction, and adaptive downsampling are specifically as follows:

[0033] Format standardization: For RGB-D data, the depth image is converted into a point cloud based on the camera intrinsics; and all data is uniformly converted to the robot's base coordinate system;

[0034] Filtering and noise reduction: Apply statistical filtering to remove outlier noise points;

[0035] Adaptive downsampling: Adaptive downsampling is performed using a voxel grid filter.

[0036] Preferably, the lightweight feature extraction specifically includes the following steps:

[0037] S2.1 Multi-view projection processing: For the preprocessed point cloud data, the 3D point cloud is projected onto 6 orthogonal view planes to convert the three-dimensional data into two-dimensional multi-views, and the depth information is retained as the third channel; the 6 orthogonal view planes include the front view, rear view, left view, right view, top view and bottom view;

[0038] S2.2 Single-view feature extraction network processing: Multi-scale feature extraction is performed on the projection map of each view using the MobileNetV3-Small network; the MobileNetV3-Small network includes 12 depthwise separable convolutional modules, and SE modules are embedded in the depthwise separable convolutional modules in the 1st, 6th, 9th and 12th layers;

[0039] S2.3 Multi-scale Feature Pyramid Fusion: The outputs of the separable convolutional modules at layers 4-6 of the MobileNetV3-Small network are extracted as low-level features, the outputs of the separable convolutional modules at layers 7-9 of the MobileNetV3-Small network are extracted as mid-level features, and the outputs of the separable convolutional modules at layers 10-12 of the MobileNetV3-Small network are extracted as high-level features. After upsampling to unify the features of different scales to the same size, the multi-scale features are fused using element-wise addition or concatenation operations.

[0040] F multi =F low +F mid +F high

[0041] Wherein: F multi For multi-scale fusion features; F low The low-level features represent the outputs of the depthwise separable convolutional modules (layers 4-6) after upsampling and fusion; F mid The intermediate-level feature represents the output of the depthwise separable convolutional modules at layers 7-9 after upsampling and fusion; F highThese are intermediate-level features, representing the outputs of the depthwise separable convolutional modules at layers 10-12 after upsampling and fusion.

[0042] S2.4 Multi-view feature fusion: Weighted average fusion is used to fuse multi-scale features from 6 perspectives.

[0043]

[0044] Wherein: F fused w represents the global feature vector. i This represents the fusion weight corresponding to the i-th viewpoint; Represents the multi-scale fusion feature of the i-th viewpoint;

[0045] Alternatively, connection fusion can be used to fuse multi-scale features from six perspectives:

[0046]

[0047] S2.5 Back projection mapping: Based on the coordinates of each 3D point in each projection view, the global feature vector F is... fused Each point in the preprocessed point cloud data is assigned a local feature descriptor, and the descriptor is weighted according to the visibility weight of the point from different viewpoints.

[0048] Preferably, the feature point matching pairs are obtained by searching the historical keyframe feature library for the nearest neighbor of the current frame feature, specifically as follows:

[0049] The historical keyframe feature library is organized using a KD-tree data structure;

[0050] For the current frame features, a fast nearest neighbor search is performed in the historical keyframe feature library based on the KD tree to obtain the nearest neighbor match between the current frame features and the historical keyframe features. Based on the matching results, the previous frame feature points and the historical keyframe feature points are used as feature point matching pairs. Each feature point matching pair contains the 3D coordinates of the two feature points and the local feature descriptor of each feature point.

[0051] Fuzzy matches are eliminated using a ratio test based on a preset ratio threshold.

[0052] Preferably, the coarse and fine registration of feature points are as follows:

[0053] Based on the feature point matching pairs after removing fuzzy matching, the feature points of the current frame are used as the source point cloud and the feature points of the historical key frame are used as the target point cloud. The RANSAC algorithm is used for coarse registration to obtain the rotation matrix and translation vector of coarse registration.

[0054] Based on the rough registration result, the feature-guided ICP algorithm is used for fine registration; the convergence criterion is that the translation transformation amount and rotation transformation amount between the feature points of the source point cloud and the corresponding feature points of the target point cloud after rigid body transformation are less than the preset threshold, or the preset maximum number of iterations is reached;

[0055] The objective function of the fine registration is as follows:

[0056]

[0057] where: R is the rotation matrix to be optimized, representing the rotation transformation from the source point cloud to the target point cloud; t is the translation vector to be optimized, representing the translation transformation from the source point cloud to the target point cloud; p src,n is the 3D coordinate of the nth feature point in the source point cloud; p tgt,corr,n is the 3D coordinate of the matching point corresponding to p src,n in the target point cloud; N is the total number of feature point pairs participating in the registration; ||·|| 2 is the square of the Euclidean distance; R*p src,i +t represents the new coordinate obtained by transforming the nth point of the source point cloud to the target coordinate system through the rigid body transformation (R, t).

[0058] Preferably, the judgment of whether the current frame feature is a key frame is specifically as follows:

[0059] Calculate the motion amount between the current frame and the previous key frame, including the translation distance d_trans and the rotation angle d_rot; calculate the cosine similarity sim_max between the current frame feature and the feature of the nearest neighbor frame in the historical key frame feature library;

[0060] When any of the following conditions is met, the current frame is determined to be a key frame:

[0061] d_trans>T_trans (0.1 meter to 0.5 meter)

[0062] d_rot>T_rot (5 degrees to 15 degrees)

[0063] sim_max<T_sim (0.8 to 0.9)

[0064] where: T_trans is the translation distance threshold, defined as the maximum allowable translation distance between adjacent key frames; T_rot is the rotation angle threshold, defined as the maximum allowable rotation angle between adjacent key frames; T_sim is the cosine similarity threshold.

[0065] Preferably, the specific steps of S4 are as follows:

[0066] Local map update: When a new keyframe is determined, the point cloud of the new keyframe is transformed into the world coordinate system and merged with the local map according to the rotation matrix R and translation vector t obtained by fine registration, and redundant points are removed using voxel mesh filtering.

[0067] Sliding window keyframe management: Maintain a fixed number of keyframes. When the number of keyframes exceeds the preset window size, remove the oldest keyframe and its information.

[0068] Loop closure detection: The global descriptor of the current keyframe is matched with the historical keyframe descriptor database. If a candidate loop closure frame is found with a similarity greater than the similarity threshold and a time interval exceeding the time threshold, geometric consistency verification is performed. After confirming the loop closure, the relative pose relationship between the current keyframe and the historical keyframes is added to the global pose graph. The relative pose relationship between the current keyframe and the historical keyframes is represented by the rotation matrix R and translation vector t obtained by fine registration.

[0069] Global optimization: The pose of all keyframes is globally optimized using a pose graph optimization library to correct accumulated errors and improve the accuracy of the pose. Figure 1 To the point of being responsive.

[0070] Preferably, the historical keyframe descriptor database includes global feature descriptors, local feature descriptors, geometric feature descriptors, timestamp information, and pose information;

[0071] Global descriptor: A global descriptor obtained by aggregating the features of the entire frame's point cloud;

[0072] Local feature descriptor: A local feature descriptor for each point in a keyframe;

[0073] Geometric feature descriptor: includes geometric structural information of planar features and edge features;

[0074] Timestamp information: The creation time of the keyframe, used for determining time intervals;

[0075] Pose information: The 6-DOF pose of the keyframe in the world coordinate system.

[0076] A lightweight deep learning-based ROS car 3D reconstruction system is provided. The system is implemented using any of the above-mentioned lightweight deep learning-based ROS car 3D reconstruction methods, including a sensor data preprocessing module, a deep lightweight feature extraction module, a point cloud registration and keyframe management module, an incremental map construction module, and a teaching visualization and parameter interaction module.

[0077] Preferably, the teaching visualization and parameter interaction module includes a visualization interface, a parameter adjustment interface, and an algorithm comparison experiment platform;

[0078] The visualization interface is used to include: real-time display of raw sensor point clouds, filtered and downsampled point clouds, and extracted feature points; display of keyframe poses, robot trajectories, local maps, and global map point clouds; and display of connection relationships between keyframes and loop closure detection results.

[0079] The parameter adjustment interface is used to: view the system status in real time and dynamically adjust key parameters; the system status includes CPU / GPU utilization, memory consumption, and processing frequency; the key parameters include filtering parameters, downsampling rate, key frame selection threshold, ICP parameters, and loop closure detection threshold;

[0080] The algorithm comparison experiment platform is used to: provide an algorithm switching interface to facilitate comparison of the effects of different parameter configurations or different algorithm modules; record experimental data and support offline analysis and evaluation.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] 1. Balance between real-time performance and accuracy: This invention achieves good real-time performance while maintaining high reconstruction accuracy through lightweight network design and optimized processing flow, effectively balancing the contradiction between accuracy and efficiency.

[0083] 2. Resource adaptability: The system can dynamically adjust parameters (such as point cloud downsampling rate, keyframe window size, etc.) according to hardware resources to adapt to platforms with different computing capabilities.

[0084] 3. Engineering Applicability: Provides a standard ROS interface, making it easy to integrate into existing robot systems. The system operates stably and reliably, with low hardware requirements reducing deployment costs, and a modular architecture that facilitates maintenance and upgrades. Attached Figure Description

[0085] Figure 1 This is a functional module diagram of the ROS car 3D reconstruction system based on lightweight deep learning, as described in this invention.

[0086] Figure 2 This is a schematic diagram of data processing for the deep lightweight feature extraction module of the present invention;

[0087] Figure 3 This is a schematic diagram of the MobileNetV3 network of the present invention. Detailed Implementation

[0088] The following is in conjunction with the appendix Figure 1-3 The technical solution of the present invention will be described in detail below.

[0089] This invention proposes a 3D reconstruction method for a ROS-based car using lightweight deep learning, specifically including the following steps:

[0090] S1. Data Preprocessing: Receive raw sensor data and perform preprocessing; the raw sensor data includes LiDAR point cloud data and RGB-D camera data; the preprocessing includes format standardization, filtering and noise reduction, and adaptive downsampling; the format standardization includes converting RGB-D camera data into point cloud data according to camera intrinsic parameters.

[0091] S2, Lightweight Feature Extraction: Perform multi-view projection processing on the preprocessed point cloud data, and perform multi-scale feature extraction and feature fusion on each obtained projection map; perform multi-view feature fusion based on the multi-scale fusion features of each projection map to obtain a global feature vector; perform back projection mapping based on the global feature vector and the 3D point cloud coordinates of the preprocessed point cloud data to obtain a local feature descriptor for each point cloud.

[0092] S3. Point cloud registration and keyframe management: Search for the nearest neighbor of the current frame feature in the historical keyframe feature library and obtain feature point matching pairs for coarse and fine registration of feature point matching pairs; and determine whether the current frame feature is a keyframe.

[0093] S4. Incremental map construction: Based on the newly acquired keyframes, update the local map, manage keyframes using a sliding window strategy, and perform loop closure detection and global optimization.

[0094] S5. Visualization and Interaction: Includes providing real-time visualization and parameter adjustment functions.

[0095] In this embodiment, the format standardization, filtering and noise reduction, and adaptive downsampling are specifically as follows:

[0096] Format standardization: For RGB-D data, the depth image is converted into a point cloud based on the camera intrinsics; and all data is uniformly converted to the robot's base coordinate system;

[0097] Filtering and noise reduction: Apply statistical filtering (Statistical Outlier Removal) to remove outlier noise points;

[0098] Adaptive downsampling: Adaptive downsampling is performed using a voxel grid filter.

[0099] refer to Figure 2 In this embodiment, the lightweight feature extraction specifically includes the following steps:

[0100] S2.1 Multi-view projection processing: For the preprocessed point cloud data, the 3D point cloud is projected onto 6 orthogonal view planes to generate 2D projection maps, thereby converting the three-dimensional data into two-dimensional multi-views. The projection method is orthogonal projection and the depth information is retained as the third channel. The 6 orthogonal view planes include the front view, rear view, left view, right view, top view, and bottom view.

[0101] S2.2 Single-view feature extraction network processing: The MobileNetV3-Small network is used to perform multi-scale feature extraction (2D feature extraction) on the projection map of each view.

[0102] The MobileNetV3-Small network is designed for high computational efficiency, reducing the number of network parameters by approximately 85% compared to traditional 3D convolutional networks (such as MinkowskiNet). The MobileNetV3-Small network decomposes standard convolution into depthwise convolution and pointwise convolution, significantly reducing computation and parameter count while maintaining feature extraction capabilities. In addition, it introduces a Squeeze-and-Excitation attention module to enhance the representation of key features.

[0103] refer to Figure 3 The core technical components involved are as follows:

[0104] a) Depthwise separable convolution (12 layers):

[0105] The standard convolution solution is: Depthwise Convolution + Pointwise Convolution.

[0106] Comparison of computational complexity:

[0107] Standard convolution: O(C in ×K×K×C out ×H×W)

[0108] Depthwise separable convolution: O(K×K×C) in ×H×W+C in ×C out ×H×W)

[0109] Where K is the kernel size, C in For the number of input channels, C out H represents the number of output channels, and H and W represent the height and width of the feature map.

[0110] b) SE attention mechanism:

[0111] SE modules are embedded in the depth-separable convolutional modules at layers 1, 6, 9, and 12.

[0112] Mathematical expression: F se=F in ⊙σ(FC2(ReLU(FC1(GAP(F in )))))

[0113] Symbol explanation:

[0114] F in Input feature map, with shape (C, H, W)

[0115] GAP: Global Average Pooling Operation

[0116] FC1: The first fully connected layer, performing dimensionality reduction.

[0117] FC2: The second fully connected layer, for dimensionality enhancement.

[0118] σ: Sigmoid activation function

[0119] ⊙: Channel-level element-wise multiplication

[0120] The SE module compression ratio is set between 4 and 16;

[0121] S2.3 Multi-scale Feature Pyramid Fusion: The outputs of convolutional layers 4-6 of the MobileNetV3-Small network are extracted as low-level features, the outputs of convolutional layers 7-9 are extracted as mid-level features, and the outputs of convolutional layers 10-12 are extracted as high-level features. After upsampling to unify features of different scales to the same size, multi-scale features are fused using element-wise addition or concatenation operations.

[0122] F multi =F low +F mid +F high

[0123] Wherein: F multi For multi-scale fusion features; F low The low-level features represent the outputs of the depthwise separable convolutional modules (layers 4-6) after upsampling and fusion; F mid The intermediate-level feature represents the output of the depthwise separable convolutional modules at layers 7-9 after upsampling and fusion; F high These are intermediate-level features, representing the outputs of the depthwise separable convolutional modules at layers 10-12 after upsampling and fusion.

[0124] By fusing features from different levels of the MobileNetV3-Small network through a multi-scale feature pyramid, contextual information at different scales is captured, and feature fusion is achieved through upsampling and element-wise addition / concatenation operations.

[0125] S2.4 Multi-view feature fusion: Weighted average fusion is used to fuse multi-scale features from 6 perspectives.

[0126]

[0127] Wherein: F fused w represents the global feature vector. i The weight w represents the fusion weight corresponding to the i-th viewpoint. i It can be learned through training or manually set according to the importance of the perspective; Represents the multi-scale fusion feature of the i-th viewpoint;

[0128] Alternatively, connection fusion can be used to fuse multi-scale features from six perspectives:

[0129]

[0130] S2.5 Back projection mapping: Based on the coordinates of each 3D point in each projection view, the global feature vector F is... fused Each point in the preprocessed point cloud data is assigned a local feature descriptor, and the allocation is weighted according to the visibility weight of the point from different viewpoints.

[0131] The weighted allocation based on the visibility weight of a point in different viewpoints is as follows: when the same 3D point is visible in multiple projection views, the weight coefficient is determined according to its position in each viewpoint (high weight when it is in the central region and unobstructed, low weight when it is in the edge region or partially obstructed, and zero weight when it is not visible). Then, the feature vectors of the point in each viewpoint are weighted and fused according to the visibility weight to obtain the final local feature descriptor of the 3D point, so as to improve the accuracy and robustness of point cloud feature mapping.

[0132] In this embodiment, the search for the nearest neighbor of the current frame feature in the historical keyframe feature library and the acquisition of feature point matching pairs are as follows:

[0133] The historical keyframe feature library is organized using data structures such as KD-trees;

[0134] For the current frame features, a fast nearest neighbor search is performed in the historical keyframe feature library based on the KD tree to obtain the nearest neighbor match between the current frame features and the historical keyframe features. Based on the matching results, the previous frame feature points and the historical keyframe feature points are used as feature point matching pairs. Each feature point matching pair contains the 3D coordinates of the two feature points and the local feature descriptor of each feature point.

[0135] Fuzzy matches are eliminated using a ratio test based on a preset ratio threshold. Optionally, the ratio threshold can be set between 0.7 and 0.9.

[0136] In this embodiment, the coarse registration and fine registration of feature points are specifically as follows:

[0137] Based on the feature point matching pairs after removing fuzzy matching, the feature points of the current frame are used as the source point cloud and the feature points of the historical key frame are used as the target point cloud. The RANSAC algorithm is used for coarse registration, the initial rigid body transformation is estimated, and the rotation matrix and translation vector of coarse registration are obtained.

[0138] Optionally, the number of RANSAC iterations is set to 100, and the interior point distance threshold is set to between 0.03 meters and 0.1 meters;

[0139] Based on the coarse registration results, the feature-guided ICP algorithm is used for fine registration; the convergence criterion is that the translation and rotation transformation amounts between the source point cloud feature points and the corresponding feature points of the target point cloud after rigid body transformation are less than the preset thresholds (e.g., translation less than 0.5mm to 5mm, rotation less than 0.05 degrees to 0.5 degrees), or the preset maximum number of iterations is reached.

[0140] The objective function for fine registration is as follows:

[0141]

[0142] Where: R is the rotation matrix to be optimized, representing the rotation transformation from the source point cloud to the target point cloud; t is the translation vector to be optimized, representing the translation transformation from the source point cloud to the target point cloud; p src,n p represents the 3D coordinates of the nth feature point in the source point cloud. tgt,corr,n For the target point cloud and p src,n The corresponding 3D coordinates of the matching points; N is the total number of feature point pairs participating in the registration; ||·|| 2 R*p is the square of the Euclidean distance. src,i +t represents transforming the nth point of the source point cloud to a new coordinate system in the target coordinate system through a rigid body transformation (R, t).

[0143] In this embodiment, the determination of whether the current frame feature is a keyframe specifically involves:

[0144] Calculate the motion between the current frame and the previous keyframe, including the translation distance d_trans and the rotation angle d_rot; calculate the cosine similarity sim_max between the features of the current frame and the features of the nearest neighbor frames in the historical keyframe feature library;

[0145] The current frame is determined to be a keyframe when any of the following conditions are met:

[0146] d_trans>T_trans(0.1 m to 0.5 m)

[0147] d_rot > T_rot (5 degrees to 15 degrees)

[0148] sim_max < T_sim (0.8 to 0.9)

[0149] Where: T_trans is the translation distance threshold, defined as the maximum allowable translation distance between adjacent key frames; T_rot is the rotation angle threshold, defined as the maximum allowable rotation angle between adjacent key frames; T_sim is the cosine similarity threshold.

[0150] In this embodiment, the S4 specifically includes:

[0151] Local map update: When a new key frame is determined, the new key frame point cloud is transformed to the world coordinate system according to the rotation matrix R and translation vector t obtained by fine registration and merged with the local map, and voxel grid filtering is used to remove redundant points.

[0152] Sliding window key frame management: Maintain a fixed number (between 10 and 50) of key frames. When the number of key frames exceeds the preset window size, the oldest key frame and its information are removed to implement a dynamic memory pool allocation mechanism and reduce memory fragmentation.

[0153] Loop closure detection: Match the global descriptor of the current key frame with the historical key frame descriptor database. If a candidate loop closure frame with a similarity greater than the similarity threshold and a time interval exceeding the time threshold (such as more than 30 s) is found, geometric consistency verification is performed. After confirming the loop closure, the relative pose relationship between the current key frame and the historical key frame is added to the global pose graph; the relative pose relationship between the current key frame and the historical key frame is represented by the rotation matrix R and translation vector t obtained by fine registration; by introducing this loop closure constraint, the system can correct the long-term accumulated drift error during the pose graph optimization process, thereby improving the consistency and accuracy of the 3D map.

[0154] Global optimization: Use a pose graph optimization library to globally optimize the poses of all key frames, correct the accumulated error, to improve the Figure 1 consistency; the pose graph optimization library can use the open-source g2o (General Graph Optimization) library for pose graph optimization.

[0155] In this embodiment, the historical key frame descriptor database includes global feature descriptors, local feature descriptors, geometric feature descriptors, timestamp information, and pose information;

[0156] Global descriptor: A global descriptor obtained by aggregating the features of the entire frame of point cloud;

[0157] Local feature descriptor: The local feature descriptor of each point in the key frame;

[0158] Geometric feature descriptor: includes geometric structural information of planar features and edge features;

[0159] Timestamp information: The creation time of the keyframe, used for determining time intervals;

[0160] Pose information: The 6-DOF pose of the keyframe in the world coordinate system.

[0161] refer to Figure 1 A lightweight deep learning-based ROS car 3D reconstruction system is provided. The system is implemented using any of the aforementioned lightweight deep learning-based ROS car 3D reconstruction methods, including a sensor data preprocessing module, a deep lightweight feature extraction module, a point cloud registration and keyframe management module, an incremental map construction module, and a teaching visualization and parameter interaction module.

[0162] In this embodiment, the teaching visualization and parameter interaction module includes a visualization interface (RViz visualization), a parameter adjustment interface (GUI parameter adjustment interface, a graphical user interface developed based on RQT or Web), and an algorithm comparison experiment platform.

[0163] The visualization interface is used to include: real-time display of raw sensor point clouds, filtered and downsampled point clouds, and extracted feature points; display of keyframe poses, robot trajectories, local maps, and global map point clouds; and display of connection relationships between keyframes and loop closure detection results.

[0164] The parameter adjustment interface is used to: view the system status in real time and dynamically adjust key parameters; the system status includes CPU / GPU utilization, memory consumption, and processing frequency; the key parameters include filtering parameters, downsampling rate, key frame selection threshold, ICP parameters, and loop closure detection threshold;

[0165] The algorithm comparison experiment platform is used to: provide an algorithm switching interface to facilitate comparison of the effects of different parameter configurations or different algorithm modules; record experimental data and support offline analysis and evaluation.

[0166] Compared with geometric feature-based SLAM techniques (such as LeGO-LOAM), this invention:

[0167] 1. Feature extraction efficiency and robustness: This invention uses a lightweight deep learning network to extract semantic features. Compared with methods based on geometric features (such as edge points and planar points) such as LeGO-LOAM, the feature extraction is more efficient and more robust to noise and environmental changes. In scenarios with similar geometric structures but different semantics (such as long corridors and repetitive structures), the semantic features of this invention can provide more discriminative information and reduce mismatches.

[0168] 2. Computational resource consumption: This invention reduces the computational complexity of feature extraction through lightweight network design, achieving a processing frequency of 5Hz on the Jetson Nano platform, while LeGO-LOAM typically only reaches 2-3Hz on the same platform; In terms of memory usage, this invention reduces runtime memory consumption by approximately 70% compared to LeGO-LOAM through a sliding window strategy and dynamic memory management.

[0169] 3. Reconstruction accuracy: Evaluation on public indoor datasets (such as TUM RGB-D) shows that the absolute trajectory error (ATE) of the present invention is reduced by about 15% compared with LeGO-LOAM, especially in complex environments such as lighting changes and texture loss.

[0170] Compared with deep learning-based 3D reconstruction methods (such as those using SparseConvNet and MinkowskiNet), this invention...

[0171] 1. Lightweight Model: This invention uses MobileNetV3 as the backbone network, combined with depthwise separable convolution and SE attention mechanism, reducing the number of model parameters by about 85% compared to standard 3D convolutional networks (such as MinkowskiNet); through multi-view projection processing method, it avoids performing convolution operations directly in 3D space, further reducing computational complexity.

[0172] 2. Ease of deployment: The system of this invention has lower hardware requirements and can run on entry-level embedded platforms such as Jetson Nano, while standard 3D convolutional networks usually require higher-end GPU support; the software dependency is simpler, supporting optimized deployment with PyTorchMobile or TensorRT, reducing the complexity of deployment and debugging.

[0173] 3. Pedagogical Applicability: This invention provides a rich RViz visual interface and GUI parameter adjustment functions, which facilitates understanding of algorithm principles and teaching demonstrations; the modular design and algorithm comparison experimental platform support the switching and comparison of different algorithm modules, which is beneficial for teaching and research.

[0174] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A 3D reconstruction method for a ROS-based car using lightweight deep learning, characterized in that, Specifically, the following steps are included: S1. Data Preprocessing: Receive raw sensor data and perform preprocessing; the raw sensor data includes LiDAR point cloud data and RGB-D camera data; the preprocessing includes format standardization, filtering and noise reduction, and adaptive downsampling; the format standardization includes converting RGB-D camera data into point cloud data according to camera intrinsic parameters. S2, Lightweight Feature Extraction: Perform multi-view projection processing on the preprocessed point cloud data, and perform multi-scale feature extraction and feature fusion on each obtained projection map; perform multi-view feature fusion based on the multi-scale fusion features of each projection map to obtain a global feature vector; perform back projection mapping based on the global feature vector and the 3D point cloud coordinates of the preprocessed point cloud data to obtain a local feature descriptor for each point cloud. S3. Point cloud registration and keyframe management: Search for the nearest neighbor of the current frame feature in the historical keyframe feature library and obtain feature point matching pairs for coarse and fine registration of feature point matching pairs; and determine whether the current frame feature is a keyframe. S4. Incremental map construction: Based on the newly acquired keyframes, update the local map, manage keyframes using a sliding window strategy, and perform loop closure detection and global optimization. S5. Visualization and Interaction: Includes providing real-time visualization and parameter adjustment functions.

2. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 1, characterized in that, The format standardization, filtering and noise reduction, and adaptive downsampling are detailed below: Format standardization: For RGB-D data, the depth image is converted into a point cloud based on the camera intrinsics; and all data is uniformly converted to the robot's base coordinate system; Filtering and noise reduction: Apply statistical filtering to remove outlier noise points; Adaptive downsampling: Adaptive downsampling is performed using a voxel grid filter.

3. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 1, characterized in that, The lightweight feature extraction specifically includes the following steps: S2.1 Multi-view projection processing: For the preprocessed point cloud data, the 3D point cloud is projected onto 6 orthogonal view planes to convert the three-dimensional data into two-dimensional multi-views, and the depth information is retained as the third channel; the 6 orthogonal view planes include the front view, rear view, left view, right view, top view and bottom view; S2.2 Single-view feature extraction network processing: Multi-scale feature extraction is performed on the projection map of each view using the MobileNetV3-Small network; the MobileNetV3-Small network includes 12 depthwise separable convolutional modules, and SE modules are embedded in the depthwise separable convolutional modules in the 1st, 6th, 9th and 12th layers; S2.3 Multi-scale Feature Pyramid Fusion: The outputs of the separable convolutional modules at layers 4-6 of the MobileNetV3-Small network are extracted as low-level features, the outputs of the separable convolutional modules at layers 7-9 of the MobileNetV3-Small network are extracted as mid-level features, and the outputs of the separable convolutional modules at layers 10-12 of the MobileNetV3-Small network are extracted as high-level features. After upsampling to unify the features of different scales to the same size, the multi-scale features are fused using element-wise addition or concatenation operations. F multi =F low +F mid +F high Wherein: F multi For multi-scale fusion features; F low The low-level features represent the outputs of the depthwise separable convolutional modules (layers 4-6) after upsampling and fusion; F mid The intermediate-level feature represents the output of the depthwise separable convolutional modules at layers 7-9 after upsampling and fusion; F high These are intermediate-level features, representing the outputs of the depthwise separable convolutional modules at layers 10-12 after upsampling and fusion. S2.4, Multi-view Feature Fusion: The multi-scale fusion features of 6 views are fused using weighted average fusion: Wherein: F fused The vector represents the global feature vector; w represents the fusion weight corresponding to the i-th viewpoint. Represents the multi-scale fusion feature of the i-th viewpoint; Or the multi-scale fusion features of 6 views are fused using concatenation fusion: S2.5 Back projection mapping: Based on the coordinates of each 3D point in each projection view, the global feature vector F is... fused Each point in the preprocessed point cloud data is assigned a local feature descriptor, and the descriptor is weighted according to the visibility weight of the point from different viewpoints.

4. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 1, characterized in that, Search for the historical key frame features that are the nearest neighbors to the current frame features in the historical key frame feature library and obtain feature point matching pairs. Specifically: Organize the historical key frame feature library using the KD tree data structure; For the current frame features, perform a fast nearest neighbor search in the historical key frame feature library based on the KD tree to obtain the nearest neighbor match between the current frame features and the historical key frame features, and use the previous frame feature points and the historical key frame feature points as feature point matching pairs according to the matching results; each feature point matching pair contains the 3D coordinates of two feature points and the local feature descriptors of each feature point; the coordinates of the historical key frame feature points are based on the world coordinate system; Eliminate ambiguous matches using ratio testing according to a preset ratio threshold.

5. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 4, characterized in that, Coarse registration and fine registration of feature points. Specifically: Based on the feature point matching pairs after eliminating ambiguous matches, use the current frame feature points as the source point cloud and the historical key frame feature points as the target point cloud, and use the RANSAC algorithm for coarse registration to obtain the rotation matrix and translation vector of the coarse registration; Perform fine registration using the feature-guided ICP algorithm based on the coarse registration result; the convergence criterion is that the translation transformation amount and rotation transformation amount between the source point cloud feature points and the corresponding target point cloud feature points after rigid body transformation are less than a preset threshold, or the preset maximum number of iterations is reached; The objective function of the fine registration is as follows: Where: R is the rotation matrix to be optimized, representing the rotation transformation from the source point cloud to the target point cloud; t is the translation vector to be optimized, representing the translation transformation from the source point cloud to the target point cloud; p src,n p represents the 3D coordinates of the nth feature point in the source point cloud. tgt,corr,n For the target point cloud and p src,n The corresponding 3D coordinates of the matching points; N is the total number of feature point pairs participating in the registration; ||·|| 2 R*p is the square of the Euclidean distance. src,i +t represents transforming the nth point of the source point cloud to a new coordinate system in the target coordinate system through a rigid body transformation (R, t).

6. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 1, characterized in that, The judgment on whether the current frame feature is a key frame is specifically: Calculate the motion amount between the current frame and the previous key frame, including the translation distance d_trans and the rotation angle d_rot; calculate the cosine similarity sim_max between the current frame features and the nearest neighbor frame features in the historical key frame feature library; When any of the following conditions is met, determine that the current frame is a key frame: d_trans > T_trans (0.1 meter to 0.5 meter) d_rot > T_rot (5 degrees to 15 degrees) sim_max < T_sim (0.8 to 0.9) Where: T_trans is the translation distance threshold, defined as the maximum allowable translation distance between adjacent key frames; T_rot is the rotation angle threshold, defined as the maximum allowable rotation angle between adjacent key frames; T_sim is the cosine similarity threshold.

7. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 1, characterized in that, The specific content of S4 includes: Local map update: When a new key frame is determined, transform the new key frame point cloud to the world coordinate system and merge it with the local map according to the rotation matrix R and translation vector t obtained by fine registration, and use voxel grid filtering to remove redundant points; Sliding window key frame management: Maintain a fixed number of key frames. When the number of key frames exceeds the preset window size, remove the oldest key frame and its information; Loop closure detection: The global descriptor of the current keyframe is matched with the historical keyframe descriptor database. If a candidate loop closure frame is found with a similarity greater than the similarity threshold and a time interval exceeding the time threshold, geometric consistency verification is performed. After confirming the loop closure, the relative pose relationship between the current keyframe and the historical keyframes is added to the global pose graph. The relative pose relationship between the current keyframe and the historical keyframes is represented by the rotation matrix R and translation vector t obtained by fine registration. Global optimization: The pose of all keyframes is globally optimized using a pose graph optimization library to correct accumulated errors and improve map consistency.

8. The ROS car 3D reconstruction method based on lightweight deep learning according to claim 7, characterized in that, The historical keyframe descriptor database includes global feature descriptors, local feature descriptors, geometric feature descriptors, timestamp information, and pose information; Global descriptor: A global descriptor obtained by aggregating the features of the entire frame's point cloud; Local feature descriptor: A local feature descriptor for each point in a keyframe; Geometric feature descriptor: includes geometric structural information of planar features and edge features; Timestamp information: The creation time of the keyframe, used for determining time intervals; Pose information: The 6-DOF pose of the keyframe in the world coordinate system.

9. A ROS-based 3D reconstruction system for a small car using lightweight deep learning, characterized in that, The system is implemented using the ROS car 3D reconstruction method based on lightweight deep learning as described in any one of claims 1-8, including a sensor data preprocessing module, a deep lightweight feature extraction module, a point cloud registration and keyframe management module, an incremental map construction module, and a teaching visualization and parameter interaction module.

10. The ROS car 3D reconstruction system based on lightweight deep learning according to claim 9, characterized in that, The teaching visualization and parameter interaction module includes a visualization interface, a parameter adjustment interface, and an algorithm comparison experiment platform. The visualization interface is used to include: real-time display of raw sensor point clouds, filtered and downsampled point clouds, and extracted feature points; display of keyframe poses, robot trajectories, local maps, and global map point clouds; and display of connection relationships between keyframes and loop closure detection results. The parameter adjustment interface is used to: view the system status in real time and dynamically adjust key parameters; the system status includes CPU / GPU utilization, memory consumption, and processing frequency; the key parameters include filtering parameters, downsampling rate, key frame selection threshold, ICP parameters, and loop closure detection threshold; The algorithm comparison experiment platform is used to: provide an algorithm switching interface to facilitate comparison of the effects of different parameter configurations or different algorithm modules; record experimental data and support offline analysis and evaluation.

Images