Self-localization and motion perception method and system based on deep learning

Abstract

The invention relates to a self-localization and motion perception method based on deep learning, and the method comprises the following steps: multi-modal perception: collecting RGB frames and event streams through employing a DAVIS346 event camera, obtaining texture and depth information through employing an RGB-D camera, and supplementing 3D structure data through a laser radar; hybrid optical flow driven motion perception: adopting a double-branch architecture of an improved RAFT network and an event optical flow private network; multi-modal 3D detection and trajectory management: fusing multi-modal data based on VoxelNeXt-Lite to generate a 3D detection frame, filtering false detection by combining point cloud density clustering and an event density threshold, then constructing a space-time diagram associated trajectory through a Spatio-Temporal Graph Transformer, and optimizing pedestrian trajectory prediction continuity by using an LSTM (Long Short Term Memory) model perceived by a gait cycle; multi-view semantic graph matching: constructing a 3D semantic voxel map containing vertical features, calculating scene similarity in combination with top view NetVLAD features and deformable graph matching, and dynamically adjusting semantic weight by using a situation encoder and knowledge graph reasoning; and carrying out multi-sensor fusion and robust positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of indoor service robots, and in particular to a method and system for autonomous positioning and motion perception based on deep learning. Background Art

[0002] In the field of indoor service robots, with increasing demands for autonomous navigation accuracy and adaptability to dynamic environments in scenarios such as smart logistics, medical delivery, and commercial navigation, autonomous positioning and motion perception technologies based on deep learning have become core research directions. Deep learning, through the hierarchical abstraction of visual features using convolutional neural networks (CNNs), the modeling of temporal dependencies using recurrent neural networks (RNNs), and the reasoning of spatial relationships using graph neural networks (GNNs), provides theoretical support for the fusion of heterogeneous data from multimodal sensors. This enables robots to simultaneously leverage texture information from RGB frames, dynamic edge features from event streams, 3D structural priors from point clouds, and contextual constraints from semantic labels to construct cross-modal spatiotemporal representations. This allows them to maintain robust positioning even in scenarios with dynamic occlusion, sensor degradation (such as RGB overexposure caused by strong light), or feature ambiguity (such as mismatches caused by repeated textures). Furthermore, end-to-end training enables continuous optimization of motion prediction, forming a closed-loop perception system from low-level sensor calibration to high-level semantic understanding. This significantly enhances the robot's autonomous service capabilities in highly dynamic, weakly textured environments with strong human-robot interaction.

[0003] Conventional tracking algorithms rely solely on short-term motion models and are unable to effectively model the target's long-term motion patterns. If the target's appearance is obscured, appearance-based matching algorithms fail, and trajectory loss can cause the robot to misjudge the target's position, potentially leading to incorrect obstacle avoidance or path replanning, reducing mission efficiency. Therefore, the proposed autonomous positioning and motion perception methods and systems based on deep learning are particularly important. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the prior art and provide an autonomous positioning and motion perception method and system based on deep learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The autonomous positioning and motion perception method based on deep learning includes the following steps:

[0007] Multimodal perception: The DAVIS346 event camera is used to capture RGB frames and event streams, while an RGB-D camera is used to obtain texture and depth information. LiDAR is used to supplement 3D structural data. Hand-eye calibration is used to jointly calibrate the spatial transformation matrices of the event camera and RGB-D camera. A spatiotemporal convolutional network (TCN) is used to interpolate the event stream onto the RGB frame timeline for spatiotemporal alignment. The Spike-YOLO network is combined with the Spike-YOLO network to extract motion edge features from the asynchronous event stream and generate a dynamic attention map to mark areas of high change.

[0008] Hybrid optical flow-driven motion perception: This approach employs a dual-branch architecture combining an improved RAFT network and an event-based optical flow network (E-FlowNet). A dynamic mask prediction head is used to mask blurry areas, a self-attention mechanism is used to fuse cross-modal optical flows, and a physical constraint loss function is embedded to suppress non-physical motion. The output optical flow field (including the dynamic mask) and the physical constraint loss term are fed back into multi-sensor fusion as a constraint on visual reprojection errors. The mask information is then used by multimodal 3D detection and trajectory management to filter out false detections of dynamic objects in the 3D detection box. Furthermore, the optical flow residual analysis results of E-FlowNet are fed into semantic graph matching to assist in weighting dynamic regions in the vertical semantic descriptor.

[0009] Multimodal 3D detection and trajectory management: VoxelNeXt-Lite is used to fuse multimodal data to generate 3D detection boxes. Point cloud density clustering (DBSCAN 3D) and event density thresholding are combined to filter false detections. The Spatio-Temporal Graph Transformer (STGT) is then used to construct spatiotemporal graph-associated trajectories. A gait-cycle-aware LSTM model is used to optimize pedestrian trajectory prediction continuity. The generated 3D detection boxes and trajectory IDs are used to construct a spatiotemporal graph using STGT. The trajectory continuity prediction results (such as gait cycle features) are input into a factor graph optimizer for multi-sensor fusion and robust localization as a motion model prior. Furthermore, the semantic labels of the detection boxes (such as "pedestrian" and "cart") are combined with the context-aware weight matrix in multi-view semantic graph matching to dynamically adjust the confidence level of the object in the semantic map.

[0010] Multi-view semantic graph matching: Builds a 3D semantic voxel map containing vertical features such as wall height distribution and ceiling structure. Extracts vertical semantic descriptors using Vertical Net. Combines top-view NetVLAD features with Deformable Graph Matching (DGM) to calculate scene similarity, enabling cross-floor loop closure detection. Dynamically adjusts semantic weights using a context encoder and knowledge graph reasoning. Fine-tuning with the Adapter module avoids global retraining. The constructed 3D semantic voxel map and vertical descriptors generate loop closure candidates through Deformable Graph Matching (DGM), triggering global pose graph optimization for multi-sensor fusion and robust localization.

[0011] Multi-sensor fusion and robust positioning: Integrate visual reprojection error, event stream consistency loss, and 3D detection box geometric constraints in tightly coupled factor graph optimization, and achieve dynamic covariance adjustment through LSTM online estimation of the IMU noise model.

[0012] The above scheme further includes:

[0013] Furthermore, the method uses a DAVIS346 event camera to collect RGB frames and event streams, uses an RGB-D camera to obtain texture and depth information, supplements 3D structure data through a lidar, jointly calibrates the spatial transformation matrix of the event camera and the RGB-D camera through hand-eye calibration, and uses a spatiotemporal convolutional network (TCN) to interpolate the event stream to the RGB frame time axis for spatiotemporal alignment, including the following steps:

[0014] Sensor configuration and data acquisition: Deploy a DAVIS346 event camera, an RGB-D camera, and a LiDAR. The DAVIS346 event camera collects RGB frames and synchronously outputs an asynchronous event stream. The RGB-D camera collects RGB images and depth maps. The LiDAR scans and generates point clouds. The acquisition is triggered by a PPS (pulse-second signal). The spatial transformation matrix T_e2r, including the rotation matrix R and the translation vector t, is solved using a checkerboard calibration board combined with OpenCV.

[0015] Hand-eye calibration: Determine the transformation T_e2r=[R|t] from the event camera coordinate system {E} to the RGB-D camera coordinate system {R}, move the calibration plate to different positions, and simultaneously collect event stream and RGB-D data. Use Harris corner detection to extract corner events in the event stream and record their coordinates (x_e, y_e) in {E}. Detect the same corner in the RGB image and obtain its coordinates (x_r, y_r) in {R}. Solve by minimizing the reprojection error. Harris corner detection and minimizing the reprojection error optimize the rotation matrix R and translation vector t to complete the spatial alignment of the event camera to the RGB-D camera.

[0016] Temporal Convolutional Network (TCN) Interpolation Event Stream: We designed a temporal convolutional network (TCN) to filter asynchronous event streams by time window, calculate the difference between the number of positive and negative events ΔN(x,y), and smooth them using Gaussian kernel convolution. This generates an event intensity map I_e(x,y,t_rgb) that is temporally and spatially aligned with the RGB frame. Reprojection verification ensures that the feature overlap error is less than 2 pixels.

[0017] Construct a dataset: Fuse the lidar point cloud to supplement the 3D structure data and construct a spatiotemporal aligned dataset containing depth information.

[0018] Furthermore, the specific steps of extracting motion edge features from asynchronous event streams and generating dynamic attention maps by combining the Spike-YOLO network to mark high-variance areas are as follows:

[0019] Event stream data representation: The asynchronous events output by the DAVIS346 event camera are represented as E = {(x_k, y_k, p_k, t_k)};

[0020] Spike-YOLO network structure: uses a temporal convolutional network (TCN) to perform spatiotemporal encoding of discrete events, and constructs a structured input tensor by statistically analyzing the polarity change rate and density of pixel-level events;

[0021] Motion edge feature extraction: This approach uses a lightweight network architecture based on an improved YOLOv5-tiny. The event encoding layer converts sparse events into spatiotemporal features. The C3 module (including the Bottleneck layer) and the SPPF pooling layer are then used to extract multi-scale spatial features. Causal convolution is then used to capture temporal dependencies.

[0022] Dynamic attention map generation: A dynamic attention map is generated through an additional decoder. After sigmoid activation and thresholding, the dynamic attention map marks high-variation areas (such as the edges of pedestrian arm swings), thereby guiding the subsequent 3D detection module to focus on dynamic targets.

[0023] Furthermore, the dual-branch architecture of the improved RAFT network and the event optical flow proprietary network (E-FlowNet) is used to shield blurred areas through a dynamic mask prediction head, fuse cross-modal optical flows through a self-attention mechanism, and embed a physical constraint loss function to suppress non-physical motion.

[0024] Dynamic mask prediction head: The event stream and RGB frame are spliced ​​into a four-dimensional tensor. MobileNetV3-Small is used to fuse the cross-modal features of the event stream and RGB frame. A dynamic region mask is generated using the Sigmoid activation function, expressed as M = σ(f_CNN(X;θ)),σ(·) = 1 / (1+e^(-x)), where σ(·) is the Sigmoid activation function and θ is the network parameter. The mask is element-wise multiplied with the optical flow field output by RAFT, thereby shielding the blurred optical flow output by the RAFT network.

[0025] Event Optical Flow Proprietary Network: The Event Optical Flow Proprietary Network (E-FlowNet) encodes the event stream into a spatiotemporal voxel grid, generates a dense optical flow field through the U-Net architecture, and fuses it with the masked RAFT optical flow through the self-attention mechanism of the Swin Transformer. The fusion formula is expressed as

[0026] Physical constraint loss function: The physical constraint loss function combines the IMU angular velocity and the camera intrinsic parameter (f x ,f y ,c x ,c y ), the theoretical optical flow is inferred through the pinhole camera model, and the network is forced to output an optical flow consistent with the actual motion. The theoretical optical flow calculation formula is expressed as Where Z(u,v) is the depth map;

[0027] Optical flow residual analysis: Calculate the optical flow difference between E-FlowNet and RAFT, generate dynamic region masks through threshold processing, and use LSTM to reduce trajectory prediction confidence within the dynamic region to avoid collision risks.

[0028] The output of each step (optical flow field, dynamic mask, physical loss, residual) is used to optimize positioning accuracy through a tightly coupled factor graph, which is fed back to the 3D detection branch to filter dynamic false detections and assist semantic graph matching in suppressing cross-floor false matches. Ultimately, it achieves breakthrough performance with high trajectory ID retention and low motion prediction error in scenarios such as corridor pedestrian following and emergency stops in conference rooms.

[0029] Furthermore, the specific steps of generating 3D detection boxes based on VoxelNeXt-Lite fusion multimodal data, combining point cloud density clustering (DBSCAN 3D) with event density thresholding to filter false detections, constructing spatio-temporal graph association trajectories through Spatio-Temporal Graph Transformer (STGT), and optimizing the continuity of pedestrian trajectory prediction using a gait cycle-aware LSTM model are as follows:

[0030] Multimodal 3D detection based on VoxelNeXt-Lite: It fuses RGB images, depth maps, and event stream data, divides the space into a voxel grid (mean, variance, density), encodes statistical features of the point cloud within each voxel, and uses 3D sparse convolution to extract spatial voxel features and generate an initial 3D detection box;

[0031] Point cloud density clustering: DBSCAN 3D clustering (parameters ∈ = 0.3m, minPts = 10) was used in combination with event density threshold to filter false detections;

[0032] Spatio-Temporal Graph Transformer (STGT) trajectory association: The Spatio-Temporal Graph Transformer constructs spatio-temporal graph association trajectories, uses node and edge features to calculate attention weights, and maintains ID consistency in high-density scenarios;

[0033] Gait cycle-aware LSTM trajectory optimization: Deploy a gait cycle-aware LSTM model, input the displacement increment and gait phase φ(t) (for example, φ(t) = 0.15 corresponds to the right foot landing phase), predict the next step position, and repair broken trajectories.

[0034] Furthermore, the specific steps of the multi-view semantic graph matching are:

[0035] Construct a 3D semantic voxel map: The RGB-D camera depth map is integrated with the event camera vertical features (such as wall height distribution and ceiling structure) to construct a semantic voxel map. Each voxel stores a semantic label (such as wall, door, furniture) and geometric information (occupancy probability and normal direction). Bayesian filtering is used to update the voxel occupancy probability to accurately encode the static environment geometry-semantic information. The update formula is expressed as P(m_i|z_(1:t))=η·P(z_t|m_i)·P(m_i|z_(1:t-1)), where mi is the semantic label of voxel i, z_t is the current frame observation, and η is the normalization factor.

[0036] Extract vertical semantic descriptor: The voxel map is pooled with height direction and feature extracted by multi-layer perceptron through Vertical Net network to generate 256-dimensional vertical semantic descriptor to capture the normal direction of the wall and the topology of the ceiling. The vertical descriptor generation formula is expressed as f_vert=MLP(max h {ψ(x,y,z)}), where ψ(x,y,z) is the semantic-geometric joint feature of voxel (x,y,z), max h Represents the maximum pooling along the height direction;

[0037] Top-view semantic feature fusion: The top-view NetVLAD semantic features are fused with the vertical descriptor to form a 512-dimensional global feature vector. The cosine similarity and deformable graph matching (DGM) techniques are used to align the spatial relationships of the scene graph nodes.

[0038] Deformable Graph Matching (DGM) generates closed-loop candidates: Construct a scene graph G = (V, E), where nodes V are semantic voxel clusters and edges E are spatial relationships between adjacent clusters. Thin plate spline (TPS) deformation field estimation is used to suppress cross-floor mismatches, expressed as Δp = U·K·W·V,p′ = p + Δp, where p is the original node coordinate, U is the basis function matrix, K is the kernel matrix, W is the learnable weight, and p′ is the deformed coordinate.

[0039] Triggering global pose graph optimization: When a loop closure candidate is detected, the loop closure constraints are injected into the tightly coupled factor graph optimizer, and the iSAM2 incremental solver is used to minimize the nonlinear least squares objective function that includes visual reprojection error, IMU pre-integration constraints, and loop closure geometric consistency.

[0040] Dynamic semantic weight adjustment: Combining the dynamic semantic weight matrix generated by the context encoder with the implicit relationship of knowledge graph reasoning, the adapter module is used to fine-tune the semantic segmentation head parameters to avoid global retraining. Multi-view semantic graph matching.

[0041] Furthermore, the specific steps of integrating visual reprojection error, event flow consistency loss and 3D detection box geometric constraints in the tightly coupled factor graph optimization and realizing dynamic covariance adjustment by estimating the IMU noise model online through LSTM are as follows:

[0042] Factor graph construction: Integrating robot pose state nodes with multi-source observation factors (visual reprojection error, event stream consistency loss, 3D detection box geometric constraints, IMU pre-integration constraints, and wheel encoder odometry) to form a global optimization framework;

[0043] Visual reprojection error term: Utilizes the optical flow field F(u,v,t) predicted by the RAFT network and the depth information of the feature points, quantifies the deviation between the optical flow prediction and the actual pixel motion through projection transformation, and introduces dynamic weight covariance to suppress interference in blurred areas;

[0044] Event stream consistency loss: This combines the dynamic edges detected by Spike-YOLO with the event stream gradients to enforce motion boundary constraints through gradient alignment errors.

[0045] 3D detection box geometric constraint: Fusing the object pose output by VoxelNeXt-Lite with the robot motion prediction, and combining the semantic label dynamic penalty term to optimize the dynamic object position estimation;

[0046] Dynamic covariance adjustment (LSTM-IMU noise estimation): predict the noise covariance through the historical IMU angular velocity sequence and dynamically adjust the factor weight;

[0047] Factor graph optimization solution: The global least squares problem is solved by integrating the Huber robust loss function through the iSAM2 incremental optimizer.

[0048] The autonomous positioning and motion perception system used by the deep learning-based autonomous positioning and motion perception method includes:

[0049] Multimodal perception layer: This layer is responsible for heterogeneous data collection and preprocessing. It uses the DAVIS346 event camera to capture event streams and synchronously output RGB frames. It combines RGB-D camera and lidar data, performs spatiotemporal alignment of heterogeneous sensors using a spatiotemporal convolutional network (TCN), and then extracts motion edge features using a Spike-YOLO network to generate a dynamic attention map.

[0050] Hybrid optical flow estimation layer: Responsible for dynamic motion field analysis, it adopts a dual-branch architecture, including an improved RAFT network and E-FlowNet. The improved RAFT network uses a dynamic mask prediction head to block blurred areas. The E-FlowNet uses U-Net to process event streams and fuse cross-modal optical flow. At the same time, it embeds a physical constraint loss function to suppress non-physical motion, significantly improving the optical flow accuracy in dynamic areas.

[0051] 3D detection and trajectory management layer: This layer is responsible for understanding dynamic targets, fusing multimodal data, generating 3D detection boxes using the lightweight VoxelNeXt-Lite algorithm, filtering false detections using DBSCAN 3D clustering and event density thresholding, constructing spatio-temporal graph-associated trajectories using the Spatio-Temporal Graph Transformer, and optimizing pedestrian trajectory prediction continuity using a gait cycle-aware LSTM model.

[0052] Semantic Graph Matching Layer: This layer is responsible for environmental recognition and closed-loop optimization. It extracts vertical features such as wall height distribution to construct a 3D semantic voxel map. Vertical Net generates vertical semantic descriptors, which are combined with top-view NetVLAD features and Deformable Graph Matching (DGM) for cross-floor closed-loop detection. It also dynamically adjusts semantic weights through context encoders and knowledge graph reasoning, and uses the Adapter module for fine-tuning to avoid global retraining.

[0053] Multi-sensor fusion layer: This layer is responsible for tightly coupled positioning optimization, integrating visual reprojection error, event stream consistency loss, 3D detection box geometric constraints, and IMU pre-integration constraints. It uses LSTM to estimate the IMU noise model online to achieve dynamic covariance adjustment. It combines the iSAM2 incremental optimizer with the Huber loss function to solve the global least squares problem and reduce positioning error.

[0054] Decision-making execution layer: Responsible for autonomous navigation and interaction. Based on the optimized posture and semantic map, it uses the A* algorithm combined with dynamic obstacle prediction to generate a safe path, and dynamically adjusts the navigation strategy through speech recognition and semantic segmentation results.

[0055] The present invention has the following beneficial effects:

[0056] In this paper, the DAVIS346 event camera, RGB-D camera, lidar and hand-eye calibration are used together. The spatial transformation matrix of the event camera and RGB-D camera is jointly calibrated through hand-eye calibration. The event stream is interpolated onto the RGB frame time axis in combination with TCN to improve the accuracy of multimodal fusion. VoxelNeXt-Lite fuses multimodal data to generate 3D detection frames, and combines DBSCAN 3D clustering with event density thresholding to filter false detections. STGT uses the correlation between modeled objects through a spatiotemporal graph and combines the gait cycle characteristics (such as cadence and stride) of LSTM to achieve smooth transition and prediction of trajectories, solving the problem of insufficient memory of historical trajectories in traditional methods and improving the reliability of long-term navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a method step diagram for the deep learning-based autonomous positioning and motion perception method proposed in the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] See also Figure 1 As shown, the present invention is an autonomous positioning and motion perception method based on deep learning, comprising the following steps:

[0060] Multimodal perception: The DAVIS346 event camera is used to capture RGB frames and event streams, while an RGB-D camera is used to obtain texture and depth information. LiDAR is used to supplement 3D structural data. Hand-eye calibration is used to jointly calibrate the spatial transformation matrices of the event camera and RGB-D camera. A spatiotemporal convolutional network (TCN) is used to interpolate the event stream onto the RGB frame timeline for spatiotemporal alignment. The Spike-YOLO network is combined with the Spike-YOLO network to extract motion edge features from the asynchronous event stream and generate a dynamic attention map to mark areas of high change.

[0061] Hybrid optical flow-driven motion perception: This approach employs a dual-branch architecture combining an improved RAFT network and an event-based optical flow network (E-FlowNet). A dynamic mask prediction head is used to mask blurry areas, a self-attention mechanism is used to fuse cross-modal optical flows, and a physical constraint loss function is embedded to suppress non-physical motion. The output optical flow field (including the dynamic mask) and the physical constraint loss term are fed back into multi-sensor fusion as a constraint on visual reprojection errors. The mask information is then used by multimodal 3D detection and trajectory management to filter out false detections of dynamic objects in the 3D detection box. Furthermore, the optical flow residual analysis results of E-FlowNet are fed into semantic graph matching to assist in weighting dynamic regions in the vertical semantic descriptor.

[0062] Multimodal 3D detection and trajectory management: VoxelNeXt-Lite is used to fuse multimodal data to generate 3D detection boxes. Point cloud density clustering (DBSCAN 3D) and event density thresholding are combined to filter false detections. The Spatio-Temporal Graph Transformer (STGT) is then used to construct spatiotemporal graph-associated trajectories. A gait-cycle-aware LSTM model is used to optimize pedestrian trajectory prediction continuity. The generated 3D detection boxes and trajectory IDs are used to construct a spatiotemporal graph using STGT. The trajectory continuity prediction results (such as gait cycle features) are input into a factor graph optimizer for multi-sensor fusion and robust localization as a motion model prior. Furthermore, the semantic labels of the detection boxes (such as "pedestrian" and "cart") are combined with the context-aware weight matrix in multi-view semantic graph matching to dynamically adjust the confidence level of the object in the semantic map.

[0063] Multi-view semantic graph matching: Builds a 3D semantic voxel map containing vertical features such as wall height distribution and ceiling structure. Extracts vertical semantic descriptors using Vertical Net. Combines top-view NetVLAD features with Deformable Graph Matching (DGM) to calculate scene similarity, enabling cross-floor loop closure detection. Dynamically adjusts semantic weights using a context encoder and knowledge graph reasoning. Fine-tuning with the Adapter module avoids global retraining. The constructed 3D semantic voxel map and vertical descriptors generate loop closure candidates through Deformable Graph Matching (DGM), triggering global pose graph optimization for multi-sensor fusion and robust localization.

[0064] Multi-sensor fusion and robust positioning: Integrate visual reprojection error, event stream consistency loss, and 3D detection box geometric constraints in tightly coupled factor graph optimization, and achieve dynamic covariance adjustment through LSTM online estimation of the IMU noise model.

[0065] In one embodiment, the method uses a DAVIS346 event camera to collect RGB frames and event streams, uses an RGB-D camera to obtain texture and depth information, supplements 3D structure data with a lidar, jointly calibrates the spatial transformation matrix of the event camera and the RGB-D camera through hand-eye calibration, and uses a spatiotemporal convolutional network (TCN) to interpolate the event stream to the RGB frame time axis for spatiotemporal alignment, including the following steps:

[0066] Sensor configuration and data acquisition: Deploy a DAVIS346 event camera, an RGB-D camera, and a LiDAR. The DAVIS346 event camera collects RGB frames and synchronously outputs an asynchronous event stream. The RGB-D camera collects RGB images and depth maps. The LiDAR scans and generates point clouds. The acquisition is triggered by a PPS (pulse-second signal). The spatial transformation matrix T_e2r, including the rotation matrix R and the translation vector t, is solved using a checkerboard calibration board combined with OpenCV.

[0067] Hand-eye calibration: Determine the transformation T_e2r=[R|t] from the event camera coordinate system {E} to the RGB-D camera coordinate system {R}, move the calibration plate to different positions, and simultaneously collect event stream and RGB-D data. Use Harris corner detection to extract corner events in the event stream and record their coordinates (x_e, y_e) in {E}. Detect the same corner in the RGB image and obtain its coordinates (x_r, y_r) in {R}. Solve by minimizing the reprojection error. Harris corner detection and minimizing the reprojection error optimize the rotation matrix R and translation vector t to complete the spatial alignment of the event camera to the RGB-D camera.

[0068] Temporal Convolutional Network (TCN) Interpolation Event Stream: We designed a temporal convolutional network (TCN) to filter asynchronous event streams by time window, calculate the difference between the number of positive and negative events ΔN(x,y), and smooth them using Gaussian kernel convolution. This generates an event intensity map I_e(x,y,t_rgb) that is temporally and spatially aligned with the RGB frame. Reprojection verification ensures that the feature overlap error is less than 2 pixels.

[0069] Construct a dataset: Fuse the lidar point cloud to supplement the 3D structure data and construct a spatiotemporal aligned dataset containing depth information.

[0070] In one embodiment, the specific steps of extracting motion edge features from asynchronous event streams and generating dynamic attention maps by combining the Spike-YOLO network to mark high-variance areas are as follows:

[0071] Event stream data representation: The asynchronous events output by the DAVIS346 event camera are represented as E = {(x_k, y_k, p_k, t_k)};

[0072] Spike-YOLO network structure: uses a temporal convolutional network (TCN) to perform spatiotemporal encoding of discrete events, and constructs a structured input tensor by statistically analyzing the polarity change rate and density of pixel-level events;

[0073] Motion edge feature extraction: This approach uses a lightweight network architecture based on an improved YOLOv5-tiny. The event encoding layer converts sparse events into spatiotemporal features. The C3 module (including the Bottleneck layer) and the SPPF pooling layer are then used to extract multi-scale spatial features. Causal convolution is then used to capture temporal dependencies.

[0074] Dynamic attention map generation: A dynamic attention map is generated through an additional decoder. After sigmoid activation and thresholding, the dynamic attention map marks high-variation areas (such as the edges of pedestrian arm swings), thereby guiding the subsequent 3D detection module to focus on dynamic targets.

[0075] In one embodiment, the dual-branch architecture of the improved RAFT network and the event optical flow proprietary network (E-FlowNet) is used to shield blurry areas through a dynamic mask prediction head, fuse cross-modal optical flows through a self-attention mechanism, and embed a physical constraint loss function to suppress non-physical motion. Specific steps include:

[0076] Dynamic mask prediction head: The event stream and RGB frame are spliced ​​into a four-dimensional tensor. MobileNetV3-Small is used to fuse the cross-modal features of the event stream and RGB frame. A dynamic region mask is generated using the Sigmoid activation function, expressed as M = σ(f_CNN(X;θ)),σ(·) = 1 / (1+e^(-x)), where σ(·) is the Sigmoid activation function and θ is the network parameter. The mask is element-wise multiplied with the optical flow field output by RAFT, thereby shielding the blurred optical flow output by the RAFT network.

[0077] Event Optical Flow Proprietary Network: The Event Optical Flow Proprietary Network (E-FlowNet) encodes the event stream into a spatiotemporal voxel grid, generates a dense optical flow field through the U-Net architecture, and fuses it with the masked RAFT optical flow through the self-attention mechanism of the Swin Transformer. The fusion formula is expressed as F_fused=

[0078] Physical constraint loss function: The physical constraint loss function combines the IMU angular velocity and the camera intrinsic parameter (f x ,f y ,c x ,c y ), the theoretical optical flow is inferred through the pinhole camera model, and the network is forced to output an optical flow consistent with the actual motion. The theoretical optical flow calculation formula is expressed as Where Z(u,v) is the depth map;

[0079] Optical flow residual analysis: Calculate the optical flow difference between E-FlowNet and RAFT, generate dynamic region masks through threshold processing, and use LSTM to reduce trajectory prediction confidence within the dynamic region to avoid collision risks.

[0080] The output of each step (optical flow field, dynamic mask, physical loss, residual) is used to optimize positioning accuracy through a tightly coupled factor graph, which is fed back to the 3D detection branch to filter dynamic false detections and assist semantic graph matching in suppressing cross-floor false matches. Ultimately, it achieves breakthrough performance with high trajectory ID retention and low motion prediction error in scenarios such as corridor pedestrian following and emergency stops in conference rooms.

[0081] In one embodiment, the specific steps of generating a 3D detection box based on VoxelNeXt-Lite fused multimodal data, combining point cloud density clustering (DBSCAN 3D) with event density thresholding to filter false detections, constructing spatio-temporal graph association trajectories through Spatio-Temporal Graph Transformer (STGT), and optimizing the continuity of pedestrian trajectory prediction using a gait cycle-aware LSTM model are as follows:

[0082] Multimodal 3D detection based on VoxelNeXt-Lite: It fuses RGB images, depth maps, and event stream data, divides the space into a voxel grid (mean, variance, density), encodes statistical features of the point cloud within each voxel, and uses 3D sparse convolution to extract spatial voxel features and generate an initial 3D detection box;

[0083] Point cloud density clustering: DBSCAN 3D clustering (parameters ∈ = 0.3m, minPts = 10) was used in combination with event density threshold to filter false detections;

[0084] Spatio-Temporal Graph Transformer (STGT) trajectory association: The Spatio-Temporal Graph Transformer constructs spatio-temporal graph association trajectories, uses node and edge features to calculate attention weights, and maintains ID consistency in high-density scenarios;

[0085] Gait cycle-aware LSTM trajectory optimization: Deploy a gait cycle-aware LSTM model, input the displacement increment and gait phase φ(t) (for example, φ(t) = 0.15 corresponds to the right foot landing phase), predict the next step position, and repair broken trajectories.

[0086] In one embodiment, the specific steps of the multi-view semantic graph matching are:

[0087] Construct a 3D semantic voxel map: The RGB-D camera depth map is integrated with the event camera vertical features (such as wall height distribution and ceiling structure) to construct a semantic voxel map. Each voxel stores a semantic label (such as wall, door, furniture) and geometric information (occupancy probability and normal direction). Bayesian filtering is used to update the voxel occupancy probability to accurately encode the static environment geometry-semantic information. The update formula is expressed as P(m_i|z_(1:t))=η·P(z_t|m_i)·P(m_i|z_(1:t-1)), where mi is the semantic label of voxel i, z_t is the current frame observation, and η is the normalization factor.

[0088] Extract vertical semantic descriptor: The voxel map is pooled with height direction and feature extracted by multi-layer perceptron through Vertical Net network to generate 256-dimensional vertical semantic descriptor to capture the normal direction of the wall and the topology of the ceiling. The vertical descriptor generation formula is expressed as f_vert=MLP(max h {ψ(x,y,z)}), where ψ(x,y,z) is the semantic-geometric joint feature of voxel (x,y,z), max h Represents the maximum pooling along the height direction;

[0089] Top-view semantic feature fusion: The top-view NetVLAD semantic features are fused with the vertical descriptor to form a 512-dimensional global feature vector. The cosine similarity and deformable graph matching (DGM) techniques are used to align the spatial relationships of the scene graph nodes.

[0090] Deformable Graph Matching (DGM) generates closed-loop candidates: Construct a scene graph G = (V, E), where nodes V are semantic voxel clusters and edges E are spatial relationships between adjacent clusters. Thin plate spline (TPS) deformation field estimation is used to suppress cross-floor mismatches, expressed as Δp = U·K·W·V,p′ = p + Δp, where p is the original node coordinate, U is the basis function matrix, K is the kernel matrix, W is the learnable weight, and p′ is the deformed coordinate.

[0091] Triggering global pose graph optimization: When a loop closure candidate is detected, the loop closure constraints are injected into the tightly coupled factor graph optimizer, and the iSAM2 incremental solver is used to minimize the nonlinear least squares objective function that includes visual reprojection error, IMU pre-integration constraints, and loop closure geometric consistency.

[0092] Dynamic semantic weight adjustment: Combining the dynamic semantic weight matrix generated by the context encoder with the implicit relationship of knowledge graph reasoning, the adapter module is used to fine-tune the semantic segmentation head parameters to avoid global retraining. Multi-view semantic graph matching.

[0093] In one embodiment, the specific steps of integrating visual reprojection error, event flow consistency loss, and 3D detection box geometric constraints in the tightly coupled factor graph optimization and realizing dynamic covariance adjustment by online estimation of the IMU noise model through LSTM are as follows:

[0094] Factor graph construction: Integrating robot pose state nodes with multi-source observation factors (visual reprojection error, event stream consistency loss, 3D detection box geometric constraints, IMU pre-integration constraints, and wheel encoder odometry) to form a global optimization framework;

[0095] Visual reprojection error term: Utilizes the optical flow field F(u,v,t) predicted by the RAFT network and the depth information of the feature points, quantifies the deviation between the optical flow prediction and the actual pixel motion through projection transformation, and introduces dynamic weight covariance to suppress interference in blurred areas;

[0096] Event stream consistency loss: This combines the dynamic edges detected by Spike-YOLO with the event stream gradients to enforce motion boundary constraints through gradient alignment errors.

[0097] 3D detection box geometric constraint: Fusing the object pose output by VoxelNeXt-Lite with the robot motion prediction, and combining the semantic label dynamic penalty term to optimize the dynamic object position estimation;

[0098] Dynamic covariance adjustment (LSTM-IMU noise estimation): predict the noise covariance through the historical IMU angular velocity sequence and dynamically adjust the factor weight;

[0099] Factor graph optimization solution: The global least squares problem is solved by integrating the Huber robust loss function through the iSAM2 incremental optimizer.

[0100] The autonomous positioning and motion perception system used by the deep learning-based autonomous positioning and motion perception method includes:

[0101] Multimodal perception layer: This layer is responsible for heterogeneous data collection and preprocessing. It uses the DAVIS346 event camera to capture event streams and synchronously output RGB frames. It combines RGB-D camera and lidar data, performs spatiotemporal alignment of heterogeneous sensors using a spatiotemporal convolutional network (TCN), and then extracts motion edge features using a Spike-YOLO network to generate a dynamic attention map.

[0102] Hybrid optical flow estimation layer: Responsible for dynamic motion field analysis, it adopts a dual-branch architecture, including an improved RAFT network and E-FlowNet. The improved RAFT network uses a dynamic mask prediction head to block blurred areas. The E-FlowNet uses U-Net to process event streams and fuse cross-modal optical flow. At the same time, it embeds a physical constraint loss function to suppress non-physical motion, significantly improving the optical flow accuracy in dynamic areas.

[0103] 3D detection and trajectory management layer: This layer is responsible for understanding dynamic targets, fusing multimodal data, generating 3D detection boxes using the lightweight VoxelNeXt-Lite algorithm, filtering false detections using DBSCAN 3D clustering and event density thresholding, constructing spatio-temporal graph-associated trajectories using the Spatio-Temporal Graph Transformer, and optimizing pedestrian trajectory prediction continuity using a gait cycle-aware LSTM model.

[0104] Semantic Graph Matching Layer: This layer is responsible for environmental recognition and closed-loop optimization. It extracts vertical features such as wall height distribution to construct a 3D semantic voxel map. Vertical Net generates vertical semantic descriptors, which are combined with top-view NetVLAD features and Deformable Graph Matching (DGM) for cross-floor closed-loop detection. It also dynamically adjusts semantic weights through context encoders and knowledge graph reasoning, and uses the Adapter module for fine-tuning to avoid global retraining.

[0105] Multi-sensor fusion layer: This layer is responsible for tightly coupled positioning optimization, integrating visual reprojection error, event stream consistency loss, 3D detection box geometric constraints, and IMU pre-integration constraints. It uses LSTM to estimate the IMU noise model online to achieve dynamic covariance adjustment. It combines the iSAM2 incremental optimizer with the Huber loss function to solve the global least squares problem and reduce positioning error.

[0106] Decision-making execution layer: Responsible for autonomous navigation and interaction. Based on the optimized posture and semantic map, it uses the A* algorithm combined with dynamic obstacle prediction to generate a safe path, and dynamically adjusts the navigation strategy through speech recognition and semantic segmentation results.

[0107] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The autonomous positioning and motion perception method based on deep learning is characterized by: The following steps are involved: Multimodal perception: The DAVIS346 event camera is used to capture RGB frames and event streams, while an RGB-D camera is used to obtain texture and depth information. LiDAR is used to supplement 3D structural data. Hand-eye calibration is used to jointly calibrate the spatial transformation matrices of the event camera and RGB-D camera. A spatiotemporal convolutional network is used to interpolate the event stream onto the RGB frame timeline for spatiotemporal alignment. The Spike-YOLO network is combined with the Spike-YOLO network to extract motion edge features from asynchronous event streams and generate dynamic attention maps to mark areas of high change. Hybrid optical flow-driven motion perception: This system uses a dual-branch architecture consisting of an improved RAFT network and a proprietary event optical flow network. It uses a dynamic mask prediction head to mask blurred areas, a self-attention mechanism to fuse cross-modal optical flows, and an embedded physical constraint loss function to suppress non-physical motion. Multimodal 3D detection and trajectory management: VoxelNeXt-Lite fuses multimodal data to generate 3D detection boxes, combines point cloud density clustering with event density thresholds to filter false detections, and constructs spatiotemporal graph-associated trajectories using the Spatio-Temporal Graph Transformer. The gait cycle-aware LSTM model is used to optimize the continuity of pedestrian trajectory prediction. Multi-view semantic graph matching: Build a 3D semantic voxel map containing vertical features, extract vertical semantic descriptors using Vertical Net, combine top-view NetVLAD features with deformable graph matching to calculate scene similarity, dynamically adjust semantic weights using context encoders and knowledge graph reasoning, and fine-tune with the Adapter module to avoid global retraining. Multi-sensor fusion and robust positioning: Integrate visual reprojection error, event stream consistency loss, and 3D detection box geometric constraints in tightly coupled factor graph optimization, and achieve dynamic covariance adjustment through LSTM online estimation of the IMU noise model.

2. The deep learning-based autonomous positioning and motion perception method according to claim 1, characterized in that: The method uses a DAVIS346 event camera to collect RGB frames and event streams, uses an RGB-D camera to obtain texture and depth information, supplements 3D structure data with a lidar, jointly calibrates the spatial transformation matrix of the event camera and RGB-D camera through hand-eye calibration, and uses a spatiotemporal convolutional network (TCN) to interpolate the event stream to the RGB frame time axis for spatiotemporal alignment. The method includes the following steps: Sensor configuration and data acquisition: Deploy a DAVIS346 event camera, an RGB-D camera, and a LiDAR. The DAVIS346 event camera collects RGB frames and synchronously outputs an asynchronous event stream. The RGB-D camera collects RGB images and depth maps. The LiDAR scans and generates point clouds. The acquisition is triggered by PPS. The spatial transformation matrix T_e2r, including the rotation matrix R and the translation vector t, is solved using a checkerboard calibration plate and OpenCV. Hand-eye calibration: Determine the transformation T_e2r=[R|t] from the event camera coordinate system {E} to the RGB-D camera coordinate system {R}, move the calibration plate to different positions, and simultaneously collect event stream and RGB-D data. Use Harris corner detection to extract corner events in the event stream and record their coordinates (x_e, y_e) in {E}. Detect the same corner in the RGB image and obtain its coordinates (x_r, y_r) in {R}. Solve by minimizing the reprojection error. Harris corner detection and minimizing the reprojection error optimize the rotation matrix R and translation vector t to complete the spatial alignment of the event camera to the RGB-D camera. Spatiotemporal Convolutional Network Interpolation Event Stream: We design a spatiotemporal convolutional network (TCN) to filter asynchronous event streams by time window, calculate the difference between the number of positive and negative events ΔN(x,y), and smooth it with Gaussian kernel convolution to generate an event intensity map I_e(x,y,t_rgb) that is spatiotemporally aligned with the RGB frame. Construct a dataset: Fuse the lidar point cloud to supplement the 3D structure data and construct a spatiotemporal aligned dataset containing depth information.

3. The autonomous positioning and motion perception method based on deep learning according to claim 2, characterized in that: The specific steps of combining the Spike-YOLO network to extract motion edge features from asynchronous event streams and generate dynamic attention maps to mark high-change areas Event stream data representation: The asynchronous events output by the DAVIS346 event camera are represented as E = {(x_k, y_k, p_k, t_k)}; Spike-YOLO network structure: uses a temporal convolutional network to perform spatiotemporal encoding of discrete events, and constructs a structured input tensor by statistically analyzing the polarity change rate and density of pixel-level events; Motion edge feature extraction: This approach uses a lightweight network architecture based on an improved YOLOv5-tiny. The event encoding layer converts sparse events into spatiotemporal features. The C3 module and SPPF pooling layer are then used to extract multi-scale spatial features. Causal convolution is then used to capture temporal dependencies. Dynamic Attention Map Generation: A dynamic attention map is generated through an additional decoder. The dynamic attention map is activated by sigmoid and thresholded to mark high-variance areas.

4. The deep learning-based autonomous positioning and motion perception method according to claim 1, characterized in that: The specific steps of using a dual-branch architecture of an improved RAFT network and a proprietary event optical flow network, shielding blurred areas through a dynamic mask prediction head, fusing cross-modal optical flow through a self-attention mechanism, and embedding a physical constraint loss function to suppress non-physical motion are as follows: Dynamic mask prediction head: The event stream and RGB frame are spliced ​​into a four-dimensional tensor. MobileNetV3-Small is used to fuse the cross-modal features of the event stream and RGB frame. A dynamic region mask is generated using the Sigmoid activation function, expressed as M = σ(f_CNN(X;θ)),σ(·) = 1 / (1+e^(-x)), where σ(·) is the Sigmoid activation function and θ is the network parameter. The mask is element-wise multiplied with the optical flow field output by RAFT, thereby shielding the blurred optical flow output by the RAFT network. Event optical flow proprietary network: The event optical flow proprietary network encodes the event flow into a spatiotemporal voxel grid, generates a dense optical flow field through the U-Net architecture, and fuses it with the masked RAFT optical flow through the self-attention mechanism of the Swin Transformer. The fusion formula is expressed as Physical constraint loss function: The physical constraint loss function combines the IMU angular velocity and the camera intrinsic parameter (f x ,f y ,c x ,c y ), the theoretical optical flow is inferred through the pinhole camera model, and the network is forced to output an optical flow consistent with the actual motion. The theoretical optical flow calculation formula is expressed as Where Z(u,v) is the depth map; Optical flow residual analysis: Calculate the optical flow difference between E-FlowNet and RAFT, generate dynamic area masks through threshold processing, and use LSTM to reduce the trajectory prediction confidence in the dynamic area to avoid collision risks.

5. The deep learning-based autonomous positioning and motion perception method according to claim 1, characterized in that: The specific steps described are: generating 3D detection boxes based on VoxelNeXt-Lite fusion multimodal data, filtering false detections by combining point cloud density clustering with event density thresholds, constructing spatiotemporal graph association trajectories using the Spatio-Temporal Graph Transformer, and optimizing the continuity of pedestrian trajectory prediction using a gait cycle-aware LSTM model. Multimodal 3D detection based on VoxelNeXt-Lite: It fuses RGB images, depth maps, and event stream data, divides the space into voxel grids, encodes statistical features of the point cloud within each voxel, and uses 3D sparse convolution to extract spatial voxel features and generate an initial 3D detection box. Point cloud density clustering: DBSCAN 3D clustering combined with event density thresholding is used to filter false positives; Spatio-Temporal Graph Transformer trajectory association: Spatio-Temporal Graph Transformer is used to construct spatio-temporal graph association trajectories and calculate attention weights using node features and edge features; Gait cycle-aware LSTM trajectory optimization: Deploy a gait cycle-aware LSTM model, input displacement increment and gait phase φ(t), predict the next step position and repair broken trajectories.

6. The deep learning-based autonomous positioning and motion perception method according to claim 1, characterized in that: The specific steps of the multi-view semantic graph matching are: Construct a 3D semantic voxel map: The RGB-D camera depth map and the event camera vertical features are integrated to construct a semantic voxel map. Each voxel stores a semantic label and geometric information. Bayesian filtering is used to update the voxel occupancy probability. The update formula is expressed as P(m_i|z_(1:t))=η·P(z_t|m_i)·P(m_i|z_(1:t-1)), where mi is the semantic label of voxel i, z_t is the current frame observation, and η is the normalization factor. Extract vertical semantic descriptor: The voxel map is pooled with height direction and feature extracted by multi-layer perceptron through Vertical Net network to generate 256-dimensional vertical semantic descriptor to capture the normal direction of the wall and the topology of the ceiling. The vertical descriptor generation formula is expressed as f_vert=MLP(max h {ψ(x,y,z)}), where ψ(x,y,z) is the semantic-geometric joint feature of voxel (x,y,z), max h Represents the maximum pooling along the height direction; Top-view semantic feature fusion: The top-view NetVLAD semantic features are fused with the vertical descriptor to form a 512-dimensional global feature vector. The spatial relationships of scene graph nodes are aligned using cosine similarity and deformable graph matching techniques. Deformable graph matching generates closed-loop candidates: Construct a scene graph G = (V, E), where nodes V are semantic voxel clusters and edges E are spatial relationships between adjacent clusters. Thin plate spline deformation field estimation is used to suppress cross-floor mismatches, expressed as Δp = U·K·W·V,p′ = p + Δp, where p is the original node coordinate, U is the basis function matrix, K is the kernel matrix, W is the learnable weight, and p′ is the deformed coordinate. Triggering global pose graph optimization: When a loop closure candidate is detected, the loop closure constraints are injected into the tightly coupled factor graph optimizer, and the iSAM2 incremental solver is used to minimize the nonlinear least squares objective function that includes visual reprojection error, IMU pre-integration constraints, and loop closure geometric consistency. Dynamic semantic weight adjustment: Combining the dynamic semantic weight matrix generated by the context encoder with the implicit relationship of knowledge graph reasoning, the adapter module is used to fine-tune the semantic segmentation head parameters.

7. The deep learning-based autonomous positioning and motion perception method according to claim 1, characterized in that: The specific steps of integrating visual reprojection error, event flow consistency loss, and 3D detection box geometric constraints in the tightly coupled factor graph optimization and realizing dynamic covariance adjustment by estimating the IMU noise model online through LSTM are as follows: Factor graph construction: integrating robot pose state nodes and multi-source observation factors to form a global optimization framework; Visual reprojection error term: Utilizes the optical flow field F(u,v,t) predicted by the RAFT network and the depth information of the feature points, quantifies the deviation between the optical flow prediction and the actual pixel motion through projection transformation, and introduces dynamic weight covariance to suppress interference in blurred areas; Event stream consistency loss: This combines the dynamic edges detected by Spike-YOLO with the event stream gradients to enforce motion boundary constraints through gradient alignment errors. 3D detection box geometric constraint: Fusing the object pose output by VoxelNeXt-Lite with the robot motion prediction, and combining the semantic label dynamic penalty term to optimize the dynamic object position estimation; Dynamic covariance adjustment: predict the noise covariance through the historical IMU angular velocity sequence and dynamically adjust the factor weight; Factor graph optimization solution: The global least squares problem is solved by integrating the Huber robust loss function through the iSAM2 incremental optimizer.

8. The autonomous positioning and motion perception system used in the deep learning-based autonomous positioning and motion perception method according to claim 1, characterized in that: include: Multimodal perception layer: This layer is responsible for heterogeneous data collection and preprocessing. It uses the DAVIS346 event camera to capture event streams and synchronously output RGB frames. It combines RGB-D camera and lidar data, performs spatiotemporal alignment of heterogeneous sensors using a spatiotemporal convolutional network, and then extracts motion edge features using a Spike-YOLO network to generate a dynamic attention map. Hybrid optical flow estimation layer: Responsible for dynamic motion field analysis, it adopts a two-branch architecture, including an improved RAFT network and E-FlowNet. The improved RAFT network uses a dynamic mask prediction head to block out blurred areas. The E-FlowNet is based on U-Net to process event streams and fuse cross-modal optical flows, while embedding a physical constraint loss function to suppress non-physical motion. 3D detection and trajectory management layer: This layer is responsible for understanding dynamic targets, fusing multimodal data, generating 3D detection boxes using the lightweight VoxelNeXt-Lite algorithm, filtering false detections using DBSCAN 3D clustering and event density thresholding, constructing spatio-temporal graph-associated trajectories using the Spatio-Temporal Graph Transformer, and optimizing pedestrian trajectory prediction continuity using a gait cycle-aware LSTM model. Semantic Graph Matching Layer: This layer is responsible for environmental recognition and closed-loop optimization. It extracts vertical features such as wall height distribution to construct a 3D semantic voxel map. Vertical Net generates vertical semantic descriptors, which are then combined with top-view NetVLAD features and deformable graph matching for cross-floor closed-loop detection. Semantic weights are dynamically adjusted through context encoders and knowledge graph reasoning, and fine-tuning is performed using the Adapter module to avoid global retraining. Multi-sensor fusion layer: This layer is responsible for tightly coupled positioning optimization, integrating visual reprojection error, event stream consistency loss, 3D detection box geometric constraints, and IMU pre-integration constraints. It uses LSTM to estimate the IMU noise model online to achieve dynamic covariance adjustment. It combines the iSAM2 incremental optimizer with the Huber loss function to solve the global least squares problem and reduce positioning error. Decision-making execution layer: Responsible for autonomous navigation and interaction. Based on the optimized posture and semantic map, it uses the A* algorithm combined with dynamic obstacle prediction to generate a safe path, and dynamically adjusts the navigation strategy through speech recognition and semantic segmentation results.

Images