Imu-aided deep slam method and system fusing language-visual multi-modal perception

Abstract

The application provides an IMU assisted deep SLAM method and system fusing language-visual multi-modal perception, comprising initial calibration and semantic map initialization, pre-integration prediction and key frame determination, dense point cloud reconstruction and relative pose estimation, semantic embedding extraction, semantic guided loop detection and semantic three-dimensional map incremental updating and the like function modules, IMU motion prior, depth geometric constraint and language model semantic factor are jointly modeled through a graph optimization framework, high-precision positioning and labeled map construction in a complex dynamic environment are realized. Compared with the prior art which only depends on geometric or inertial information, the application reduces the loop mismatching rate, improves the loop convergence efficiency and long-time repositioning robustness, and provides a semantic interface for natural language navigation, target retrieval and the like upper layer tasks. The application can be widely applied in fields of service robots, security inspection, intelligent driving and post-disaster search and rescue and the like.

Description

Technical Field

[0001] This invention relates to the field of robot autonomous localization and mapping, specifically to an IMU-assisted deep SLAM method and system that integrates language-visual multimodal perception. Background Technology

[0002] As intelligent robots continue to expand their applications in service robots, security patrols, home assistance, disaster relief, and complex indoor navigation, the demand for robots to achieve robust localization and semantic understanding mapping in unstructured and dynamically changing environments is increasing. Traditional SLAM systems mainly rely on single or limited modal sensors such as vision, LiDAR, or inertial measurement. Although some progress has been made, their performance in changing, dynamic, and semantically complex environments remains limited.

[0003] While current multi-sensor SLAM methods have improved localization accuracy and map density, they still face several challenges. Inertial measurement units (IMUs), despite their high-frequency motion tracking capabilities, are prone to long-term errors due to integral drift. Depth cameras can acquire dense geometric information of a scene, but their stability is insufficient in scenarios with varying lighting and dynamic occlusion. Furthermore, existing SLAM systems rely on traditional visual algorithms for dynamic object detection, semantic segmentation, and scene understanding, lacking the modeling and utilization of high-level semantic information of the environment, which limits their relocalization, loop closure detection, and intelligent planning capabilities in complex scenes.

[0004] It is evident that existing mainstream solutions all have shortcomings in terms of structure or performance, and are still unable to meet the comprehensive requirements of high precision, strong robustness and semantic awareness.

[0005] In recent years, Large Language Models (LLMs) have demonstrated powerful capabilities in image-text joint modeling and natural language understanding, showing potential for extracting high-level semantic information, performing contextual reasoning, and expressing spatial relationships. How to introduce language models into SLAM systems, combining the dense geometric perception provided by depth cameras with the short-term high-frequency motion estimation provided by IMUs, to achieve a localization and mapping method that integrates geometry, semantics, and language, has become an important research direction in the field of robot perception and cognitive navigation.

[0006] Therefore, there is still a need to provide a new technological solution that can achieve high-precision synchronous positioning and mapping in dynamic and semantically variable environments. Summary of the Invention

[0007] This application provides an IMU-assisted deep SLAM method and system that integrates language-visual multimodal perception to overcome the limitations of existing SLAM methods and systems in terms of dynamic environment, semantic understanding, and relocation robustness.

[0008] According to a first aspect, the present invention provides an IMU-assisted deep SLAM method that integrates language-visual multimodal perception, comprising:

[0009] S1. Initial Calibration and Semantic Map Initialization: Perform initial bias estimation on the IMU sensor to complete the extrinsic parameter calibration of the depth camera and IMU; construct an initial point cloud using multi-frame depth maps to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embeddings and initialize the semantic map.

[0010] S2. Pre-integration prediction and keyframe determination: Using the angular velocity and acceleration data output by the IMU at high frequency, the robot's pose change between two frames is predicted through pre-integration technology. At the same time, a time synchronization mechanism is combined to maintain data alignment between the IMU and the depth camera. When the cumulative displacement or rotation exceeds the set threshold, the current frame is automatically determined as a keyframe and the corresponding timestamp is recorded.

[0011] S3. Dense Point Cloud Reconstruction and Relative Pose Estimation: After the keyframe is triggered, dense point clouds are reconstructed using RGB-D images acquired by the depth camera. The relative pose between the current frame and the previous keyframe is estimated using the point cloud registration method. At the same time, motion priors provided by IMU pre-integration are introduced as initial values ​​for registration to improve registration convergence efficiency and accuracy, and the registration residuals are constrained and corrected.

[0012] S4. Keyframe semantic embedding extraction: Input the keyframe image into the pre-trained and frozen parameter large language model image encoder CLIP to extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop closure detection and map annotation.

[0013] S5. Loop closure candidate screening and verification: After the new keyframe is generated, its semantic embedding is similar to the semantic vector of the historical keyframe to screen out potential loop closure candidate frames; then, combined with geometric consistency test, point cloud registration is performed on the candidate frames to verify the loop closure relationship. If the match is successful, semantic-geometric joint constraint edges are constructed to improve the system's relocation capability and loop closure detection stability in large-scale motion or long-term operation.

[0014] S6. Semantic-guided loop closure detection: Based on the similarity calculation between semantic embedding vectors, a semantic-guided approach is used to detect loop closures, quickly filtering potential closed-loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, geometric verification is performed by combining deep point clouds to improve the robustness of relocation and the long-term consistency of the system.

[0015] S7. Semantic 3D Map Incremental Update: The global map is incrementally updated using the optimized keyframe poses. The dense point cloud reconstructed from the corresponding depth map is registered and fused. Semantic embedding information is overlaid to construct a 3D map with semantic labels, achieving synchronous maintenance of high-precision map construction and semantic awareness.

[0016] Furthermore, in S1, the initial zero bias is calculated using the acceleration and angular velocity data collected by the IMU during the stationary phase, and the bias calibration of the inertial unit is completed; the visual encoder module of the language model is initialized to provide a consistent image input format for subsequent semantic feature extraction.

[0017] Furthermore, in S2, the system receives angular velocity and acceleration data output by the IMU in real time, estimates the pose change between key frames through a pre-integration method, sets dual thresholds for displacement and rotation, and automatically selects the current frame as a key frame when the cumulative motion exceeds the threshold, providing a high-frequency motion prior and key frame triggering mechanism for subsequent depth point cloud registration and semantic processing.

[0018] Furthermore, in step S3, the RGB-D image corresponding to the key frame is acquired, a dense point cloud is generated and registered with the point cloud of the previous key frame, and the relative pose is estimated using ICP or visual feature matching methods; the initial registration value is provided by the IMU pre-integration result to improve the registration convergence efficiency and local pose estimation accuracy, and to complete the geometric alignment and local trajectory optimization of the current frame.

[0019] Furthermore, S4 specifically includes:

[0020] S401. Embed image preprocessing and visual patch construction; let the image acquired at time t be... Divide it into N image patches of size P×P, and denote each image patch as:

[0021]

[0022] Where H and W represent the height and width of the image, and the number of channels 3 corresponds to the RGB three color channels; Original Image I t The i-th image patch; vec() is the matrix flattening operation; p i It is the flattened image patch vector; the image patch is mapped to the feature space through linear projection and then positional encoding is added to obtain the initial embedding:

[0023]

[0024] in, W represents the feature representation of the i-th image patch; e The projection matrix representing the patch embedding; Represents the bias term; d represents the position code corresponding to the i-th patch; v It is a visual feature dimension;

[0025] S402. Perform Transformer encoding to obtain global visual features of the image. First, construct the initial sequence:

[0026]

[0027] in, The initial image sequence is N, where N is the number of image patches;

[0028] Next, the multi-layer Transformer with l layers is encoded:

[0029]

[0030] The output sequence of the Transformer is denoted as... is the output sequence of the Transformer at layer l; MSA() is the multi-head self-attention mechanism; MLP() is the feedforward fully connected subnetwork; LayerNorm() is the layer normalization operation; l is the total number of Transformer layers;

[0031] Finally, the features of all image patches are averaged and pooled to obtain a global visual description:

[0032]

[0033] Among them, v t Let be the global visual feature vector of the image at time t; It is the output feature of the i-th image patch in the last Transformer layer;

[0034] S403. Generate semantic embeddings through the semantic projection module, and embed the visual representation v t Input semantic projection network, mapped to semantic space:

[0035]

[0036] And perform normalization:

[0037]

[0038] in,

[0039] W s (1) and W s (2)These are all network parameters for semantic projection; σ() represents a non-linear activation function; Represents the corresponding bias; Represents output bias;

[0040] S404. Align visual features and fuse semantic representations, combining visual representations v t Projected into the semantic space and fused with semantic embedding:

[0041]

[0042] Where, v′ t The vector after alignment with visual features; Alignment is represented by an affine transformation matrix; Representation of bias terms; Re-fusion feature representation:

[0043] h t =λ·f t +(1-λ)·v′ t ,λ∈[0,1]

[0044] Among them, h t It is the vector obtained after multimodal fusion; f t It is the semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantic and visual elements.

[0045] S405. Construct a keyframe index set. If the current frame meets the keyframe determination criteria, add its timestamp to the keyframe set.

[0046]

[0047] in, It is a set of keyframe time indices; t is the current keyframe time;

[0048] S406. Construct a complete semantic database and write the fused features into the keyframe semantic database, that is, to form a data record by combining the timestamp of the current keyframe with the fused representation:

[0049]

[0050] in, It is a semantic database of keyframes, with the database dimension being... K is the number of keyframes currently recorded by the system; t i The symbol represents the acquisition time of the frame on the timeline; h represents the time when the i-th keyframe is acquired. i The transpose of the extracted semantic-visual fusion feature vector.

[0051] Furthermore, S5 specifically includes:

[0052] S501. Normalize the semantic vectors of the current frame and historical keyframes. To perform semantic similarity matching, the semantic vectors of the current frame and the features of historical frames are first normalized to unify the vector scale.

[0053]

[0054] Among them, h t It is the multimodal fusion feature of the current frame; It is a multimodal fusion feature of historical frames; It is a vector The modulus length; It is a unit-normalized vector;

[0055] S502. Calculate and rank the semantic similarity scores. Calculate the semantic cosine similarity using the dot product of normalized vectors to construct candidate score vectors.

[0056]

[0057] in, Represents the current frame t and the historical frame t i The semantic similarity score is then used to construct a vector from all similarities:

[0058]

[0059] Finally, regarding s t Sort in descending order and retain the top N similarity scores. c The frames are used as the initial selection set:

[0060]

[0061] Wherein, π(t) is the index sequence after the score vector is sorted in descending order of value; N represents the j-th historical keyframe with the highest similarity; c This refers to the number of candidates to be retained, typically chosen to be 20-40; C top This indicates that the current frame ranks among the top N in semantic similarity scoring. c A collection of historical keyframes;

[0062] S503. Perform threshold filtering and inter-frame distance limiting to obtain the final candidate set; further remove frames with low similarity or too close in time from the sorting results:

[0063]

[0064]

[0065] Among them, C loop θ represents the final set of candidate loop closure frames to be verified. sSemantic similarity threshold; Δt min Minimum inter-frame interval;

[0066] S504. Perform point cloud registration, estimate relative pose, and verify geometric consistency; select each t i ∈C loop Construct the initial relative pose:

[0067]

[0068] Among them, T t Represents the relative pose change at the current moment; This represents the relative pose transformation at the previous moment; then, the point-to-plane ICP method is used to minimize the geometric error:

[0069]

[0070] in, This represents the estimated relative transformation from the current frame to historical frames; This represents the k-th point in the point cloud of the current frame; Indicates the relationship between historical frames and Matching points; Point The unit normal vector of the local surface; M is the number of point pairs participating in the matching optimization; SE(3) is the rigid body transformation group in three-dimensional space;

[0071] Finally, define the mean of the matching error:

[0072]

[0073] Where, ∈ fit This represents the matching error, used for ∈ geo The comparison is used to check whether a geometrically valid closure is formed;

[0074] S505. Calculate the loop closure confidence score and dynamically set the loop closure factor weights, then integrate semantic similarity and geometric residuals to construct a confidence index:

[0075]

[0076] in, σ represents the confidence level of the loop closure matching; σ is a scaling factor that controls the influence of the matching residuals, and this value is used for weighting the information matrix in subsequent graph optimization.

[0077] S506. In graph optimization, the loop-closing residual term and joint factor, geometric residual and semantic residual are defined as follows:

[0078]

[0079] Where, rgeo Represents the geometric residual, derived from Lie group errors; r sem Represents semantic residuals; This refers to semantic similarity scoring; finally, the overall loop closure loss function can be obtained as follows:

[0080]

[0081] Among them, E loop This indicates the contribution of the lapsing factor to the overall graph optimization; Ω geo The information matrix for the geometric error term; Represents the confidence level of loop closure matching; w sem This represents the basic coefficient of semantic loss.

[0082] Furthermore, in S6, based on the constructed odometry factor and loop closure factor, a nonlinear optimization method is used to globally adjust the pose of the key frame; during the optimization process, geometric residuals and semantic residuals are considered together, and the influence of each factor is controlled by weighting the information matrix to improve the consistency and accuracy of trajectory estimation and map construction.

[0083] Furthermore, S7 specifically includes:

[0084] S701. Dense point cloud reconstructed from keyframe pose and depth map after fusion and optimization. The optimized keyframe pose is represented as follows:

[0085]

[0086] in, This represents the optimized keyframe pose. This represents the optimized pose of the i-th frame; then, the keyframe images of each frame are... Convert the data into the corresponding depth map and intrinsic parameter matrix, and convert each pixel into a point cloud:

[0087]

[0088] in, These are the corresponding three-dimensional spatial points; K represents the depth value at that pixel. -1 This is the depth camera intrinsic parameter matrix; (u,v) represents a pixel; finally, all frames are aggregated to generate a point cloud map:

[0089]

[0090] in, An abstract symbol for a point cloud map, representing a set of points;

[0091] S702. Map the semantic features of each frame to three-dimensional points to construct a semantic annotation map; let the keyframe t be... iThe semantic vector is The semantic vector of each frame of the image is then assigned to the generated point cloud as follows:

[0092]

[0093] Reconstruct the set of point semantic pairs:

[0094]

[0095] Where P is S is an abbreviation for point in three-dimensional space; S is... The abbreviation for represents the semantic description vector of the keyframe to which the point belongs; It is a semantically enhanced 3D map;

[0096] S703, Complexity of semantic target matching modeling; While calculating single-point matching, a global semantic matching distribution is constructed, and a softmax weighting mechanism is added for 3D spatial localization: the target semantic vector is set to h. auery Its normalization is Calculate the cosine similarity for all points on the map:

[0097]

[0098] Among them, s j Cosine similarity; Normalize the cosine similarity; then construct the softmax weight distribution:

[0099]

[0100] Among them, w j Parameters are assigned to the weights; α>0 is the gradient coefficient, which controls the steepness of the similarity distribution; finally, the semantic similarity of the semantic target points within each range is calculated, and the weighted average of the positions of all points is taken to obtain a spatial centroid position:

[0101]

[0102] Where, p goal That is, the location of the centroid of the point cloud space; M is the number of semantic points; p j Let J be the coordinates of the j-th semantic point on the map.

[0103] S704. Establish semantic database structure and query interface modeling; model semantic point clouds. Structured into a queryable database table D, each record can be represented as:

[0104] D j =(p j ,s j,t j ,(u j ,v j ))

[0105] Where, p j Indicates the position of a three-dimensional point; s j t represents a semantic description vector; j Indicates the keyframe time to which this point belongs; (u j ,v j () represents the image pixel coordinates corresponding to that point; finally, the query interface is modeled:

[0106]

[0107] Among them, h input It is the semantic vector after natural language conversion; τ is the set semantic matching threshold; the returned set is the set of spatial points whose semantic matching degree is greater than the threshold.

[0108] According to a second aspect, the present invention provides an IMU-assisted deep SLAM system that integrates language-visual multimodal perception, the system comprising:

[0109] Initial calibration and semantic map initialization module: used to perform initial bias estimation of the IMU sensor and complete the extrinsic parameter calibration of the depth camera and IMU; construct initial point cloud through multi-frame depth map to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embedding and initialize semantic map;

[0110] Pre-integration prediction and keyframe determination module: It is used to predict the robot's pose change between two frames by using the angular velocity and acceleration data output by the IMU at high frequency and through pre-integration technology. At the same time, it combines a time synchronization mechanism to maintain data alignment between the IMU and the depth camera. When the cumulative displacement or rotation exceeds the set threshold, it automatically determines the current frame as a keyframe and records the corresponding timestamp.

[0111] Dense point cloud reconstruction and relative pose estimation module: After a keyframe is triggered, it reconstructs a dense point cloud using RGB-D images acquired by a depth camera, estimates the relative pose between the current frame and the previous keyframe through point cloud registration, and introduces motion priors provided by IMU pre-integration as initial values ​​for registration to improve registration convergence efficiency and accuracy, and performs constraint correction on the registration residuals.

[0112] Keyframe semantic embedding extraction module: This module is used to input keyframe images into the pre-trained and frozen parameters of the large language model image encoder CLIP, extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop closure detection and map annotation.

[0113] The loop closure candidate screening and verification module is used to calculate the similarity between the semantic embedding of a new keyframe and the semantic vector of the historical keyframe after the new keyframe is generated, and to screen out potential loop closure candidate frames. Then, combined with the geometric consistency test, point cloud registration is performed on the candidate frames to verify the loop closure relationship. If the match is successful, semantic-geometric joint constraint edges are constructed to improve the relocation capability and loop closure detection stability of the system in large-scale motion or long-term operation.

[0114] Semantic-guided loop closure detection module: It is used to calculate the similarity between semantic embedding vectors and perform loop closure detection in a semantic-guided manner. It quickly filters potential closed loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, it combines deep point cloud for geometric verification to improve the relocation robustness and long-term consistency of the system.

[0115] Semantic 3D Map Incremental Update Module: This module uses the optimized keyframe poses to incrementally update the global map, register and fuse the dense point cloud reconstructed from the corresponding depth map, and overlay semantic embedding information to construct a 3D map with semantic labels, thus achieving simultaneous maintenance of high-precision map construction and semantic awareness.

[0116] According to a third aspect, the present invention provides an electronic device, the device comprising: a processor and a memory;

[0117] The memory is used to store one or more program instructions;

[0118] The processor is configured to run one or more program instructions to perform the steps of the IMU-assisted deep SLAM method for fused language-visual multimodal perception as described in any one of claims 1 to 8.

[0119] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0120] This invention presents a method and system based on a multimodal collaborative localization and mapping approach that integrates a depth camera, an IMU (Integrated Mutual Acuity Detection Unit), and a large language model. Within a graph optimization framework, it leverages the advantages of these three types of sensor information: the high-frequency short-term motion prediction provided by the IMU serves as a motion prior, ensuring the timeliness and continuity of sensor fusion; the depth camera outputs dense spatial geometric information, which, after alignment with keyframe depth features, achieves accurate relative pose estimation; and the large language model performs semantic understanding of image frames and generates embeddings for constructing a semantic embedding map. Through a language-guided keyframe selection mechanism and a dynamic target removal strategy, the system significantly improves robustness in semantically complex and dynamically disturbed environments. In loop closure detection, the integration of language-visual bimodal matching enhances long-term relocalization capabilities. Overall, the design integrating IMU pre-integration, depth geometric matching, and semantic prior guidance significantly improves the localization accuracy, semantic consistency, and environmental adaptability of SLAM, possessing good engineering practical value and expansion potential, and capable of meeting the needs of robots for long-term autonomous operation in real-world complex scenarios. Attached Figure Description

[0121] Figure 1 This is a structural diagram of the method of the present invention;

[0122] Figure 2 This is an overall structural diagram of the IMU-assisted deep SLAM method that integrates language-visual multimodal perception according to an embodiment of the present invention;

[0123] Figure 3 This is a flowchart of the keyframe semantic embedding and extraction process in an embodiment of the present invention;

[0124] Figure 4 This is a flowchart of the loopback candidate frame screening and verification process according to an embodiment of the present invention;

[0125] Figure 5 This is a flowchart illustrating the semantic map construction and incremental update process according to an embodiment of the present invention.

[0126] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0127] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0128] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0129] like Figure 1 As shown, this invention proposes a multimodal simultaneous localization and mapping (SLAM) method integrating language model, depth camera, and IMU information, named LLM-VINS (Language-enhanced Visual-Inertial Navigation System). Based on multi-source perceptual information, this method significantly improves the system's localization stability and semantic representation capability in complex dynamic environments through a joint visual-semantic mapping and loop closure detection strategy. First, in the system initialization phase, IMU calibration, initial point cloud generation, and image segmentation tasks are completed, and then the system enters the visual-semantic processing module. This module combines IMU pre-integration and semantic segmentation results, constructs and optimizes factor graphs, and outputs the estimated pose for map updating and maintenance. Figure 3 As shown, the system introduces a large language model image encoder to perform Transformer semantic encoding on keyframe images. The image is divided into multiple image patches and mapped to a feature space. Global visual features are extracted through a multi-layer Transformer, and then input into a semantic projection network to obtain a unified semantic embedding vector. Finally, this semantic information is aligned with and fused with the visual information to construct a multimodal representation for keyframe identification and management. If a frame is identified as a keyframe, its timestamp is recorded and it is added to the keyframe set. Figure 4As shown, in the loop closure detection stage, the system compares the similarity between the semantic vector of the current frame and the semantic embeddings of historical keyframes, and constructs a ranked scoring list. The top N candidate frames by similarity are retained, and frames with insufficient time intervals are removed, forming the final loop closure candidate set. Subsequently, point cloud registration and ICP evaluation are performed on the candidate frames. If the geometric error meets the set conditions, semantic and geometric factors are added to the graph optimization to form closed-loop constraint edges, improving the system's long-term consistency and loop closure robustness. Figure 5 As shown, the optimized keyframe pose and depth map are used together to generate a dense point cloud map, while the semantic embedding of the keyframes is used to construct semantic annotations for the point cloud. This semantic point cloud is then registered and fused with an existing map to complete a global map update. The system further determines whether the current semantic point meets the storage conditions. If it does, the point's location, semantic vector, time, and image index are written into the semantic database to construct a queryable semantic map structure, providing support for subsequent natural language navigation and target matching.

[0130] See Figure 1 A visual-inertial simultaneous localization and mapping method integrating semantic embedding of language models is summarized as follows:

[0131] S1. System initialization, including calibrating the IMU sensor in a stationary state, and estimating its initial bias b by acquiring multiple frames of acceleration and angular velocity data. a b g Simultaneously, the images acquired by the depth camera I t The system is converted into a dense point cloud and the initial pose T0 of the system is estimated using an initial registration method. Further, the extrinsic parameters between the IMU and the camera are estimated based on a calibration board or structured light method, and a language model image encoding module with frozen weights is loaded for subsequent multimodal semantic feature extraction.

[0132] In S1, system initialization includes static estimation of the IMU's zero bias. Multiple frames of IMU observation data are continuously acquired while the system is stationary. The biases of the gyroscope and accelerometer are estimated using statistical averaging, where ω... t a t Let represent the observed angular velocity and acceleration at time t, respectively, and let the unit vector of gravity be . according to The accelerometer bias b was obtained respectively. g gyroscope bias b g Simultaneously, extrinsic parameter calibration between the depth camera and IMU was completed. By acquiring synchronous pose pairs of both under known trajectories, rotation and translation error models were constructed, and the extrinsic parameter transformation matrix from the camera to the IMU was estimated. Furthermore, to initialize the visual encoder for the language model, image size normalization and channel normalization operations were performed, and the IMU values ​​for each frame were adjusted. t Process input tensors into a uniform format Where I(u,v,c) is the original pixel value at pixel position (u,v) in the c-th channel; μ c and σ c , , are the mean and standard deviation of the image in this channel, respectively, used for standardization to meet the input requirements of the subsequent Transformer structure, and the language visual encoding module with frozen parameters is preloaded to ensure the stability and comparability of the subsequent semantic feature extraction results.

[0133] S2, IMU data processing and keyframe selection: The system acquires the high-frequency angular velocity ω of the IMU sensor in real time. t and acceleration a t The data was obtained by approximating the displacement, velocity, and attitude increments using a pre-integration method. Compensation for bias term b during integration a b g Set rotation and translation thresholds ∈ p ,∈ r When the cumulative pose change exceeds a threshold, a keyframe is automatically triggered to control computational complexity and maintain the accuracy of state estimation.

[0134] In S2, the system acquires the high-frequency angular velocity ω output by the IMU sensor in real time during operation. t and acceleration a t The data is used to perform integral estimation of the pose change between any two time points using a pre-integral model, where the angular velocity is subtracted from the gyroscope bias b. g The rotation increment is calculated by accumulating the rotation in the SO(3) group space. Simultaneously, by combining the corrected acceleration with the current attitude estimate, the velocity and displacement increments are calculated. Used to describe the relative motion trend between keyframes, where exp() is the exponential mapping from Lie algebra to Lie group; ∧ Represent the antisymmetric matrix form of so(3); R t v t b represents the rotation matrix and velocity of the system at time t, respectively; g b a These are the estimated gyroscope and accelerometer biases, respectively; to control the number of system states and maintain global consistency, keyframe triggering conditions are set, including displacement thresholds ∈ p With rotation angle threshold ∈ r When the cumulative displacement ||Δp|| and rotation angle ∠(ΔR) between consecutive frames exceed the set threshold, the current frame is triggered to be written into the factor graph as a new keyframe.

[0135] S3, Point cloud registration and relative pose estimation, for the RGB-D image corresponding to the current keyframe I t Depth image D t The image pixels are back-projected into a dense 3D point cloud using the intrinsic parameter matrix K. The point cloud of the previous keyframe is then registered to a plane using the ICP method. The initial registration values ​​are provided by IMU prediction. The relative pose between the two frames is output, and the geometric constraint factors between the keyframes are constructed and added to the backend graph optimization module.

[0136] In S3, the system acquires the color image I corresponding to the current frame after the keyframe is triggered. t With depth image D t Based on the camera intrinsic parameter matrix K = [f x ,f y ,c x ,c y The pixel (u,v) is back-projected into 3D space to generate a dense point cloud in the camera coordinate system. The coordinates of each point are obtained through The calculation is then performed; subsequently, the point cloud of the previous keyframe is extracted, and the IMU pre-integration result is used as the initial pose estimate. The ICP algorithm is used to register the point cloud of the current frame with that of the previous frame, minimizing the geometric residual between the two frames.

[0137] S4. Keyframe semantic embedding and extraction: The current frame image is segmented to generate an image patch sequence p. i After linear transformation and positional encoding, the initial embedding is formed. Input into the Transformer structure to extract visual features v t Then, a normalized semantic embedding is output through a semantic projection network and combined with the visual features f after affine transformation. t The final multimodal semantic vector h is generated by proportional fusion. t It is written into the semantic database along with the timestamp t to support subsequent semantic loop recognition and semantic map annotation.

[0138] In S4, after keyframe selection, the system inputs the current frame's RGB image into the preloaded language model image encoder, first dividing the image into several image blocks. in, Original Image I t The i-th image patch; vec() is the matrix flattening operation; p i It is a flattened image patch vector; each image patch is flattened, linearly mapped, and positionally encoded to generate an initial embedding vector. in W represents the feature representation of the i-th image patch; eThe projection matrix representing the patch embedding; b e Represents the bias term; d represents the position code corresponding to the i-th patch; v The visual feature dimension is 256; then, through a deep Transformer structure, layer-by-layer encoding is performed to extract the semantic feature representation of each image patch. v t Let be the global visual feature vector of the image at time t; It is the output feature of the i-th image patch in the last Transformer layer; the outputs of all image patches are averaged and pooled to generate the global visual feature vector of the current frame. and These are all network parameters for semantic projection; σ() represents a non-linear activation function; Represents the corresponding bias; The output bias is represented by the visual features, which are then aligned and fused with the semantic representation; the visual representation v is then... t Projecting onto the semantic space and fusing it with semantic embedding yields v′. t =MLP align (v t ) = W a ·v t +b a v′ t The vector aligned with visual features; W a Alignment is indicated by an affine transformation matrix; b a Represents the bias term; re-fusion feature representation: h t =λ·f t +(1-λ)·v′ t h t It is the vector obtained after multimodal fusion; f t It is a semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantic and visual elements; finally, it is written into the semantic database along with the timestamp t.

[0139] S5. The semantic embedding vector h between the current frame and historical keyframes t , Perform cosine similarity calculation and extract similarity scores. Candidate frames exceeding a set threshold are filtered and geometric registration is performed. The loop closure confidence score is then calculated using the mean error. Thus, the overall loss function of the cyclic factor is obtained as E. loop .

[0140] In S5, after generating the current keyframe, the system extracts its semantic embedding vector from the semantic database. and semantic vectors corresponding to all historical keyframes After normalization, the matching score for each frame is obtained by calculating cosine similarity. Represents the current frame t and the historical frame t i The semantic similarity scores are then calculated, and the scores are sorted in descending order, retaining the top N similarity scores. c The frames are used as the initial selection set: Wherein, π(t) is the index sequence after the score vector is sorted in descending order of value; N represents the j-th historical keyframe with the highest similarity; c C is the number of candidates retained; top This indicates that the current frame ranks among the top N in semantic similarity scoring. c The historical keyframe set; further thresholding and inter-frame temporal distance filtering are applied to candidate frames, retaining only frames with high semantic similarity and sufficiently long temporal distance: θ s Semantic similarity threshold; Δt min Minimize inter-frame interval; perform point cloud registration to estimate its relative pose transformation, and then use the point-to-plane ICP method for registration optimization to minimize geometric error. Among them This represents the estimated relative transformation from the current frame to historical frames; This represents the k-th point in the point cloud of the current frame; Indicates the relationship between historical frames and Matching points; Point The unit normal vector of the local surface; M is the number of point pairs participating in the matching optimization; finally, the joint factor residual term is constructed and added to the graph optimization structure;

[0141] S6. The IMU pre-integration factor, point cloud registration factor, and semantic loop closure factor are uniformly modeled as a nonlinear factor graph. The key frame state is adjusted by iterative optimization method, and the trajectory estimation results are updated by global solution through Levenberg-Marquardt method.

[0142] In S6, the system uses a factor graph optimization framework to uniformly model and jointly solve the state constraints provided by multiple sensors, and then uses the joint lapsing loss function E obtained in the previous step. loop Construct the entire factor graph, including: IMU pre-integration factor, loop closure constraint factor, semantic consistency factor, and geometric registration factor, and construct the above factors into a nonlinear optimization objective function. Where X represents the pose of all optimization variables and keyframes; r i For various residual terms, including IMU, geometric, semantic, and loop closure; Ω i This is the corresponding information matrix; the final output is the globally consistent optimized keyframe trajectory result.

[0143] S7. Based on the optimized keyframe pose Projecting its corresponding point cloud onto the world coordinate system generates a global point cloud. The semantic vectors of the keyframes are then assigned to each point cloud to form a semantically labeled map; a semantic query interface is then built to support subsequent target localization and language navigation.

[0144] In S7, after completing the global optimization of the factor graph, the system uses the updated keyframe pose estimation. Perform coordinate transformation on the dense point cloud of the corresponding frame to unify it to the world coordinate system, and construct a global dense point cloud map. To support downstream semantic queries and natural language interaction, the system establishes a structured semantic database, storing information such as point locations, semantic features, keyframe timestamps, and image pixel indices. When the input target semantic description vector h is... query At that time, the system calculates the cosine similarity s between it and the semantic vectors in all databases. j The weight distribution is obtained by softmax normalization using the gradient coefficient α. The final output is a weighted average point estimate of the spatial location of the target semantics in the map. p goal That is, the location of the centroid of the point cloud space; M is the number of semantic points; p j The system sets the location coordinates of the j-th semantic point on the map, and then writes all semantic points, keyframe timestamps, image pixel indexes, and other information into a structured database to form a complete semantic map D that can be used for semantic retrieval and navigation. j =(p j ,s j ,t j ,(u j ,v j ), where p j Indicates the position of a three-dimensional point; s j t represents a semantic description vector; j Indicates the keyframe time to which this point belongs; (u j ,v j () represents the pixel coordinates of the corresponding point in the image.

[0145] The following combination Figure 2 A specific embodiment of the present invention will be described in detail below. See also: Figure 2 A method for IMU-assisted deep SLAM that integrates language-visual multimodal perception, comprising:

[0146] S1. Initial Calibration and Semantic Map Initialization: Perform initial bias estimation on the IMU sensor to complete the extrinsic parameter calibration of the depth camera and IMU; construct an initial point cloud using multi-frame depth maps to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embeddings and initialize the semantic map.

[0147] In step S1, the initial zero bias is calculated using the acceleration and angular velocity data collected by the IMU during the stationary phase, and the bias calibration of the inertial unit is completed; the visual encoder module of the language model is initialized to provide a consistent image input format for subsequent semantic feature extraction.

[0148] S2. Pre-integration prediction and keyframe determination: Using the angular velocity and acceleration data output by the IMU at high frequency, the robot's pose change between two frames is predicted through pre-integration technology. At the same time, a time synchronization mechanism is combined to maintain data alignment between the IMU and the depth camera. When the cumulative displacement or rotation exceeds the set threshold, the current frame is automatically determined as a keyframe and the corresponding timestamp is recorded.

[0149] In S2, the system receives angular velocity and acceleration data output by the IMU in real time, estimates the pose change between key frames through a pre-integration method, sets dual thresholds for displacement and rotation, and automatically selects the current frame as a key frame when the cumulative motion exceeds the threshold, providing a high-frequency motion prior and key frame triggering mechanism for subsequent depth point cloud registration and semantic processing.

[0150] S3. Dense Point Cloud Reconstruction and Relative Pose Estimation: After the keyframe is triggered, dense point clouds are reconstructed using RGB-D images acquired by the depth camera. The relative pose between the current frame and the previous keyframe is estimated using the point cloud registration method. At the same time, motion priors provided by IMU pre-integration are introduced as initial values ​​for registration to improve registration convergence efficiency and accuracy, and the registration residuals are constrained and corrected.

[0151] In step S3, the RGB-D image corresponding to the key frame is acquired, a dense point cloud is generated and registered with the point cloud of the previous key frame, and the relative pose is estimated using ICP or visual feature matching methods. The initial registration value is provided by the IMU pre-integration result to improve the registration convergence efficiency and local pose estimation accuracy, and to complete the geometric alignment and local trajectory optimization of the current frame.

[0152] S4. Keyframe semantic embedding extraction: Input the keyframe image into the pre-trained and frozen parameter large language model image encoder CLIP to extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop closure detection and map annotation.

[0153] S4 specifically includes:

[0154] S401. Embed image preprocessing and visual patch construction; let the image acquired at time t be... Divide it into N image patches of size P×P, and denote each image patch as:

[0155]

[0156] Where H and W represent the height and width of the image, and the number of channels 3 corresponds to the RGB three color channels; Original Image I t The i-th image patch; vec() is the matrix flattening operation; p i It is the flattened image patch vector; the image patch is mapped to the feature space through linear projection and then positional encoding is added to obtain the initial embedding:

[0157]

[0158] in, W represents the feature representation of the i-th image patch; e The projection matrix representing the patch embedding; Represents the bias term; d represents the position code corresponding to the i-th patch; v It is a visual feature dimension;

[0159] S402. Perform Transformer encoding to obtain global visual features of the image. First, construct the initial sequence:

[0160]

[0161] in, The initial image sequence is N, where N is the number of image patches;

[0162] Next, the multi-layer Transformer with l layers is encoded:

[0163]

[0164] The output sequence of the Transformer is denoted as... is the output sequence of the Transformer at layer l; MSA() is the multi-head self-attention mechanism; MLP() is the feedforward fully connected subnetwork; LayerNorm() is the layer normalization operation; l is the total number of Transformer layers;

[0165] Finally, the features of all image patches are averaged and pooled to obtain a global visual description:

[0166]

[0167] Among them, v tLet be the global visual feature vector of the image at time t; It is the output feature of the i-th image patch in the last Transformer layer;

[0168] S403. Generate semantic embeddings through the semantic projection module, and embed the visual representation v t Input semantic projection network, mapped to semantic space:

[0169]

[0170] And perform normalization:

[0171]

[0172] in,

[0173] Its main purpose is to convert it into a unit vector; W s (1) and W s (2) These are all network parameters for semantic projection; σ() represents a non-linear activation function; Represents the corresponding bias; This represents the output bias; the final normalization operation is mainly to make it suitable for subsequent similarity measurement.

[0174] S404. Align visual features and fuse semantic representations, combining visual representations v t Projected into the semantic space and fused with semantic embedding:

[0175]

[0176] Where, v′ t The vector after alignment with visual features; Alignment is represented by an affine transformation matrix; Representation of bias terms; Re-fusion feature representation:

[0177] h t =λ·f t +(1-λ)·v′ t ,λ∈[0,1]

[0178] Among them, h t It is the vector obtained after multimodal fusion; f t It is the semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantic and visual elements.

[0179] S405. Construct a keyframe index set. If the current frame meets the keyframe determination criteria, add its timestamp to the keyframe set.

[0180]

[0181] in, It is a set of keyframe time indices; t is the current keyframe time;

[0182] S406. Construct a complete semantic database and write the fused features into the keyframe semantic database, that is, to form a data record by combining the timestamp of the current keyframe with the fused representation:

[0183]

[0184] in, It is a semantic database of keyframes, with the database dimension being... K is the number of keyframes currently recorded by the system; t i The symbol represents the acquisition time of the frame on the timeline; h represents the time when the i-th keyframe is acquired. i The transpose of the extracted semantic-visual fusion feature vector.

[0185] S5. Loop closure candidate screening and verification: After the new keyframe is generated, its semantic embedding is similar to the semantic vector of the historical keyframe to screen out potential loop closure candidate frames; then, combined with geometric consistency test, point cloud registration is performed on the candidate frames to verify the loop closure relationship. If the match is successful, semantic-geometric joint constraint edges are constructed to improve the system's relocation capability and loop closure detection stability in large-scale motion or long-term operation.

[0186] S5 specifically includes:

[0187] S501. Normalize the semantic vectors of the current frame and historical keyframes. To perform semantic similarity matching, the semantic vectors of the current frame and the features of historical frames are first normalized to unify the vector scale.

[0188]

[0189] Among them, h t It is the multimodal fusion feature of the current frame; It is a multimodal fusion feature of historical frames; It is a vector The modulus length; It is a unit-normalized vector;

[0190] S502. Calculate and rank the semantic similarity scores. Calculate the semantic cosine similarity using the dot product of normalized vectors to construct candidate score vectors.

[0191]

[0192] in, Represents the current frame t and the historical frame ti The semantic similarity score is then used to construct a vector from all similarities:

[0193]

[0194] Finally, regarding s t Sort in descending order and retain the top N similarity scores. c The frames are used as the initial selection set:

[0195]

[0196] Wherein, π(t) is the index sequence after the score vector is sorted in descending order of value; N represents the j-th historical keyframe with the highest similarity; c This refers to the number of candidates to be retained, typically chosen to be 20-40; C top This indicates that the current frame ranks among the top N in semantic similarity scoring. c A collection of historical keyframes;

[0197] S503. Perform threshold filtering and inter-frame distance limiting to obtain the final candidate set; further remove frames with low similarity or too close in time from the sorting results:

[0198]

[0199] Among them, C loop θ represents the final set of candidate loop closure frames to be verified. s The semantic similarity threshold is typically set to 0.75; Δt min The minimum inter-frame interval is typically set to 2 seconds.

[0200] S504. Perform point cloud registration, estimate relative pose, and verify geometric consistency; select each t i ∈C loop Construct the initial relative pose:

[0201]

[0202] Among them, T t Represents the relative pose change at the current moment; This represents the relative pose transformation at the previous moment; then, the point-to-plane ICP method is used to minimize the geometric error:

[0203]

[0204] in, This represents the estimated relative transformation from the current frame to historical frames; This represents the k-th point in the point cloud of the current frame; Indicates the relationship between historical frames and Matching points; Point The unit normal vector of the local surface; M is the number of point pairs participating in the matching optimization; SE(3) is the rigid body transformation group in three-dimensional space;

[0205] Finally, define the mean of the matching error:

[0206]

[0207] Where, ∈ fit This represents the matching error, used for ∈ geo The comparison is used to check whether a geometrically valid closure is formed;

[0208] S505. Calculate the loop closure confidence score and dynamically set the loop closure factor weights, then integrate semantic similarity and geometric residuals to construct a confidence index:

[0209]

[0210] in, σ represents the confidence level of the loop closure matching; σ is a scaling factor that controls the influence of the matching residuals, and this value is used for weighting the information matrix in subsequent graph optimization.

[0211] S506. In graph optimization, the loop-closing residual term and joint factor, geometric residual and semantic residual are defined as follows:

[0212]

[0213] Where, r geo Represents the geometric residual, derived from Lie group errors; r sem Represents semantic residuals; This refers to semantic similarity scoring; finally, the overall loop closure loss function can be obtained as follows:

[0214]

[0215] Among them, E loop This indicates the contribution of the lapsing factor to the overall graph optimization; Ω geo The information matrix for the geometric error term; Represents the confidence level of loop closure matching; w sem This represents the basic coefficient of semantic loss.

[0216] S6. Semantic-guided loop closure detection: Based on the similarity calculation between semantic embedding vectors, a semantic-guided approach is used to detect loop closures, quickly filtering potential closed-loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, geometric verification is performed by combining deep point clouds to improve the robustness of relocation and the long-term consistency of the system.

[0217] In step S6, based on the constructed odometry factor and closure factor, a nonlinear optimization method is used to globally adjust the pose of the key frame. During the optimization process, geometric residuals and semantic residuals are considered together, and the influence of each factor is controlled by weighting the information matrix to improve the consistency and accuracy of trajectory estimation and map construction.

[0218] S7. Semantic 3D Map Incremental Update: The global map is incrementally updated using the optimized keyframe poses. The dense point cloud reconstructed from the corresponding depth map is registered and fused. Semantic embedding information is overlaid to construct a 3D map with semantic labels, achieving synchronous maintenance of high-precision map construction and semantic awareness.

[0219] Specifically, S7 includes:

[0220] S701. Dense point cloud reconstructed from keyframe pose and depth map after fusion and optimization. The optimized keyframe pose is represented as follows:

[0221]

[0222] in, This represents the optimized keyframe pose. This represents the optimized pose of the i-th frame; then, the keyframe images of each frame are... Convert the data into the corresponding depth map and intrinsic parameter matrix, and convert each pixel into a point cloud:

[0223]

[0224] in, These are the corresponding three-dimensional spatial points; K represents the depth value at that pixel. -1 This is the depth camera intrinsic parameter matrix; (u,v) represents a pixel; finally, all frames are aggregated to generate a point cloud map:

[0225]

[0226] in, An abstract symbol for a point cloud map, representing a set of points;

[0227] S702. Map the semantic features of each frame to three-dimensional points to construct a semantic annotation map; let the keyframe t be... i The semantic vector is The semantic vector of each frame of the image is then assigned to the generated point cloud as follows:

[0228]

[0229] Reconstruct the set of point semantic pairs:

[0230]

[0231] Where P is S is an abbreviation for point in three-dimensional space; S is... The abbreviation for represents the semantic description vector of the keyframe to which the point belongs; It is a semantically enhanced 3D map;

[0232] S703, Complexity of semantic target matching modeling; While calculating single-point matching, a global semantic matching distribution is constructed, and a softmax weighting mechanism is added for 3D spatial localization: the target semantic vector is set to h. query Its normalization is Calculate the cosine similarity for all points on the map:

[0233]

[0234] Among them, s j Cosine similarity; Normalize the cosine similarity; then construct the softmax weight distribution:

[0235]

[0236] Among them, w j Parameters are assigned to the weights; α>0 is the gradient coefficient, which controls the steepness of the similarity distribution; finally, the semantic similarity of the semantic target points within each range is calculated, and the weighted average of the positions of all points is taken to obtain a spatial centroid position:

[0237]

[0238] Where, p goal That is, the location of the centroid of the point cloud space; M is the number of semantic points; p j Let J be the coordinates of the j-th semantic point on the map.

[0239] S704. Establish semantic database structure and query interface modeling; model semantic point clouds. Structured into a queryable database table D, each record can be represented as:

[0240] D j =(p j ,s j ,t j ,(u j ,v j ))

[0241] Where, p j Indicates the position of a three-dimensional point; s j t represents a semantic description vector; j Indicates the keyframe time to which this point belongs; (u j ,vj () represents the image pixel coordinates corresponding to that point; finally, the query interface is modeled:

[0242]

[0243] Among them, h input It is the semantic vector after natural language conversion; τ is the set semantic matching threshold; the returned set is the set of spatial points whose semantic matching degree is greater than the threshold.

[0244] This embodiment integrates an IMU-assisted deep SLAM method with language-visual multimodal perception. First, it introduces semantic embedding from a large language model to perform cosine similarity filtering on keyframes, and only performs geometric verification on frames with a similarity higher than 0.75 and sufficient time intervals. This significantly reduces the number of loop closure candidates and lowers the risk of mismatches, thus maintaining more stable loop closure detection and global consistency in large-scale or long-term operation. Furthermore, in the point cloud registration stage, the IMU pre-integration results are used as the initial pose, and the pre-integration is corrected by the residuals after registration. This effectively improves the ICP convergence speed and compresses local pose errors, improving the efficiency and accuracy of dense point cloud alignment and ensuring the smoothness of trajectory estimation during continuous mapping. Moreover, by synchronously writing the optimized keyframe poses, corresponding dense point clouds, and semantic vectors into a structured semantic database, the system constructs a queryable 3D semantic map. This enables downstream tasks to directly achieve target retrieval and path planning based on natural language, expanding the application capabilities of the SLAM system in semantic navigation and human-computer interaction scenarios.

[0245] Corresponding to the IMU-assisted deep SLAM method fused with language-visual multimodal perception disclosed in the above embodiments, this invention also discloses an IMU-assisted deep SLAM system fused with language-visual multimodal perception, which specifically includes:

[0246] Initial calibration and semantic map initialization module: used to perform initial bias estimation of the IMU sensor and complete the extrinsic parameter calibration of the depth camera and IMU; construct initial point cloud through multi-frame depth map to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embedding and initialize semantic map;

[0247] Pre-integration prediction and keyframe determination module: It is used to predict the robot's pose change between two frames by using the angular velocity and acceleration data output by the IMU at high frequency and through pre-integration technology. At the same time, it combines a time synchronization mechanism to maintain data alignment between the IMU and the depth camera. When the cumulative displacement or rotation exceeds the set threshold, it automatically determines the current frame as a keyframe and records the corresponding timestamp.

[0248] Dense point cloud reconstruction and relative pose estimation module: After a keyframe is triggered, it reconstructs a dense point cloud using RGB-D images acquired by a depth camera, estimates the relative pose between the current frame and the previous keyframe through point cloud registration, and introduces motion priors provided by IMU pre-integration as initial values ​​for registration to improve registration convergence efficiency and accuracy, and performs constraint correction on the registration residuals.

[0249] Keyframe semantic embedding extraction module: This module is used to input keyframe images into the pre-trained and frozen parameters of the large language model image encoder CLIP, extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop closure detection and map annotation.

[0250] The loop closure candidate screening and verification module is used to calculate the similarity between the semantic embedding of a new keyframe and the semantic vector of the historical keyframe after the new keyframe is generated, and to screen out potential loop closure candidate frames. Then, combined with the geometric consistency test, point cloud registration is performed on the candidate frames to verify the loop closure relationship. If the match is successful, semantic-geometric joint constraint edges are constructed to improve the relocation capability and loop closure detection stability of the system in large-scale motion or long-term operation.

[0251] Semantic-guided loop closure detection module: It is used to calculate the similarity between semantic embedding vectors and perform loop closure detection in a semantic-guided manner. It quickly filters potential closed loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, it combines deep point cloud for geometric verification to improve the relocation robustness and long-term consistency of the system.

[0252] Semantic 3D Map Incremental Update Module: This module uses the optimized keyframe poses to incrementally update the global map, register and fuse the dense point cloud reconstructed from the corresponding depth map, and overlay semantic embedding information to construct a 3D map with semantic labels, thus achieving simultaneous maintenance of high-precision map construction and semantic awareness.

[0253] It should be noted that for a detailed description of the IMU-assisted deep SLAM system with fused language-visual multimodal perception provided in the embodiments of the present invention, please refer to the relevant description of the IMU-assisted deep SLAM method with fused language-visual multimodal perception provided in the embodiments of this application, which will not be repeated here.

[0254] This embodiment of the IMU-assisted deep SLAM system, which integrates language and vision multimodal perception, adopts a modular architecture that combines geometry, inertia, and semantics. It coordinates low-level motion priors, dense point cloud registration, and high-level semantic embedding into a unified graph optimization framework. The initial calibration module ensures consistency of extrinsic parameters and zero-bias reliability of multiple sensors, laying the foundation for accuracy in subsequent estimations. The pre-integration and keyframe determination module filters keyframes in real time with millisecond-level time synchronization, balancing computational load and trajectory integrity. The dense point cloud reconstruction module significantly improves ICP convergence efficiency and dynamically constrains registration residuals by leveraging IMU priors, suppressing local error drift. The four semantically related sub-modules (embedding extraction, loop closure candidate, semantic-guided detection, and 3D map incremental update) are based on the large language model CLIP, coupling fast cosine similarity retrieval with geometric consistency verification to achieve high-confidence loop closure detection and labeled map generation. Compared to traditional solutions that rely solely on geometry or inertia, this system demonstrates higher relocation robustness and semantic interaction capabilities in dynamic lighting, texture-sparse, or long-running scenarios. It also provides direct interfaces for downstream applications such as natural language navigation and object retrieval, balancing accuracy, real-time performance, and scalability overall.

[0255] In addition, embodiments of this application also provide an electronic device for IMU-assisted deep SLAM that integrates language-visual multimodal perception. Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. The electronic device may include: a processor 301, a communications interface 302, a memory 303, and a bus 304. The processor 301, communications interface 302, and memory 303 communicate with each other via the bus 304. The processor 301 can call a computer program stored in the memory 303 and executable on the processor 301 to perform the IMU-assisted deep SLAM method with fused language-vision multimodal perception provided in the above embodiment.

[0256] Furthermore, the logical instructions in the aforementioned memory 303 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application embodiment, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0257] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for IMU-assisted deep SLAM that integrates language and visual multimodal perception, characterized in that, include: S1. Initial calibration and semantic map initialization: Perform initial bias estimation for the IMU sensor and complete the extrinsic parameter calibration of the depth camera and IMU; An initial point cloud is constructed using multi-frame depth maps to complete the initial pose estimation; semantic features of keyframes are extracted and input into the language model to generate semantic embeddings and initialize the semantic map; S2. Pre-integration prediction and keyframe determination: Using the angular velocity and acceleration data output by the IMU at high frequency, the robot's pose change between two frames is predicted by pre-integration technology, while the time synchronization mechanism is combined to maintain data alignment between the IMU and the depth camera. When the cumulative displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a key frame, and the corresponding timestamp is recorded. S3. Dense point cloud reconstruction and relative pose estimation: After the keyframe is triggered, the dense point cloud is reconstructed using the RGB-D image acquired by the depth camera, and the relative pose between the current frame and the previous keyframe is estimated by the point cloud registration method. Meanwhile, the motion priors provided by IMU pre-integration are introduced as initial values ​​for registration to improve registration convergence efficiency and accuracy, and to constrain and correct the registration residuals. S4. Keyframe semantic embedding extraction: Input the keyframe image into the pre-trained and frozen parameter large language model image encoder CLIP to extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop closure detection and map annotation. S5. Loop closure candidate screening and verification: After the new keyframe is generated, its semantic embedding is similar to the semantic vector of the historical keyframe to screen out potential loop closure candidate frames; then, combined with geometric consistency test, point cloud registration is performed on the candidate frames to verify the loop closure relationship. If the match is successful, semantic-geometric joint constraint edges are constructed to improve the system's relocation capability and loop closure detection stability in large-scale motion or long-term operation. S6. Semantic-guided loop closure detection: Based on the similarity calculation between semantic embedding vectors, a semantic-guided approach is used to detect loop closures, quickly filtering potential closed-loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, geometric verification is performed by combining deep point clouds to improve the robustness of relocation and the long-term consistency of the system. S7. Semantic 3D Map Incremental Update: The global map is incrementally updated using the optimized keyframe poses. The dense point cloud reconstructed from the corresponding depth map is registered and fused. Semantic embedding information is overlaid to construct a 3D map with semantic labels, achieving synchronous maintenance of high-precision map construction and semantic awareness.

2. The IMU-assisted deep SLAM method with fused language-visual multimodal perception as described in claim 1, characterized in that, In step S1, the initial zero bias is calculated using the acceleration and angular velocity data collected by the IMU during the stationary phase, and the bias calibration of the inertial unit is completed; the visual encoder module of the language model is initialized to provide a consistent image input format for subsequent semantic feature extraction.

3. The IMU-assisted deep SLAM method with fused language-visual multimodal perception as described in claim 1, characterized in that, In S2, the system receives angular velocity and acceleration data output by the IMU in real time, estimates the pose change between key frames through a pre-integration method, sets dual thresholds for displacement and rotation, and automatically selects the current frame as a key frame when the cumulative motion exceeds the threshold, providing a high-frequency motion prior and key frame triggering mechanism for subsequent depth point cloud registration and semantic processing.

4. The IMU-assisted deep SLAM method with fused language-visual multimodal perception according to claim 1, characterized in that, In step S3, the RGB-D image corresponding to the key frame is acquired, a dense point cloud is generated and registered with the point cloud of the previous key frame, and the relative pose is estimated using ICP or visual feature matching methods. The initial registration value is provided by the IMU pre-integration result to improve the registration convergence efficiency and local pose estimation accuracy, and to complete the geometric alignment and local trajectory optimization of the current frame.

5. The IMU-assisted deep SLAM method with fused language-visual multimodal perception according to claim 1, characterized in that, S4 specifically includes: S401. Embed image preprocessing and visual patch construction; let the image acquired at time t be... Divide it into The size is The image blocks, each flattened out, are denoted as: in, This represents the height and width of the image; the number 3 channels correspond to the RGB three color channels. Original image The Middle Image blocks; This is a matrix flattening operation; It is the flattened image patch vector; the image patch is mapped to the feature space through linear projection and then positional encoding is added to obtain the initial embedding: in, Representing the Feature representation of an image patch; The projection matrix representing the patch embedding; Represents the bias term; Representing the The position encoding corresponding to each patch; It is a visual feature dimension; S402. Perform Transformer encoding to obtain global visual features of the image. First, construct the initial sequence: in, For the initial image sequence, Number of image patches; Next, for the number of layers... Encoding using multi-layered Transformers: The output sequence of the Transformer is denoted as... , For the first The Transformer output sequence of the layer; This is a multi-head self-attention mechanism; It is a feedforward fully connected subnetwork; It is a layer normalization operation; This is the total number of Transformer layers; Finally, the features of all image patches are averaged and pooled to obtain a global visual description: in, For a moment The global visual feature vector of the image; It is the first The output features of each image patch in the last Transformer layer; S403. Generate semantic embeddings through the semantic projection module, and incorporate visual representations. Input semantic projection network, mapped to semantic space: And perform normalization: in, and These are all network parameters for semantic projection; Represents a non-linear activation function; Represents the corresponding bias; Represents output bias; S404. Visual feature alignment and fusion with semantic representation, combining visual representation... Projected into the semantic space and fused with semantic embedding: in, The vector after alignment with visual features; Alignment is represented by an affine transformation matrix; Representation of bias terms; Re-fusion feature representation: in, It is a vector obtained after multimodal fusion; It is a semantic space embedding vector; This represents the fusion weight coefficient, which controls the proportion of semantic and visual weights. S405. Construct a keyframe index set. If the current frame meets the keyframe determination criteria, add its timestamp to the keyframe set. in, It is a set of keyframe time indexes; This is the current keyframe time; S406. Construct a complete semantic database and write the fused features into the keyframe semantic database, that is, to form a data record by combining the timestamp of the current keyframe with the fused representation: in, It is a semantic database of keyframes, with the database dimension being... ; This is the number of keyframes currently recorded by the system; This indicates the acquisition time of the frame on the timeline; Representing the Each keyframe at time The transpose of the extracted semantic-visual fusion feature vector .

6. The IMU-assisted deep SLAM method with fused language-visual multimodal perception according to claim 1, characterized in that, S5 specifically includes: S501. Normalize the semantic vectors of the current frame and historical keyframes. To perform semantic similarity matching, the semantic vectors of the current frame and the features of historical frames are first normalized to unify the vector scale. in, It is the multimodal fusion feature of the current frame; It is a multimodal fusion feature of historical frames; It is a vector The modulus length; It is a unit-normalized vector; S502. Calculate and rank the semantic similarity scores. Calculate the semantic cosine similarity using the dot product of normalized vectors to construct candidate score vectors. in, Represents the current frame With historical frames The semantic similarity score is then used to construct a vector from all similarities: Finally, Sort in descending order, retaining the first similarity score. The frames are used as the initial selection set: in, It is an index sequence of the score vectors arranged in descending order of value; Indicates the first Highly similar historical keyframes; This is the number of candidates to retain; choose 20-40. This indicates that the current frame ranks first in the semantic similarity score. A collection of historical keyframes; S503. Perform threshold filtering and inter-frame distance limiting to obtain the final candidate set; remove frames with low similarity or too close in time from the sorting results: in, This represents the final set of candidate loop closure frames to be verified. Semantic similarity threshold; Minimum inter-frame interval; S504. Perform point cloud registration, estimate relative pose, and verify geometric consistency; select each... Construct the initial relative pose: in, Represents the relative pose change at the current moment; This represents the relative pose transformation at the previous moment; then, the point-to-plane ICP method is used to minimize the geometric error: in, This represents the estimated relative transformation from the current frame to historical frames; Indicates the first point in the current frame point cloud One point; Indicates the relationship between historical frames and Matching points; Point The unit normal vector of the local surface; It is the number of point pairs participating in the matching optimization; It is a rigid body transformation group in three-dimensional space; Finally, define the mean of the matching error: in, This represents the matching error, used in conjunction with... The comparison is used to check whether a geometrically valid closure is formed; S505. Calculate the loop closure confidence score and dynamically set the loop closure factor weights, then integrate semantic similarity and geometric residuals to construct a confidence index: in, Represents the confidence level of loop closure matching; It is a scaling factor that controls the influence of matching residuals. This value is used for weighting the information matrix in subsequent graph optimization. S506. In graph optimization, the loop-closing residual term and joint factor, geometric residual and semantic residual are defined as follows: in, This represents the geometric residual, which originates from the Lie group error. Represents semantic residuals; This refers to semantic similarity scoring; finally, the overall loop closure loss function can be obtained as follows: in, This indicates the contribution of the lapsing factor to the overall graph optimization; The information matrix for the geometric error term; Represents the confidence level of loop closure matching; This represents the basic coefficient of semantic loss.

7. The IMU-assisted deep SLAM method with fused language-visual multimodal perception according to claim 1, characterized in that, In step S6, based on the constructed odometry factor and closure factor, a nonlinear optimization method is used to globally adjust the pose of the key frame. During the optimization process, geometric residuals and semantic residuals are considered together, and the influence of each factor is controlled by weighting the information matrix to improve the consistency and accuracy of trajectory estimation and map construction.

8. The IMU-assisted deep SLAM method with fused language-visual multimodal perception according to claim 1, characterized in that, Specifically, S7 includes: S701. Dense point cloud reconstructed from keyframe pose and depth map after fusion and optimization. The optimized keyframe pose is represented as follows: in, This represents the optimized keyframe pose. Represents the optimized first Frame pose; then keyframe images of each frame. Convert the data into the corresponding depth map and intrinsic parameter matrix, and convert each pixel into a point cloud: in, These are the corresponding three-dimensional spatial points; This represents the depth value at that pixel. It is the intrinsic parameter matrix of the depth camera; Representing pixels; finally, aggregating all frames to generate a point cloud map: in, An abstract symbol for a point cloud map, representing a set of points; S702. Map the semantic features of each frame to 3D points to construct a semantic annotation map; set keyframes. The semantic vector is Then, the semantic vector of each frame of the image is used to generate the point cloud as follows: Reconstruct the set of point semantic pairs: in, for It is an abbreviation for a point in three-dimensional space; The abbreviation for represents the semantic description vector of the keyframe to which the point belongs; It is a semantically enhanced 3D map; S703, semantic target matching modeling becomes more complex; a global semantic matching distribution is constructed while calculating single-point matching, and a softmax weighting mechanism is added for 3D spatial localization: the target semantic vector is set as... Its normalization is Calculate the cosine similarity for all points on the map: in, Cosine similarity; Normalize the cosine similarity; then construct the softmax weight distribution: in, Assign parameters to the weights; The gradient coefficient controls the steepness of the similarity distribution; finally, the semantic similarity of each semantic target point within each range is calculated, and the weighted average of the positions of all points is taken to obtain a spatial centroid position. in, That is, the location of the centroid of the point cloud space; The number of semantic points; For the first one on the map The position coordinates of the semantic point; S704. Establish semantic database structure and query interface modeling; model semantic point clouds. Structured into a queryable database table Each record can be represented as: in, Indicates the position of a three-dimensional point; Represents a semantic description vector; Indicates the keyframe time to which this point belongs; This refers to the image pixel coordinates corresponding to that point; finally, the query interface is modeled: in, It is a semantic vector converted from natural language; It is the set semantic matching threshold; the returned set is the set of all spatial points whose semantic matching degree is greater than the threshold.

9. An IMU-assisted deep SLAM system integrating language and visual multimodal perception, characterized in that, The system includes: Initial calibration and semantic map initialization module: used to perform initial bias estimation of the IMU sensor and complete the extrinsic parameter calibration of the depth camera and IMU; construct initial point cloud through multi-frame depth map to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embedding and initialize semantic map; Pre-integration prediction and keyframe determination module: It is used to predict the robot's pose change between two frames by using the angular velocity and acceleration data output by the IMU at high frequency and through pre-integration technology. At the same time, it combines a time synchronization mechanism to maintain data alignment between the IMU and the depth camera. When the cumulative displacement or rotation exceeds the set threshold, it automatically determines the current frame as a keyframe and records the corresponding timestamp. Dense point cloud reconstruction and relative pose estimation module: After a keyframe is triggered, it reconstructs a dense point cloud using RGB-D images acquired by a depth camera, estimates the relative pose between the current frame and the previous keyframe through point cloud registration, and introduces motion priors provided by IMU pre-integration as initial values ​​for registration to improve registration convergence efficiency and accuracy, and performs constraint correction on the registration residuals. Keyframe semantic embedding extraction module: This module is used to input keyframe images into the pre-trained and frozen parameters of the large language model image encoder CLIP, extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop closure detection and map annotation. The loop closure candidate screening and verification module is used to calculate the similarity between the semantic embedding of a new keyframe and the semantic vector of the historical keyframe after the new keyframe is generated, and to screen out potential loop closure candidate frames. Then, combined with the geometric consistency test, point cloud registration is performed on the candidate frames to verify the loop closure relationship. If the match is successful, semantic-geometric joint constraint edges are constructed to improve the relocation capability and loop closure detection stability of the system in large-scale motion or long-term operation. Semantic-guided loop closure detection module: It is used to calculate the similarity between semantic embedding vectors and perform loop closure detection in a semantic-guided manner. It quickly filters potential closed loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, it combines deep point cloud for geometric verification to improve the relocation robustness and long-term consistency of the system. Semantic 3D Map Incremental Update Module: This module uses the optimized keyframe poses to incrementally update the global map, register and fuse the dense point cloud reconstructed from the corresponding depth map, and overlay semantic embedding information to construct a 3D map with semantic labels, thus achieving simultaneous maintenance of high-precision map construction and semantic awareness.

10. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of the IMU-assisted deep SLAM method for fused language-visual multimodal perception as described in any one of claims 1 to 8.

Images