IMU (Inertial Measurement Unit)-assisted deep SLAM (Simultaneous Localization and Mapping) method and system fusing language-vision multi-mode perception

Abstract

The invention provides an IMU (inertial measurement unit)-assisted depth SLAM (simultaneous localization and mapping) method and system fusing language-vision multi-mode perception, and the system comprises functional modules such as initial calibration and semantic map initialization, pre-integration prediction and key frame judgment, dense point cloud reconstruction and relative pose estimation, semantic embedding extraction, semantic guidance loopback detection and semantic three-dimensional map incremental updating. IMU motion priori, depth geometric constraint and language model semantic factors are subjected to combined modeling through a graph optimization framework, and high-precision positioning and labeled map construction in a complex dynamic environment are achieved. Compared with the prior art which only depends on geometric or inertial information, the method has the advantages that the loop-back mismatching rate is reduced, the closed-loop convergence efficiency and the long-time relocation robustness are improved, and a semantic interface is provided for upper-layer tasks such as natural language navigation and target retrieval. The method can be widely applied to the fields of service robots, security inspection, intelligent driving, post-disaster search and rescue and the like.

Description

Technical Field

[0001] The present invention relates to the field of robot autonomous positioning and map construction, and in particular to an IMU-assisted deep SLAM method and system integrating language-vision multimodal perception. Background Art

[0002] As intelligent robots continue to expand into application scenarios such as service robotics, security patrol, home assistance, post-disaster rescue, and complex indoor navigation, the demand for highly robust positioning and semantic understanding and mapping in unstructured, dynamically changing environments is increasing. Traditional SLAM systems primarily rely on single or a small number of modal sensors, such as vision, lidar, or inertial measurement. While achieving some success, their performance in highly variable, dynamic, and semantically complex environments remains limited.

[0003] Although current multi-sensor SLAM methods have improved positioning accuracy and map density, they still face multiple challenges. Although the inertial measurement unit (IMU) has the ability to track high-frequency motion, it is prone to long-term errors due to integral drift; depth cameras can obtain dense geometric information of the scene, but lack stability in scenes such as lighting changes and dynamic occlusion; in addition, existing SLAM systems rely on traditional vision algorithms for dynamic target detection, semantic segmentation and scene understanding, and lack the modeling and utilization of high-level semantic information of the environment, which limits their relocalization, loop detection and intelligent planning capabilities in complex scenes.

[0004] It can be seen that the existing mainstream solutions have deficiencies in structure or performance, and it is still difficult to take into account the comprehensive requirements of high precision, strong robustness and semantic perception.

[0005] In recent years, large language models (LLMs) have demonstrated powerful capabilities in joint image-text modeling and natural language understanding, with the potential to extract high-level semantic information, perform contextual reasoning, and express spatial relationships. Integrating language models into SLAM systems, combining the dense geometric perception provided by depth cameras with the short-term, high-frequency motion estimation provided by IMUs, to achieve a "geometry-semantics-language" fusion of localization and mapping, has become a key research topic in the field of robotic perception and cognitive navigation.

[0006] Therefore, a new technical solution that can achieve high-precision simultaneous localization and mapping in dynamic and semantically changing environments remains to be provided. Summary of the Invention

[0007] The present application provides an IMU-assisted deep SLAM method and system that integrates language-vision multimodal perception to overcome the limitations of existing SLAM methods and systems in dynamic environments, semantic understanding, and relocalization robustness.

[0008] According to a first aspect, the present invention provides an IMU-assisted deep SLAM method integrating language-visual multimodal perception, comprising:

[0009] S1. Initial calibration and semantic map initialization: Perform initial bias estimation on the IMU sensor and complete external parameter calibration of the depth camera and IMU; construct the initial point cloud from multi-frame depth maps to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embedding and initialize the semantic map;

[0010] S2. Pre-integration prediction and keyframe determination: Utilizing the angular velocity and acceleration data output by the IMU at high frequency, the robot's posture change between two frames is predicted through pre-integration technology. At the same time, a time synchronization mechanism is used to maintain data alignment between the IMU and the depth camera. When the accumulated displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a keyframe and the corresponding timestamp is recorded.

[0011] S3, Dense Point Cloud Reconstruction and Relative Pose Estimation: After a keyframe is triggered, a dense point cloud is reconstructed using the RGB-D image captured by the depth camera. The relative pose between the current frame and the previous keyframe is estimated using a point cloud registration method. The motion prior provided by the IMU pre-integration is introduced as the initial registration value to improve the registration convergence efficiency and accuracy, and constrained correction is performed on the registration residual.

[0012] S4. Keyframe semantic embedding extraction: The keyframe image is input into the pre-trained and parameter-frozen large language model image encoder CLIP to extract the semantic embedding vector, which is used to describe the semantic features of the current frame and participate in subsequent loop detection and map annotation.

[0013] S5. Loop Candidate Screening and Verification: After a new keyframe is generated, its semantic embedding is similarly calculated with the semantic vectors of previous keyframes to screen out potential loop candidates. Then, combined with geometric consistency checks, point cloud registration is performed on the candidate frames to verify the loop relationship. If the match is successful, a semantic-geometric joint constraint edge is constructed to improve the system's relocalization capability and loop closure detection stability during large-scale motion or long-term operation.

[0014] S6, Semantic-guided Loop Detection: Based on the similarity calculation between semantic embedding vectors, a semantic-guided approach is used for loop detection to quickly screen potential closed-loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, geometric verification is performed in conjunction with the depth point cloud to improve relocalization robustness and long-term system consistency.

[0015] S7. Incremental update of semantic 3D maps: The global map is incrementally updated using the optimized keyframe poses. The dense point cloud reconstructed from the corresponding depth map is registered and fused. At the same time, semantic embedding information is superimposed to construct a 3D map with semantic labels, achieving simultaneous maintenance of high-precision map construction and semantic perception.

[0016] Furthermore, in S1, the acceleration and angular velocity data collected by the IMU during the stationary phase are used to calculate its initial zero bias to complete the bias calibration of the inertial unit; the visual encoder module of the language model is initialized to provide a consistent image input format for subsequent semantic feature extraction.

[0017] Furthermore, in S2, the system receives the angular velocity and acceleration data output by the IMU in real time, and estimates the pose change between key frames through the pre-integration method; sets dual thresholds for displacement and rotation, and automatically selects the current frame as the key frame when the cumulative motion exceeds the threshold, providing high-frequency motion prior and key frame triggering mechanism for subsequent deep point cloud registration and semantic processing.

[0018] Furthermore, in S3, the RGB-D image corresponding to the key frame is obtained, a dense point cloud is generated and aligned with the point cloud of the previous key frame, and the relative pose is estimated using ICP or visual feature matching methods; the initial value of the alignment is provided by the IMU pre-integration result to improve the alignment convergence efficiency and local pose estimation accuracy, and complete the geometric alignment and local trajectory optimization of the current frame.

[0019] Furthermore, the S4 specifically includes:

[0020] S401, embedding image preprocessing and visual patch construction; assuming that the image collected at time t is Divide it into N image blocks of size P×P (patches), and each image block is flattened and recorded as:

[0021]

[0022] Among them, H and W represent the height and width of the image, and the number of channels 3 corresponds to the RGB three color channels; Representative original image I t The i-th image block in ; vec() is the flattening operation of the matrix; p i is the flattened image patch vector; the image patch is mapped to the feature space by linear projection and then positionally encoded to obtain the initial embedding:

[0023]

[0024] in, Represents the feature representation of the i-th image block; W e Represents the projection matrix of the patch embedding; represents the bias term; Represents the position code corresponding to the i-th patch; d v is the visual feature dimension;

[0025] S402, perform Transformer encoding to obtain the global visual features of the image. First, construct the initial sequence:

[0026]

[0027] in, is the initial image sequence, N is the number of image blocks;

[0028] Next, encode the multi-layer Transformer with a layer number of l:

[0029]

[0030] Among them, the Transformer output sequence is recorded as is the output sequence of the Transformer at layer l; MSA() is the multi-head self-attention mechanism; MLP() is the feed-forward fully connected subnetwork; LayerNorm() is the layer normalization operation; l is the total number of Transformer layers;

[0031] Finally, the features of all image blocks are averaged and pooled to obtain a global visual description:

[0032]

[0033] Among them, v t is the global visual feature vector of the image at time t; is the output feature of the i-th image block in the last layer of Transformer;

[0034] S403, generate semantic embedding through semantic projection module, and transform the visual representation v t Input semantic projection network, mapped to semantic space:

[0035]

[0036] And perform normalization:

[0037]

[0038] in,

[0039] W s (1) and W s (2)These are all network parameters for semantic projection; σ() represents a nonlinear activation function; represents the corresponding bias; Represents output bias;

[0040] S404, visual features are aligned and semantic representations are integrated to form the visual representation v t Project to semantic space and fuse with semantic embedding:

[0041]

[0042] Among them, v′ t Align the posterior vector for visual features; Represents the affine transformation matrix for alignment; Represents the bias term; the fusion feature representation is:

[0043] h t =λ·f t +(1-λ)·v′ t ,λ∈[0,1]

[0044] Among them, h t is the vector after multimodal fusion; f t is the semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantics and vision;

[0045] S405: Construct a key frame index set. If the current frame meets the key frame determination condition, add its timestamp to the key frame set:

[0046]

[0047] in, is a set of key frame time indexes; t is the current key frame time;

[0048] S406: Build a complete semantic database and write the fusion features into the key frame semantic database, that is, the timestamp of the current key frame and the fusion representation form a data record:

[0049]

[0050] in, It is the semantic database of key frames, and the database dimension is K is the number of key frames currently recorded by the system; t i represents the acquisition time of the frame on the time axis; h represents the i-th key frame at time t i The transpose of the extracted semantic-visual fusion feature vector.

[0051] Furthermore, the S5 specifically includes:

[0052] S501: Normalize the semantic vectors of the current frame and the historical key frames. To perform semantic similarity matching, first normalize the semantic vector of the current frame and the features of the historical frames to unify the vector scale:

[0053]

[0054] Among them, h t is the multimodal fusion feature of the current frame; It is the multimodal fusion feature of the historical frame; is a vector Length of the module; is a unit normalized vector;

[0055] S502: Calculate semantic similarity scores and sort them, calculate semantic cosine similarity by dot product of normalized vectors, and construct candidate score vectors:

[0056]

[0057] in, Represents the current frame t and the historical frame t i The semantic similarity score of , and then all similarities are constructed into a vector:

[0058]

[0059] Finally, t Sort in descending order, retaining the top N similarities c The frames are used as the primary set:

[0060]

[0061] Among them, π(t) is the index sequence after sorting the score vector in descending order; represents the jth historical key frame with high similarity; N c C is the number of candidates to be retained, usually 20-40; top Indicates that the current frame ranks in the top N in the semantic similarity score c A set of historical key frames;

[0062] S503: Perform threshold screening and inter-frame distance restriction to obtain the final candidate set; further eliminate frames with low similarity or too close in time from the sorting results:

[0063]

[0064]

[0065] Among them, C loop represents the final set of loop candidate frames to be verified; θ sSemantic similarity threshold; Δt min Minimum interframe interval;

[0066] S504, perform point cloud registration, estimate relative pose and verify geometric consistency; select each t i ∈C loop Construct the initial relative pose:

[0067]

[0068] Among them, T t Represents the relative posture transformation at the current moment; Represents the relative pose transformation at the previous moment; then, the point-to-plane ICP method is used to minimize the geometric error:

[0069]

[0070] in, represents the estimated relative transformation from the current frame to the historical frame; Represents the kth point in the point cloud of the current frame; Indicates the historical frame and Matched points; Indicates a point The unit normal vector of the local surface; M is the number of point pairs involved in the matching optimization; SE(3) is the rigid body transformation group in three-dimensional space;

[0071] Finally, define the mean matching error:

[0072]

[0073] Among them, ∈ fit It represents the matching error, which is used to represent the matching error and ∈ geo Compare to check whether a geometrically valid loop is formed;

[0074] S505: Calculate loop confidence and dynamically set loop factor weights, integrating semantic similarity and geometric residual to construct confidence indicators:

[0075]

[0076] in, represents the loop matching confidence; σ is the scaling factor that controls the impact of the matching residual, which is used to weight the information matrix in subsequent graph optimization;

[0077] S506. Construct the loop residual term and joint factor in graph optimization. The geometric residual and semantic residual are defined as:

[0078]

[0079] Among them, rgeo represents the geometric residual, which comes from the Lie group error; r sem represents semantic residual; That is the semantic similarity score; finally, the overall loop loss function can be obtained as:

[0080]

[0081] Among them, E loop Indicates the contribution of the loop factor to the overall graph optimization; Ω geo is the information matrix of the geometric error term; represents the loop matching confidence; w sem Represents the basic coefficient of semantic loss.

[0082] Furthermore, in S6, based on the constructed odometry factor and loop closure factor, a nonlinear optimization method is used to globally adjust the key frame pose; during the optimization process, geometric residuals and semantic residuals are jointly considered, and the influence of each factor is controlled by weighting the information matrix to improve the consistency and accuracy of trajectory estimation and map construction.

[0083] Furthermore, the S7 specifically includes:

[0084] S701, fuse the optimized key frame pose and the depth map to reconstruct the dense point cloud. The optimized key frame pose is expressed as:

[0085]

[0086] in, represents the optimized keyframe pose, Represents the pose of the i-th frame after optimization; then each key frame image Convert to the corresponding depth map and intrinsic parameter matrix, and convert each pixel into a point cloud:

[0087]

[0088] in, is the corresponding three-dimensional space point; Represents the depth value of the pixel; K -1 is the depth camera intrinsic parameter matrix; (u, v) represents the pixel point; finally, aggregate all frames to generate a point cloud map:

[0089]

[0090] in, It is an abstract representation symbol of a point cloud map, representing a point cloud set;

[0091] S702, map the semantic features of each frame to three-dimensional points to construct a semantic annotation map; set the key frame t iThe semantic vector of The semantic vector of each frame image is assigned to the generated point cloud as a whole:

[0092]

[0093] Then construct the point semantic pair set:

[0094]

[0095] Among them, P is The abbreviation of represents a three-dimensional space point; S is The abbreviation of , which represents the semantic description vector of the key frame to which the point belongs; It is a semantically enhanced three-dimensional map;

[0096] S703, semantic target matching modeling is complicated; while calculating single point matching, a global semantic matching distribution is constructed, and a softmax weighting mechanism is added for three-dimensional spatial positioning: the target semantic vector is set to h auery , which is normalized to Calculate the cosine similarity for all points on the map:

[0097]

[0098] Among them, s j is the cosine similarity; is the normalization of cosine similarity; then construct the softmax weight distribution:

[0099]

[0100] Among them, w j is the weight distribution parameter; α>0 is the gradient coefficient, which controls the steepness of the similarity distribution; finally, the semantic similarity of each semantic target point within the range is calculated, and the weighted average of the positions of all points is taken to obtain a spatial center of gravity position:

[0101]

[0102] Among them, p goal That is, the center of gravity of the point cloud space; M is the number of semantic points; p j is the position coordinate of the jth semantic point in the map;

[0103] S704, establish semantic database structure and query interface modeling; semantic point cloud Structured into a queryable database table D, each record can be represented as:

[0104] D j =(p j ,s j,t j ,(u j ,v j ))

[0105] Among them, p j Indicates the three-dimensional point position; s j represents the semantic description vector; t j Indicates the key frame time to which this point belongs; (u j ,v j ) is the pixel coordinate of the image corresponding to the point; finally, the query interface is modeled:

[0106]

[0107] Among them, h input is the semantic vector after natural language conversion; τ is the set semantic matching threshold; the returned set is the result of all spatial points whose semantic matching degree is greater than the threshold.

[0108] According to a second aspect, the present invention provides an IMU-assisted deep SLAM system integrating language-visual multimodal perception, the system comprising:

[0109] Initial calibration and semantic map initialization module: used to estimate the initial bias of the IMU sensor and complete the external parameter calibration of the depth camera and IMU; construct the initial point cloud through multi-frame depth maps to complete the initial pose estimation; extract the semantic features of key frames and input them into the language model to generate semantic embedding and initialize the semantic map;

[0110] Pre-integration prediction and keyframe determination module: This module uses the angular velocity and acceleration data output by the IMU at high frequency to predict the robot's posture change between two frames through pre-integration technology. It also uses a time synchronization mechanism to maintain data alignment between the IMU and the depth camera. When the accumulated displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a keyframe and the corresponding timestamp is recorded.

[0111] Dense point cloud reconstruction and relative pose estimation module: After a keyframe is triggered, it uses the RGB-D image captured by the depth camera to reconstruct a dense point cloud and estimates the relative pose between the current frame and the previous keyframe through point cloud registration. It also introduces the motion prior provided by IMU pre-integration as the initial registration value to improve the registration convergence efficiency and accuracy, and performs constrained correction on the registration residual.

[0112] Keyframe semantic embedding extraction module: This module is used to input keyframe images into the pre-trained and parameter-frozen large language model image encoder CLIP to extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop detection and map annotation.

[0113] Loop closure candidate screening and verification module: After a new keyframe is generated, its semantic embedding is similarly calculated with the semantic vectors of historical keyframes to screen out potential loop closure candidate frames. Then, combined with geometric consistency checks, point cloud registration is performed on the candidate frames to verify the loop relationship. If the match is successful, a semantic-geometric joint constraint edge is constructed to improve the system's relocalization capability and loop closure detection stability during large-scale motion or long-term operation.

[0114] Semantic-guided loop detection module: This module calculates similarities between semantic embedding vectors and uses a semantic-guided approach to perform loop detection. It quickly screens potential closed-loop frame pairs in historical keyframes. After a successful match, it constructs a semantic factor and adds it to the optimization graph. It also combines it with the depth point cloud for geometric verification, improving relocalization robustness and long-term system consistency.

[0115] Semantic 3D map incremental update module: used to incrementally update the global map using the optimized keyframe poses, align and fuse the dense point cloud reconstructed from the corresponding depth map, and simultaneously overlay semantic embedding information to construct a 3D map with semantic labels, achieving simultaneous maintenance of high-precision map construction and semantic perception.

[0116] According to a third aspect, the present invention provides an electronic device, comprising: a processor and a memory;

[0117] The memory is used to store one or more program instructions;

[0118] The processor is used to run one or more program instructions to execute the steps of the IMU-assisted deep SLAM method integrating language-visual multimodal perception as described in any one of claims 1 to 8.

[0119] Compared with the prior art, the present invention has the following beneficial effects:

[0120] The method and system of the present invention are based on a multimodal collaborative localization and mapping method that integrates a depth camera, an IMU, and a large language model. They jointly leverage the advantages of three types of sensor information within a graph optimization framework: the high-frequency, short-term motion prediction provided by the IMU serves as a motion prior to ensure the timeliness and continuity of sensor fusion; the depth camera outputs dense spatial geometric information, which is aligned with keyframe depth features to achieve accurate relative pose estimation; the large language model performs semantic understanding on image frames and generates embeddings for constructing semantic embedding maps. Through a language-guided keyframe selection mechanism and a dynamic target rejection strategy, the system significantly improves robustness in semantically complex and dynamically disturbed environments; and by combining language-vision bimodal matching in loop detection, the long-term relocalization capability is enhanced. Overall, the design that integrates IMU pre-integration, deep geometric matching, and semantic prior guidance significantly improves the positioning accuracy, semantic consistency, and environmental adaptability of SLAM. It has good engineering practical value and expansion potential, and can meet the needs of long-term autonomous operation of robots in real complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 is a structural diagram of the method of the present invention;

[0122] Figure 2 This is an overall structural diagram of the IMU-assisted deep SLAM method that integrates language-vision multimodal perception according to an embodiment of the present invention;

[0123] Figure 3 This is a flowchart of key frame semantic embedding and extraction according to an embodiment of the present invention;

[0124] Figure 4 This is a flowchart of loop candidate frame screening and verification according to an embodiment of the present invention;

[0125] Figure 5 This is a flowchart of semantic map construction and incremental update according to an embodiment of the present invention;

[0126] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0127] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0128] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0129] like Figure 1 As shown, the present invention proposes a multimodal synchronous positioning and mapping method that integrates language model, depth camera and IMU information, named LLM-VINS (Language-enhanced Visual-Inertial Navigation System). This method is based on multi-source perception information, and through visual-semantic joint mapping and loop detection strategies, it significantly improves the positioning stability and semantic expression ability of the system in complex dynamic environments. First, in the system initialization stage, the IMU calibration, initial point cloud generation and image segmentation tasks are completed, and the visual semantic processing module is entered. This module combines the IMU pre-integration and semantic segmentation results, and outputs the estimated pose for map update and maintenance through factor graph construction and optimization. As Figure 3 As shown in the figure, the system introduces a large language model image encoder and performs Transformer semantic encoding on the key frame image. The image is divided into multiple image blocks and mapped to the feature space. The global visual features are extracted through a multi-layer Transformer and then input into the semantic projection network to obtain a unified semantic embedding vector. Finally, the semantic information is aligned and fused with the visual information to construct a multimodal representation for key frame judgment and management. If it is determined to be a key frame, the timestamp is recorded and added to the key frame set. Figure 4As shown in , in the loop detection stage, the system compares the similarity between the semantic vector of the current frame and the semantic embedding of the historical key frames, and constructs a ranked score list. The top N candidate frames with the highest similarity are retained, and frames with insufficient time intervals are removed to form the final loop candidate set. The candidate frames are then subjected to point cloud registration and ICP evaluation. If the geometric error meets the set conditions, semantic factors and geometric factors are added to the graph optimization to form closed-loop constraint edges, thereby improving the long-term consistency and loop robustness of the system. Figure 5 As shown in the figure, the optimized keyframe poses and depth map are combined to generate a dense point cloud map. The keyframe semantics are then embedded into the point cloud to create semantic annotations. This semantic point cloud is then registered and fused with the existing map to complete a global map update. The system further determines whether the current semantic point meets the storage requirements. If so, the point's location, semantic vector, time, and image index are written to the semantic database, creating a queryable semantic map structure that supports subsequent natural language navigation and target matching.

[0130] See also Figure 1 , a visual-inertial simultaneous localization and mapping method that integrates language model semantic embedding, which is summarized as follows:

[0131] S1. System initialization, including calibration of the IMU sensor in a stationary state, and estimation of its initial bias b by collecting multiple frames of acceleration and angular velocity data a , b g ; At the same time, the image I collected by the depth camera t Convert it into a dense point cloud and use the initial registration method to estimate the initial pose T0 of the system; further complete the external parameter estimation between the IMU and the camera based on the calibration plate or structured light method, and load the language model image encoding module with frozen weights for subsequent multimodal semantic feature extraction.

[0132] In S1, the system initialization includes static estimation of the IMU zero bias, continuous acquisition of multiple frames of IMU observation data when the system is stationary, and estimation of the bias of the gyroscope and accelerometer respectively by statistical averaging, where ω t , a t They represent the angular velocity and acceleration observation values ​​at the tth moment respectively, and the unit vector of the gravity direction is according to The accelerometer bias b is obtained respectively g and gyroscope bias b g At the same time, the extrinsic parameter calibration between the depth camera and the IMU is completed. By collecting the synchronous pose pairs of the two under the known trajectory, the rotation and translation error model is constructed to estimate the extrinsic parameter transformation matrix from the camera to the IMU. In addition, in order to initialize the language model visual encoder, the image is size normalized and channel normalized, and each frame image I is converted to t Process the input tensor into a unified format Where I(u,v,c) is the original pixel value of the pixel position (u,v) in the cth channel; μ c and σ c , are the mean and standard deviation of the channel image, respectively, which are used for standardization processing to meet the input requirements of the subsequent Transformer structure and preload the language and visual encoding module with frozen parameters to ensure the stability and comparability of the subsequent semantic feature extraction results.

[0133] S2, IMU data processing and key frame selection, the system collects the high-frequency angular velocity ω of the IMU sensor in real time t and acceleration a t Data, using the pre-integration method to integrate the displacement, velocity and attitude increments to approximate Compensate for the bias term b during the integration process a , b g , set the rotation and translation threshold ∈ p ,∈ r , when the cumulative pose change exceeds a threshold, a keyframe is automatically triggered to control the computational complexity and maintain the state estimation accuracy.

[0134] In S2, the system collects the high-frequency angular velocity ω output by the IMU sensor in real time during operation. t and acceleration a t Data, the pre-integration model is used to estimate the integral of the pose change between any two time nodes, where the gyroscope bias b is subtracted from the angular velocity. g And the rotation amount is accumulated in the SO(3) group space, and the rotation increment is calculated At the same time, the corrected acceleration is combined with the current attitude estimate to calculate the velocity and displacement increments. Used to describe the relative motion trend between key frames. Where exp() is the exponential mapping from Lie algebra to Lie group; () ∧ represents the antisymmetric matrix form of so(3); R t , v t are the rotation matrix and velocity of the system at time t; b g , b a are the estimated gyroscope and accelerometer biases respectively; in order to control the number of system states and maintain global consistency, the judgment conditions for key frame triggering are set, including the displacement threshold ∈ p With the rotation angle threshold ∈ r ,When the cumulative displacement ||Δp|| and the rotation angle ∠(ΔR) between consecutive frames exceed the set threshold, the current frame is triggered to be written into the factor graph as a new key frame.

[0135] S3, point cloud registration and relative pose estimation, the RGB-D image I corresponding to the current key frame t , depth image D t , use the intrinsic parameter matrix K to back-project the image pixels into a dense three-dimensional point cloud The ICP method is used to perform point-to-plane registration with the previous keyframe point cloud. The initial registration value is provided by the IMU prediction, and the relative pose between the two frames is output. The geometric constraint factor between the keyframes is constructed and added to the back-end graph optimization module.

[0136] In S3, the system obtains the color image I corresponding to the current frame after the key frame is triggered. t With the depth image D t According to the camera internal parameter matrix K = [f x ,f y ,c x ,c y ], back-project the pixel point (u, v) into three-dimensional space to generate a dense point cloud in the camera coordinate system The coordinates of each point are given by Then, the point cloud of the previous key frame is extracted, and the IMU pre-integration result is used as the initial pose estimate. The ICP algorithm aligns the point cloud of the current frame with the previous frame to minimize the geometric residual between the two frames.

[0137] S4, extract key frame semantic embedding, segment the current frame image, and generate image block sequence p i , after linear transformation and position encoding, it forms the initial embedding Input into the Transformer structure to extract visual features v t , and then the semantic projection network outputs the normalized semantic embedding, which is combined with the visual feature f after affine transformation t Generate the final multimodal semantic vector h by proportional fusion t , and is written into the semantic database together with the timestamp t to support subsequent semantic loop recognition and semantic map annotation.

[0138] In S4, after the key frame is selected, the system inputs the current frame RGB image into the preloaded language model image encoder, first dividing the image into several image blocks in, Representative original image I t The i-th image block in ; vec() is the flattening operation of the matrix; p i is the flattened image block vector; each image block is flattened, linearly mapped and positionally encoded to generate the initial embedding vector in Represents the feature representation of the i-th image block; W erepresents the projection matrix of the patch embedding; b e represents the bias term; Represents the position code corresponding to the i-th patch; d v The visual feature dimension is 256; it is then encoded layer by layer through the deep Transformer structure to extract the semantic feature representation of each image block v t is the global visual feature vector of the image at time t; is the output feature of the i-th image block in the last layer of Transformer; all image block outputs are averaged and pooled to generate the global visual feature vector of the current frame and These are all network parameters for semantic projection; σ() represents a nonlinear activation function; represents the corresponding bias; Represents the output bias, then aligns the visual features and fuses the semantic representation; the visual representation v t Projected into the semantic space and fused with the semantic embedding to obtain v′ t =MLP align (v t )=W a ·v t +b a where v′ t W is the vector after visual feature alignment; a represents the affine transformation matrix for alignment; b a Represents the bias term; the fusion feature representation: h t =λ·f t +(1-λ)·v′ t , h t is the vector after multimodal fusion; f t is the semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantics and vision; and is finally written into the semantic database together with the timestamp t.

[0139] S5. The semantic embedding vector h between the current frame and the historical key frames t , Perform cosine similarity calculation and extract similarity score Filter candidate frames above the set threshold and perform geometric registration, and calculate the loop confidence based on the error mean Thus, the overall loss function of the loop factor is E loop .

[0140] In S5, after generating the current key frame, the system extracts its semantic embedding vector from the semantic database And the semantic vectors corresponding to all historical key frames After normalization, the cosine similarity calculation method is used to obtain the matching score of each frame Represents the current frame t and the historical frame t i The semantic similarity score is then sorted in descending order, and the top N similarities are retained. c The frames are used as the primary set: Among them, π(t) is the index sequence after sorting the score vector in descending order; represents the jth historical key frame with high similarity; N c is the number of candidates retained; C top Indicates that the current frame ranks in the top N in the semantic similarity score c The candidate frames are further filtered by threshold and time distance between frames, and only the frames with high semantic similarity and sufficiently distant time are retained: θ s Semantic similarity threshold; Δt min Minimum inter-frame interval; perform point cloud registration to estimate its relative pose transformation and then use the point-to-plane ICP method to optimize the registration to minimize the geometric error Among them represents the estimated relative transformation from the current frame to the historical frame; Represents the kth point in the point cloud of the current frame; Indicates the history frame and Matched points; Indicates a point The unit normal vector of the local face; M is the number of point pairs involved in the matching optimization; finally, the joint factor residual term is constructed and added to the graph optimization structure;

[0141] S6. The IMU pre-integration factor, point cloud registration factor, and semantic loop closure factor are uniformly modeled as a nonlinear factor graph. The key frame state is adjusted using an iterative optimization method. At the same time, the Levenberg-Marquardt method is used for global solution to update the trajectory estimation result.

[0142] In S6, the system uniformly models and jointly solves the state constraints provided by multiple sensors based on the factor graph optimization framework, and converts the joint loop loss function E obtained in the previous step into loop Construct the entire factor graph, including: IMU pre-integration factor, loop constraint factor, semantic consistency factor and geometric registration factor, and construct the above factors into a nonlinear optimization objective function Where X represents the poses of all optimization variables and key frames; r i are various residual terms, including IMU, geometry, semantics, and loop closure; Ω i is the corresponding information matrix; the final output is the key frame trajectory result after global consistent optimization.

[0143] S7, according to the optimized key frame pose Project its corresponding point cloud to the world coordinate system to generate the global point cloud map The keyframe semantic vector is assigned to each point cloud to form a semantic annotation map; then a semantic query interface is constructed to support subsequent target positioning and language navigation.

[0144] In S7, after completing the global optimization of the factor graph, the system uses the updated key frame pose estimation Perform coordinate transformation on the dense point cloud of the corresponding frame, unify it to the world coordinate system, and construct a global dense point cloud map In order to support downstream semantic queries and natural language interaction, the system establishes a structured semantic database to store information such as point location, semantic features, key frame timestamps, and image pixel indexes. query When , the system calculates its cosine similarity s with all semantic vectors in the database j And perform softmax normalization with gradient coefficient α to obtain the weight distribution , the final output target semantics spatial position in the map is estimated as the weighted average point Among them, p goal That is, the center of gravity of the point cloud space; M is the number of semantic points; p j The system then writes all semantic points, keyframe timestamps, image pixel indexes and other information into a structured database to form a complete semantic map D for semantic retrieval and navigation. j =(p j ,s j ,t j ,(u j ,v j )), where p j Indicates the three-dimensional point position; s j represents the semantic description vector; t j Indicates the key frame time to which this point belongs; (u j ,v j ) is the pixel coordinate of the image corresponding to the point.

[0145] The following combination Figure 2 An embodiment of the present invention is described in detail. Figure 2 , an IMU-assisted deep SLAM method integrating language-visual multimodal perception, the method includes:

[0146] S1. Initial calibration and semantic map initialization: Perform initial bias estimation on the IMU sensor and complete external parameter calibration of the depth camera and IMU; construct the initial point cloud from multi-frame depth maps to complete initial pose estimation; extract keyframe semantic features and input them into the language model to generate semantic embedding and initialize the semantic map;

[0147] In S1, the acceleration and angular velocity data collected by the IMU during the stationary phase are used to calculate its initial zero bias to complete the bias calibration of the inertial unit; the visual encoder module of the language model is initialized to provide a consistent image input format for subsequent semantic feature extraction.

[0148] S2. Pre-integration prediction and keyframe determination: Utilizing the angular velocity and acceleration data output by the IMU at high frequency, the robot's posture change between two frames is predicted through pre-integration technology. At the same time, a time synchronization mechanism is used to maintain data alignment between the IMU and the depth camera. When the accumulated displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a keyframe and the corresponding timestamp is recorded.

[0149] In S2, the system receives the angular velocity and acceleration data output by the IMU in real time, and estimates the pose change between key frames through the pre-integration method; sets dual thresholds for displacement and rotation, and automatically selects the current frame as the key frame when the accumulated motion exceeds the threshold, providing high-frequency motion prior and key frame triggering mechanism for subsequent deep point cloud registration and semantic processing.

[0150] S3, Dense Point Cloud Reconstruction and Relative Pose Estimation: After a keyframe is triggered, a dense point cloud is reconstructed using the RGB-D image captured by the depth camera. The relative pose between the current frame and the previous keyframe is estimated using a point cloud registration method. The motion prior provided by the IMU pre-integration is introduced as the initial registration value to improve the registration convergence efficiency and accuracy, and constrained correction is performed on the registration residual.

[0151] In S3, the RGB-D image corresponding to the key frame is obtained, a dense point cloud is generated and registered with the point cloud of the previous key frame, and the relative pose is estimated using ICP or visual feature matching methods; the initial registration value is provided by the IMU pre-integration result to improve the registration convergence efficiency and local pose estimation accuracy, and complete the geometric alignment and local trajectory optimization of the current frame.

[0152] S4. Keyframe semantic embedding extraction: The keyframe image is input into the pre-trained and parameter-frozen large language model image encoder CLIP to extract the semantic embedding vector, which is used to describe the semantic features of the current frame and participate in subsequent loop detection and map annotation.

[0153] The S4 specifically includes:

[0154] S401, embedding image preprocessing and visual patch construction; assuming that the image collected at time t is Divide it into N image blocks of size P×P (patches), and each image block is flattened and recorded as:

[0155]

[0156] Among them, H and W represent the height and width of the image, and the number of channels 3 corresponds to the RGB three color channels; Representative original image I t The i-th image block in ; vec() is the flattening operation of the matrix; p i is the flattened image patch vector; the image patch is mapped to the feature space by linear projection and then positionally encoded to obtain the initial embedding:

[0157]

[0158] in, Represents the feature representation of the i-th image block; W e Represents the projection matrix of the patch embedding; represents the bias term; Represents the position code corresponding to the i-th patch; d v is the visual feature dimension;

[0159] S402, perform Transformer encoding to obtain the global visual features of the image. First, construct the initial sequence:

[0160]

[0161] in, is the initial image sequence, N is the number of image blocks;

[0162] Next, encode the multi-layer Transformer with a layer number of l:

[0163]

[0164] Among them, the Transformer output sequence is recorded as is the output sequence of the Transformer at layer l; MSA() is the multi-head self-attention mechanism; MLP() is the feed-forward fully connected subnetwork; LayerNorm() is the layer normalization operation; l is the total number of Transformer layers;

[0165] Finally, the features of all image blocks are averaged and pooled to obtain a global visual description:

[0166]

[0167] Among them, v tis the global visual feature vector of the image at time t; is the output feature of the i-th image block in the last layer of Transformer;

[0168] S403, generate semantic embedding through semantic projection module, and transform the visual representation v t Input semantic projection network, mapped to semantic space:

[0169]

[0170] And perform normalization:

[0171]

[0172] in,

[0173] Its main purpose is to convert it into a unit vector; W s (1) and W s (2) These are all network parameters for semantic projection; σ() represents a nonlinear activation function; represents the corresponding bias; Represents the output bias; the final normalization operation is mainly to make it suitable for subsequent similarity measurement;

[0174] S404, visual features are aligned and semantic representations are integrated to form the visual representation v t Project to semantic space and fuse with semantic embedding:

[0175]

[0176] Among them, v′ t Align the posterior vector for visual features; Represents the affine transformation matrix for alignment; Represents the bias term; the fusion feature representation is:

[0177] h t =λ·f t +(1-λ)·v′ t ,λ∈[0,1]

[0178] Among them, h t is the vector after multimodal fusion; f t is the semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantics and vision;

[0179] S405: Construct a key frame index set. If the current frame meets the key frame determination condition, add its timestamp to the key frame set:

[0180]

[0181] in, is a set of key frame time indexes; t is the current key frame time;

[0182] S406: Build a complete semantic database and write the fusion features into the key frame semantic database, that is, the timestamp of the current key frame and the fusion representation form a data record:

[0183]

[0184] in, It is the semantic database of key frames, and the database dimension is K is the number of key frames currently recorded by the system; t i represents the acquisition time of the frame on the time axis; h represents the i-th key frame at time t i The transpose of the extracted semantic-visual fusion feature vector.

[0185] S5. Loop Candidate Screening and Verification: After a new keyframe is generated, its semantic embedding is similarly calculated with the semantic vectors of previous keyframes to screen out potential loop candidates. Then, combined with geometric consistency checks, point cloud registration is performed on the candidate frames to verify the loop relationship. If the match is successful, a semantic-geometric joint constraint edge is constructed to improve the system's relocalization capability and loop closure detection stability during large-scale motion or long-term operation.

[0186] The S5 specifically includes:

[0187] S501: Normalize the semantic vectors of the current frame and the historical key frames. To perform semantic similarity matching, first normalize the semantic vector of the current frame and the features of the historical frames to unify the vector scale:

[0188]

[0189] Among them, h t is the multimodal fusion feature of the current frame; It is the multimodal fusion feature of the historical frame; is a vector Length of the module; is a unit normalized vector;

[0190] S502: Calculate semantic similarity scores and sort them, calculate semantic cosine similarity by dot product of normalized vectors, and construct candidate score vectors:

[0191]

[0192] in, Represents the current frame t and the historical frame ti The semantic similarity score of , and then all similarities are constructed into a vector:

[0193]

[0194] Finally, t Sort in descending order, retaining the top N similarities c The frames are used as the primary set:

[0195]

[0196] Among them, π(t) is the index sequence after sorting the score vector in descending order; represents the jth historical key frame with high similarity; N c C is the number of candidates to be retained, usually 20-40; top Indicates that the current frame ranks in the top N in the semantic similarity score c A set of historical key frames;

[0197] S503: Perform threshold screening and inter-frame distance restriction to obtain the final candidate set; further eliminate frames with low similarity or too close in time from the sorting results:

[0198]

[0199] Among them, C loop represents the final set of loop candidate frames to be verified; θ s Semantic similarity threshold, generally set to 0.75; Δt min Minimum inter-frame interval, generally set to 2s;

[0200] S504, perform point cloud registration, estimate relative pose and verify geometric consistency; select each t i ∈C loop Construct the initial relative pose:

[0201]

[0202] Among them, T t Represents the relative posture transformation at the current moment; Represents the relative pose transformation at the previous moment; then, the point-to-plane ICP method is used to minimize the geometric error:

[0203]

[0204] in, represents the estimated relative transformation from the current frame to the historical frame; Represents the kth point in the point cloud of the current frame; Indicates the history frame and Matched points; Indicates a point The unit normal vector of the local surface; M is the number of point pairs involved in the matching optimization; SE(3) is the rigid body transformation group in three-dimensional space;

[0205] Finally, define the mean matching error:

[0206]

[0207] Among them, ∈ fit It represents the matching error, which is used to represent the matching error and ∈ geo Compare to check whether a geometrically valid loop is formed;

[0208] S505: Calculate loop confidence and dynamically set loop factor weights, integrating semantic similarity and geometric residual to construct confidence indicators:

[0209]

[0210] in, represents the loop matching confidence; σ is the scaling factor that controls the impact of the matching residual, which is used to weight the information matrix in subsequent graph optimization;

[0211] S506. Construct the loop residual term and joint factor in graph optimization. The geometric residual and semantic residual are defined as:

[0212]

[0213] Among them, r geo represents the geometric residual, which comes from the Lie group error; r sem represents semantic residual; That is the semantic similarity score; finally, the overall loop loss function can be obtained as:

[0214]

[0215] Among them, E loop Indicates the contribution of the loop factor to the overall graph optimization; Ω geo is the information matrix of the geometric error term; represents the loop matching confidence; w sem Represents the basic coefficient of semantic loss.

[0216] S6, Semantic-guided Loop Detection: Based on the similarity calculation between semantic embedding vectors, a semantic-guided approach is used for loop detection to quickly screen potential closed-loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, geometric verification is performed in conjunction with the depth point cloud to improve relocalization robustness and long-term system consistency.

[0217] In S6, based on the constructed odometry factor and loop closure factor, a nonlinear optimization method is used to globally adjust the keyframe pose; the geometric residual and semantic residual are jointly considered during the optimization process, and the influence of each factor is controlled by weighting the information matrix to improve the consistency and accuracy of trajectory estimation and map construction.

[0218] S7, Semantic 3D Map Incremental Update: The global map is incrementally updated using the optimized keyframe poses. The dense point cloud reconstructed from the corresponding depth map is registered and fused, and semantic embedding information is superimposed to construct a 3D map with semantic labels. This achieves the simultaneous maintenance of high-precision map construction and semantic perception.

[0219] The S7 specifically includes:

[0220] S701, fuse the optimized key frame pose and the depth map to reconstruct the dense point cloud. The optimized key frame pose is expressed as:

[0221]

[0222] in, represents the optimized keyframe pose, Represents the pose of the i-th frame after optimization; then each key frame image Convert to the corresponding depth map and intrinsic parameter matrix, and convert each pixel into a point cloud:

[0223]

[0224] in, is the corresponding three-dimensional space point; Represents the depth value of the pixel; K -1 is the depth camera intrinsic parameter matrix; (u, v) represents the pixel point; finally, aggregate all frames to generate a point cloud map:

[0225]

[0226] in, It is an abstract representation symbol of a point cloud map, representing a point cloud set;

[0227] S702, map the semantic features of each frame to three-dimensional points to construct a semantic annotation map; set the key frame t i The semantic vector of The semantic vector of each frame image is assigned to the generated point cloud as a whole:

[0228]

[0229] Then construct the point semantic pair set:

[0230]

[0231] Among them, P is The abbreviation of represents a three-dimensional space point; S is The abbreviation of , which represents the semantic description vector of the key frame to which the point belongs; It is a semantically enhanced three-dimensional map;

[0232] S703, semantic target matching modeling is complicated; while calculating single point matching, a global semantic matching distribution is constructed, and a softmax weighting mechanism is added for three-dimensional spatial positioning: the target semantic vector is set to h query , which is normalized to Calculate the cosine similarity for all points on the map:

[0233]

[0234] Among them, s j is the cosine similarity; is the normalization of cosine similarity; then construct the softmax weight distribution:

[0235]

[0236] Among them, w j is the weight distribution parameter; α>0 is the gradient coefficient, which controls the steepness of the similarity distribution; finally, the semantic similarity of each semantic target point within the range is calculated, and the weighted average of the positions of all points is taken to obtain a spatial center of gravity position:

[0237]

[0238] Among them, p goal That is, the center of gravity of the point cloud space; M is the number of semantic points; p j is the position coordinate of the jth semantic point in the map;

[0239] S704, establish semantic database structure and query interface modeling; semantic point cloud Structured into a queryable database table D, each record can be represented as:

[0240] D j =(p j ,s j ,t j ,(u j ,v j ))

[0241] Among them, p j Indicates the three-dimensional point position; s j represents the semantic description vector; t j Indicates the key frame time to which this point belongs; (u j ,vj ) is the pixel coordinate of the image corresponding to the point; finally, the query interface is modeled:

[0242]

[0243] Among them, h input is the semantic vector after natural language conversion; τ is the set semantic matching threshold; the returned set is the result of all spatial points whose semantic matching degree is greater than the threshold.

[0244] This embodiment integrates language-visual multimodal perception into an IMU-assisted deep SLAM method. First, the semantic embedding of a large language model is introduced to perform cosine similarity screening on key frames, and geometric verification is performed only on frames with a similarity greater than 0.75 and sufficient time intervals. This significantly reduces the number of loop candidates and reduces the risk of mismatching, thereby maintaining more stable closed-loop detection and global consistency in large-scale or long-term operations. Furthermore, the IMU pre-integration result is used as the initial pose in the point cloud registration stage, and the pre-integration component is reversely corrected with the residual after registration. This can effectively improve the ICP convergence speed and compress local pose errors, improve the efficiency and accuracy of dense point cloud alignment, and ensure the smoothness of trajectory estimation during continuous mapping. Furthermore, by synchronously writing the optimized key frame poses, corresponding dense point clouds, and semantic vectors into a structured semantic database, the system constructs a queryable three-dimensional semantic map, enabling downstream tasks to directly implement target retrieval and path planning based on natural language, expanding the application capabilities of the SLAM system in semantic navigation and human-computer interaction scenarios.

[0245] Corresponding to the IMU-assisted deep SLAM method for integrating language and visual multimodal perception disclosed in the above embodiment, the embodiment of the present invention also discloses an IMU-assisted deep SLAM system for integrating language and visual multimodal perception, which specifically includes:

[0246] Initial calibration and semantic map initialization module: used to estimate the initial bias of the IMU sensor and complete the external parameter calibration of the depth camera and IMU; construct the initial point cloud through multi-frame depth maps to complete the initial pose estimation; extract the semantic features of key frames and input them into the language model to generate semantic embedding and initialize the semantic map;

[0247] Pre-integration prediction and keyframe determination module: This module uses the angular velocity and acceleration data output by the IMU at high frequency to predict the robot's posture change between two frames through pre-integration technology. It also uses a time synchronization mechanism to maintain data alignment between the IMU and the depth camera. When the accumulated displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a keyframe and the corresponding timestamp is recorded.

[0248] Dense point cloud reconstruction and relative pose estimation module: After a keyframe is triggered, it uses the RGB-D image captured by the depth camera to reconstruct a dense point cloud and estimates the relative pose between the current frame and the previous keyframe through point cloud registration. It also introduces the motion prior provided by IMU pre-integration as the initial registration value to improve the registration convergence efficiency and accuracy, and performs constrained correction on the registration residual.

[0249] Keyframe semantic embedding extraction module: This module is used to input keyframe images into the pre-trained and parameter-frozen large language model image encoder CLIP to extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop detection and map annotation.

[0250] Loop closure candidate screening and verification module: After a new keyframe is generated, its semantic embedding is similarly calculated with the semantic vectors of historical keyframes to screen out potential loop closure candidate frames. Then, combined with geometric consistency checks, point cloud registration is performed on the candidate frames to verify the loop relationship. If the match is successful, a semantic-geometric joint constraint edge is constructed to improve the system's relocalization capability and loop closure detection stability during large-scale motion or long-term operation.

[0251] Semantic-guided loop detection module: This module calculates similarities between semantic embedding vectors and uses a semantic-guided approach to perform loop detection. It quickly screens potential closed-loop frame pairs in historical keyframes. After a successful match, it constructs a semantic factor and adds it to the optimization graph. It also combines it with the depth point cloud for geometric verification, improving relocalization robustness and long-term system consistency.

[0252] Semantic 3D map incremental update module: used to incrementally update the global map using the optimized keyframe poses, align and fuse the dense point cloud reconstructed from the corresponding depth map, and simultaneously overlay semantic embedding information to construct a 3D map with semantic labels, achieving simultaneous maintenance of high-precision map construction and semantic perception.

[0253] It should be noted that for the detailed description of an IMU-assisted deep SLAM system that integrates language and visual multimodal perception provided in an embodiment of the present invention, reference can be made to the relevant description of an IMU-assisted deep SLAM method that integrates language and visual multimodal perception provided in an embodiment of the present application, which will not be repeated here.

[0254] This embodiment integrates the IMU-assisted deep SLAM system of language-visual multimodal perception through a modular architecture of "geometry-inertia-semantics", and coordinates low-level motion prior, dense point cloud registration and high-level semantic embedding into a unified graph optimization framework: the initial calibration module ensures the consistency of multi-sensor extrinsic parameters and zero-bias reliability, laying the foundation for the accuracy of subsequent estimation; the pre-integration and key frame determination module screens key frames in real time under millisecond-level time synchronization, taking into account both computational load and trajectory integrity; the dense point cloud reconstruction module significantly improves the ICP convergence efficiency with the help of IMU prior and dynamically constrains the registration residual to suppress local error drift; the four semantically related submodules (embedding extraction, loop candidate, semantically guided detection and 3D map incremental update) are based on the large language model CLIP, which couples the fast retrieval of cosine similarity with geometric consistency verification to achieve high-confidence loop closure detection and labeled map generation. Compared with traditional solutions that rely solely on geometry or inertia, the system exhibits higher relocalization robustness and semantic interaction capabilities in dynamic lighting, texture-scarce or long-running scenarios, and provides a direct interface for downstream natural language navigation, target retrieval and other applications, taking into account accuracy, real-time performance and scalability as a whole.

[0255] In addition, the embodiment of the present application also provides an electronic device for IMU-assisted deep SLAM that integrates language-visual multimodal perception. Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. The electronic device may include: a processor (processor) 301, a communication interface (CommunicationsInterface) 302, a memory (memory) 303 and a bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the bus 304. The processor 301 can call a computer program stored in the memory 303 and can be run on the processor 301 to execute the IMU-assisted deep SLAM method for integrating language and visual multimodal perception provided in the above embodiment.

[0256] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0257] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. An IMU-assisted deep SLAM method integrating language-visual multimodal perception, characterized by: include: S1. Initial calibration and semantic map initialization: perform initial bias estimation on the IMU sensor and complete external parameter calibration of the depth camera and IMU; Build the initial point cloud through multi-frame depth maps to complete the initial pose estimation; extract the semantic features of key frames and input them into the language model to generate semantic embedding and initialize the semantic map; S2, Pre-integration Prediction and Key Frame Determination: Using the angular velocity and acceleration data output by the IMU at high frequency, the robot's posture change between two frames is predicted through pre-integration technology. At the same time, a time synchronization mechanism is used to maintain data alignment between the IMU and the depth camera. When the accumulated displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a key frame and the corresponding timestamp is recorded; S3, dense point cloud reconstruction and relative pose estimation: After the key frame is triggered, the dense point cloud is reconstructed using the RGB-D image collected by the depth camera, and the relative pose between the current frame and the previous key frame is estimated through the point cloud registration method; At the same time, the motion prior provided by IMU pre-integration is introduced as the initial value of registration to improve the registration convergence efficiency and accuracy, and constrain the registration residual correction; S4. Keyframe semantic embedding extraction: The keyframe image is input into the pre-trained and parameter-frozen large language model image encoder CLIP to extract the semantic embedding vector, which is used to describe the semantic features of the current frame and participate in subsequent loop detection and map annotation. S5. Loop Candidate Screening and Verification: After a new keyframe is generated, its semantic embedding is similarly calculated with the semantic vectors of previous keyframes to screen out potential loop candidates. Then, combined with geometric consistency checks, point cloud registration is performed on the candidate frames to verify the loop relationship. If the match is successful, a semantic-geometric joint constraint edge is constructed to improve the system's relocalization capability and loop closure detection stability during large-scale motion or long-term operation. S6, Semantic-guided Loop Detection: Based on the similarity calculation between semantic embedding vectors, a semantic-guided approach is used for loop detection to quickly screen potential closed-loop frame pairs in historical keyframes. After successful matching, semantic factors are constructed and added to the optimization graph. At the same time, geometric verification is performed in conjunction with the depth point cloud to improve relocalization robustness and long-term system consistency. S7. Incremental update of semantic 3D maps: The global map is incrementally updated using the optimized keyframe poses. The dense point cloud reconstructed from the corresponding depth map is registered and fused. At the same time, semantic embedding information is superimposed to construct a 3D map with semantic labels, achieving simultaneous maintenance of high-precision map construction and semantic perception.

2. The IMU-assisted depth SLAM method of claim 1 that integrates language-visual multimodal perception, characterized in that In S1, the acceleration and angular velocity data collected by the IMU during the stationary phase are used to calculate its initial zero bias to complete the bias calibration of the inertial unit; the visual encoder module of the language model is initialized to provide a consistent image input format for subsequent semantic feature extraction.

3. The IMU-assisted depth SLAM method of the fusion language-visual multimodal perception according to claim 1, characterized in that, In S2, the system receives the angular velocity and acceleration data output by the IMU in real time, and estimates the pose change between key frames through the pre-integration method; sets dual thresholds for displacement and rotation, and automatically selects the current frame as the key frame when the accumulated motion exceeds the threshold, providing high-frequency motion prior and key frame triggering mechanism for subsequent deep point cloud registration and semantic processing.

4. The IMU-assisted depth SLAM method of the fusion language-visual multimodal perception according to claim 1, characterized in that In S3, the RGB-D image corresponding to the key frame is obtained, a dense point cloud is generated and registered with the point cloud of the previous key frame, and the relative pose is estimated using ICP or visual feature matching methods; the initial registration value is provided by the IMU pre-integration result to improve the registration convergence efficiency and local pose estimation accuracy, and complete the geometric alignment and local trajectory optimization of the current frame.

5. The IMU-assisted depth SLAM method of fusion language-visual multimodal perception according to claim 1, characterized in that, The S4 specifically includes: S401, embedding image preprocessing and visual patch construction; assuming that the image collected at time t is Divide it into N image blocks of size P×P, and each image block is flattened and recorded as: Among them, H and W represent the height and width of the image, and the number of channels 3 corresponds to the RGB three-color channels; Representative original image I t The i-th image block in ; vec() is the flattening operation of the matrix; p i is the flattened image patch vector; the image patch is mapped to the feature space by linear projection and then positionally encoded to obtain the initial embedding: in, Represents the feature representation of the i-th image block; W e Represents the projection matrix of the patch embedding; represents the bias term; Represents the position code corresponding to the i-th patch; d v is the visual feature dimension; S402, perform Transformer encoding to obtain the global visual features of the image. First, construct the initial sequence: in, is the initial image sequence, N is the number of image blocks; Next, encode the multi-layer Transformer with a layer number of l: Among them, the Transformer output sequence is recorded as is the output sequence of the Transformer at layer l; MSA() is the multi-head self-attention mechanism; MLP() is the feed-forward fully connected subnetwork; LayerNorm() is the layer normalization operation; l is the total number of Transformer layers; Finally, the features of all image blocks are averaged and pooled to obtain a global visual description: Among them, v t is the global visual feature vector of the image at time t; is the output feature of the i-th image block in the last layer of Transformer; S403, generate semantic embedding through semantic projection module, and transform the visual representation v t Input semantic projection network, mapped to semantic space: And perform normalization: in, W s (1) and W s (2) These are all network parameters for semantic projection; σ() represents a nonlinear activation function; represents the corresponding bias; Represents output bias; S404, visual feature alignment and fusion of semantic representation, visual representation v t Project to semantic space and fuse with semantic embedding: Among them, v′ t Align the posterior vector for visual features; Represents the affine transformation matrix for alignment; Represents the bias term; the fusion feature representation is: h t =λ·f t +(1-λ)·v′ t ,λ∈[0,1] Among them, h t is the vector after multimodal fusion; f t is the semantic space embedding vector; λ represents the fusion weight coefficient, which controls the proportion of semantics and vision; S405: Build a key frame index set. If the current frame meets the key frame determination criteria, add its timestamp to the key frame set: in, is a set of key frame time indexes; t is the current key frame time; S406: Build a complete semantic database and write the fusion features into the key frame semantic database, that is, the timestamp of the current key frame and the fusion representation form a data record: in, It is the semantic database of key frames, and the database dimension is K is the number of key frames currently recorded by the system; t i represents the acquisition time of the frame on the time axis; h represents the i-th key frame at time t i The transpose of the extracted semantic-visual fusion feature vector.

6. The IMU-assisted deep SLAM method integrating language-visual multimodal perception according to claim 1, characterized in that: The S5 specifically includes: S501: Normalize the semantic vectors of the current frame and the historical key frames. To perform semantic similarity matching, first normalize the semantic vector of the current frame and the features of the historical frames to unify the vector scale: Among them, h t is the multimodal fusion feature of the current frame; It is the multimodal fusion feature of the historical frame; is a vector Length of the module; is a unit normalized vector; S502: Calculate semantic similarity scores and sort them, calculate semantic cosine similarity by dot product of normalized vectors, and construct candidate score vectors: in, Represents the current frame t and the historical frame t i The semantic similarity score of , and then all similarities are constructed into a vector: Finally, t Sort in descending order, retaining the top N similarities c The frames are used as the primary set: Among them, π(t) is the index sequence after sorting the score vector in descending order; represents the jth historical key frame with high similarity; N c C is the number of candidates to be retained, usually 20-40; top Indicates that the current frame ranks in the top N in the semantic similarity score c A set of historical key frames; S503: Perform threshold screening and inter-frame distance restriction to obtain the final candidate set; further eliminate frames with low similarity or too close in time from the sorting results: Among them, C loop represents the final set of loop candidate frames to be verified; θ s Semantic similarity threshold; Δt min Minimum interframe interval; S504, perform point cloud registration, estimate relative pose and verify geometric consistency; select each t i ∈C loop Construct the initial relative pose: Among them, T t Represents the relative posture transformation at the current moment; Represents the relative pose transformation at the previous moment; then, the point-to-plane ICP method is used to minimize the geometric error: Among them, represents the estimated relative transformation from the current frame to the historical frame; Represents the kth point in the point cloud of the current frame; Indicates the history frame and Matched points; Indicates a point The unit normal vector of the local surface; M is the number of point pairs involved in the matching optimization; SE(3) is the rigid body transformation group in three-dimensional space; Finally, define the mean matching error: Among them, ∈ fit It represents the matching error, which is used to represent the matching error and ∈ geo Compare to check whether a geometrically valid loop is formed; S505: Calculate loop confidence and dynamically set loop factor weights, integrating semantic similarity and geometric residual to construct confidence indicators: in, represents the loop matching confidence; σ is the scaling factor that controls the impact of the matching residual, which is used to weight the information matrix in subsequent graph optimization; S506. Construct the loop residual term and joint factor in graph optimization. The geometric residual and semantic residual are defined as: Among them, r geo represents the geometric residual, which comes from the Lie group error; r sem represents semantic residual; That is the semantic similarity score; finally, the overall loop loss function can be obtained as: Among them, E loop Indicates the contribution of the loop factor to the overall graph optimization; Ω geo is the information matrix of the geometric error term; represents the loop matching confidence; w sem Represents the basic coefficient of semantic loss.

7. The IMU-assisted deep SLAM method integrating language-visual multimodal perception according to claim 1, characterized in that: In S6, based on the constructed odometry factor and loop closure factor, a nonlinear optimization method is used to globally adjust the keyframe pose; the geometric residual and semantic residual are jointly considered during the optimization process, and the influence of each factor is controlled by weighting the information matrix to improve the consistency and accuracy of trajectory estimation and map construction.

8. The IMU-assisted depth SLAM method integrating language-visual multimodal perception according to claim 1, characterized in that: The S7 specifically includes: S701, fuse the optimized key frame pose and the depth map to reconstruct the dense point cloud. The optimized key frame pose is expressed as: in, represents the optimized keyframe pose, Represents the optimized i-th frame pose; then each key frame image Convert to the corresponding depth map and intrinsic parameter matrix, and convert each pixel into a point cloud: in, is the corresponding three-dimensional space point; Represents the depth value of the pixel; K -1 is the depth camera intrinsic parameter matrix; (u, v) represents the pixel point; finally, aggregate all frames to generate a point cloud map: Among them, It is an abstract representation symbol of a point cloud map, representing a point cloud set; S702, map the semantic features of each frame to three-dimensional points to construct a semantic annotation map; set the key frame t i The semantic vector of The semantic vector of each frame image is assigned to the generated point cloud as a whole: Then construct the point semantic pair set: Where P is The abbreviation of represents a three-dimensional space point; S is The abbreviation of , which represents the semantic description vector of the key frame to which the point belongs; It is a semantically enhanced three-dimensional map; S703, semantic target matching modeling is complicated; while calculating single point matching, a global semantic matching distribution is constructed, and a softmax weighting mechanism is added for three-dimensional spatial positioning: the target semantic vector is set to h query , which is normalized to Calculate the cosine similarity for all points on the map: Among them, s j is the cosine similarity; is the normalization of cosine similarity; then construct the softmax weight distribution: Among them, w j is the weight distribution parameter; α>0 is the gradient coefficient, which controls the steepness of the similarity distribution; finally, the semantic similarity of each semantic target point within the range is calculated, and the weighted average of the positions of all points is taken to obtain a spatial center of gravity position: Among them, p goal That is, the center of gravity of the point cloud space; M is the number of semantic points; p j is the position coordinate of the jth semantic point in the map; S704, establish semantic database structure and query interface modeling; semantic point cloud Structured into a queryable database table D, each record can be represented as: D j =(p j ,s j ,t j ,(u j ,v j )) Among them, p j Indicates the three-dimensional point position; s j represents the semantic description vector; t j Indicates the key frame time to which this point belongs; (u j ,v j ) is the pixel coordinate of the image corresponding to the point; finally, the query interface is modeled: Among them, h input is the semantic vector after natural language conversion; τ is the set semantic matching threshold; the returned set is the result of all spatial points whose semantic matching degree is greater than the threshold.

9. An IMU-assisted deep SLAM system integrating language-visual multimodal perception, characterized by: The system comprises: Initial calibration and semantic map initialization module: used to estimate the initial bias of the IMU sensor and complete the external parameter calibration of the depth camera and IMU; construct the initial point cloud through multi-frame depth maps to complete the initial pose estimation; extract the semantic features of key frames and input them into the language model to generate semantic embedding and initialize the semantic map; Pre-integration prediction and keyframe determination module: This module uses the angular velocity and acceleration data output by the IMU at high frequency to predict the robot's posture change between two frames through pre-integration technology. It also uses a time synchronization mechanism to maintain data alignment between the IMU and the depth camera. When the accumulated displacement or rotation exceeds the set threshold, the current frame is automatically determined to be a keyframe and the corresponding timestamp is recorded. Dense point cloud reconstruction and relative pose estimation module: After a keyframe is triggered, it uses the RGB-D image captured by the depth camera to reconstruct a dense point cloud and estimates the relative pose between the current frame and the previous keyframe through point cloud registration. It also introduces the motion prior provided by IMU pre-integration as the initial registration value to improve the registration convergence efficiency and accuracy, and performs constrained correction on the registration residual. Keyframe semantic embedding extraction module: This module is used to input keyframe images into the pre-trained and parameter-frozen large language model image encoder CLIP to extract semantic embedding vectors, which are used to describe the semantic features of the current frame and participate in subsequent loop detection and map annotation. Loop closure candidate screening and verification module: After a new keyframe is generated, its semantic embedding is similarly calculated with the semantic vectors of historical keyframes to screen out potential loop closure candidate frames. Then, combined with geometric consistency checks, point cloud registration is performed on the candidate frames to verify the loop relationship. If the match is successful, a semantic-geometric joint constraint edge is constructed to improve the system's relocalization capability and loop closure detection stability during large-scale motion or long-term operation. Semantic-guided loop detection module: This module calculates similarities between semantic embedding vectors and uses a semantic-guided approach to perform loop detection. It quickly screens potential closed-loop frame pairs in historical keyframes. After a successful match, it constructs a semantic factor and adds it to the optimization graph. It also combines it with the depth point cloud for geometric verification, improving relocalization robustness and long-term system consistency. Semantic 3D map incremental update module: used to incrementally update the global map using the optimized keyframe poses, align and fuse the dense point cloud reconstructed from the corresponding depth map, and simultaneously overlay semantic embedding information to construct a 3D map with semantic labels, achieving simultaneous maintenance of high-precision map construction and semantic perception.

10. An electronic device, characterized in that: The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of the IMU-assisted deep SLAM method integrating language-visual multimodal perception as described in any one of claims 1 to 8.

Images