Visual positioning and mapping method based on contrast temporal semantic consistency

Abstract

The application discloses a visual positioning and mapping method based on contrast timing semantic consistency, and belongs to the technical field of computer vision SLAM. The method first completes instance segmentation and dynamic mask generation on an RGB-D sequence, extracts dense semantic features and reduces dimensions in batches, constructs an internal four-dimensional Gaussian radiation field, splices a spherical harmonic RGB tensor and a semantic tensor into a mixed tensor, synchronously outputs a rendering graph through single rasterization, and optimizes Gaussian parameters through coupling of photometric, depth, semantic, sliding window contrast timing multi-modal loss. After sequence processing, color is globally refined, and a camera trajectory and a semantic Gaussian map are output. The application realizes collaborative optimization of semantic and geometric gradients, reduces computing power through single rendering and eliminates gradient conflicts, suppresses semantic drift in dynamic scenes through contrast timing loss, supports a semantic quantification evaluation system, and is suitable for dynamic scenes such as service robot navigation, AR virtual-real anchoring and indoor digital twin reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision SLAM technology, specifically involving a visual localization and mapping method based on contrastive temporal semantic consistency. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology in the fields of robot autonomous navigation, augmented reality, and 3D reconstruction. Traditional sparse / semi-dense SLAM can only generate low-precision geometric maps, lacks scene semantic understanding capabilities, and is prone to trajectory drift in dynamic scenes.

[0003] However, existing technologies have the following shortcomings: they only utilize RGB luminance and depth geometric information to optimize Gaussian units, without embedding semantic information into the model, and only add semantic labels in post-processing, which cannot achieve co-optimization of semantic and geometric gradients. The consistency of semantic features of the same object across frames is poor, and semantic drift in dynamic scenes is severe. If it is necessary to render RGB images and semantic feature maps at the same time, rasterization rendering needs to be performed independently twice, resulting in redundant computational overhead. Moreover, the dual rendering paths generate conflicting gradients, which undermine the stability of optimization convergence. There is a lack of quantitative evaluation indicators specifically for the temporal stability of semantics. Relying solely on geometric rendering indicators such as trajectory error and image PSNR / SSIM cannot measure the consistency of semantic mapping across frames. Directly loading the dense semantic features of high-dimensional CLIP visual language into long video sequences will cause memory overflow, and existing technologies do not have a lightweight batch dimensionality reduction solution.

[0004] Given the shortcomings of existing four-dimensional Gaussian SLAM schemes in terms of semantic modeling integrity, cross-frame consistency, rendering computational efficiency, and semantic temporal quantization evaluation, there is an urgent need for a new method that can couple and compare temporal consistency to overcome the deficiencies of existing technologies. Summary of the Invention

[0005] Overcoming the technical shortcomings of existing 4D Gaussian SLAM, such as missing semantic modeling, cross-frame semantic drift, redundant computing power in dual rendering, and lack of semantic temporal quantification indicators, this paper provides a visual localization and mapping method based on comparative temporal semantic consistency. This method simultaneously improves the localization accuracy, rendering efficiency, and semantic temporal stability of dynamic scenes, and constructs a complete semantic quantification evaluation system.

[0006] To achieve the above objectives, this invention provides a visual localization and mapping method based on contrastive temporal semantic consistency, comprising the following steps: S1. Acquire RGB-D image sequences of indoor dynamic scenes using RGB-D sensors, extract frame-by-frame object masks using instance segmentation, complete cross-frame object ID temporal tracking by relying on mask IoU matching, and generate a unified instance mask sequence; combine RGB optical flow or mask displacement to generate dynamic region motion masks, and synchronously send the instance masks, motion masks, and original RGB images to S2 and S5. S2 receives the RGB image from S1, extracts dense semantic feature maps through CLIP vision-language encoder, and uses incremental principal component analysis to fit the projection matrix in batches to compress the high-dimensional semantic features into low-dimensional pixel-by-pixel semantic target feature maps. The reduced feature maps and feature dimension parameters are then sent to S3, S4, and S5 simultaneously. S3. Receive feature compression dimension parameters, build a 4D Gaussian radiation field model, configure three-dimensional coordinates, covariance, spherical harmonic color, opacity, deformation network and learnable semantic vector matching compression dimension for each Gaussian ellipsoid, and send the fully initialized Gaussian parameter set to S4. S4. Receive Gaussian parameters and camera line vector, concatenate the RGB tensor and Gaussian built-in semantic tensor channels obtained by spherical harmonic conversion into a hybrid tensor, input it into the differentiable Gaussian rasterizer renderer for a single rendering, and simultaneously output RGB, depth and semantic three-class rendering images to S5. S5. Gather the rendering results of S4, motion mask and object ID sequence of S1, and low-dimensional semantic truth map of S2. Calculate RGB luminance loss, dynamic mask constraint depth loss, opacity mask constraint semantic loss, and InfoNCE contrast temporal loss of sliding window + momentum encoder architecture in sequence. Weighted fusion is used to obtain the total loss. Backpropagation is used to update all Gaussian parameters synchronously and backpropagate to S4 for iterative optimization. S6. After the full sequence iteration is completed, read the optimized Gaussian parameters and object temporal semantic cache, fix the geometric and semantic parameters and only refine the spherical harmonic color and exposure; calculate the semantic cosine similarity of adjacent matching objects, solve the three evaluation indicators of STC semantic temporal consistency, Drift semantic drift ratio, and MeanSim average similarity, and finally output the camera trajectory, the semantic 4D Gaussian scene model and semantic evaluation indicators.

[0007] Preferably, in S1, the cross-frame object identity ID timing tracking and object mask matching determination satisfy the following: ; In the formula, The mask for the object in the current frame; This is the mask for historical objects in the previous frame; The intersection and union of two masks; If the maximum intersection-union ratio is greater than the preset matching threshold, the current object inherits the object ID from the previous frame; otherwise, a new object ID is assigned. At the same time, tiny invalid masks with pixel areas smaller than the preset threshold are filtered out.

[0008] Preferably, in S4, the RGB color tensor, semantic feature tensor, and blend color tensor satisfy the following calculation formula: ; In the formula, For the entire Gaussian unit RGB color matrix, ; This represents the total number of Gaussian ellipsoids. This is a function for converting spherical harmonic coefficients to RGB. This is a truncation function; The spherical harmonic color coefficient; The line-of-sight vector; ; In the formula, This is the semantic feature matrix of the Gaussian unit; The semantic feature dimensions after compression are determined by incremental principal component analysis; ; In the formula, For the color mixing tensor, ; This is a channel-dimensional splicing operation.

[0009] Preferably, in S5, the RGB photometric loss, depth geometric loss, and semantic consistency loss are respectively expressed as: ; In the formula, For RGB luminance loss; To render RGB images; The weights for the SSIM structural similarity loss; L1 pixel-wise norm; SSIM is the image structure similarity function; ; In the formula, For depth geometry loss; To render a depth map; Input the actual depth map; This is a pixel-by-pixel multiplication operation; For dynamic motion masks; ; In the formula, For pixel-wise semantic consistency loss; To render semantic feature maps; This is a low-dimensional semantic target feature map; A high-opacity effective pixel mask; For pixel-by-pixel L2 squared error; This represents the total number of valid pixels. To prevent the division of zero into minimal constants.

[0010] Preferably, in S5, the comparison time-consistency InfoNCE loss satisfies the formula: ; In the formula, To compare the timing consistency loss; A set of positive sample pairs of the same object across frames. This represents the total number of positive samples. A set of negative samples from different objects; Normalize the semantic features of the object at the anchor point of the current frame; Historical features of the same object are cached for the momentum encoder; For negative sample features of foreign objects; The cosine similarity function; Temperature coefficient; The momentum encoder uses an exponential moving average to update historical features by caching temporal object features through a fixed-capacity sliding window.

[0011] Preferably, in S5, the multimodal total loss fusion formula is: ; In the formula, This represents the total multimodal loss. The weighted coefficients are, in order, the RGB luminance loss, depth geometry loss, semantic consistency loss, and contrast temporal consistency loss.

[0012] Preferably, in S6, the calculation formulas for the three semantic temporal evaluation metrics are as follows: ; In the formula, It serves as a semantic-temporal consistency indicator. It is the variance function; For the first Frame Object Normalized semantic features; For the first The entire set of objects in the frame; To adjust the sliding window size; ; In the formula, This refers to the semantic drift ratio; Set a preset similarity threshold; ; In the formula, This is the average semantic similarity index; It is a mean function.

[0013] Therefore, the present invention employs the above-mentioned visual localization and mapping method based on contrastive temporal semantic consistency. Compared with the prior art, the technical solution of the present invention has the following beneficial effects: (1) This invention generates dynamic motion masks through cross-frame tracking of instance IoU to filter geometric interference of dynamic objects. At the same time, it introduces sliding window InfoNCE to compare temporal loss and forces the semantic features of the same object to remain consistent across frames. This greatly suppresses camera trajectory drift and semantic map temporal distortion in dynamic scenes and significantly improves the SLAM positioning accuracy of indoor dynamic scenes. (2) This invention constructs a hybrid tensor by concatenating the channels of the spherical harmonic RGB color tensor and the PCA compressed semantic feature tensor. It outputs RGB, depth, and semantic multi-channel rendering maps simultaneously through single-time differentiable rasterization, abandoning the traditional scheme of rendering independently twice, reducing the amount of repeated rasterization computation, and eliminating the automatic differential gradient conflict caused by multiple rendering paths, thereby improving the model training convergence stability and real-time rendering efficiency; incremental PCA batch compression of high-dimensional CLIP semantic features; (3) This invention incorporates learnable low-dimensional semantic vectors as inherent parameters of a four-dimensional Gaussian ellipsoid. Through multimodal composite loss, it achieves simultaneous optimization of geometric gradient and semantic gradient coupling, thereby simultaneously improving RGB rendering fidelity, depth reconstruction accuracy and semantic modeling consistency. It is the first to create three semantic temporal evaluation indicators: STC, Drift and MeanSim, filling the gap in the lack of semantic stability quantitative evaluation in existing Gaussian SLAM, and can fully quantify the temporal quality of semantic mapping.

[0014] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] Figure 1 This is a data flow diagram of an embodiment of the visual localization and mapping method based on comparative temporal semantic consistency of the present invention; Figure 2 This is an algorithm flowchart of an embodiment of the visual localization and mapping method based on comparative temporal semantic consistency of the present invention; Figure 3 This is a flowchart of the dual-branch parallel processing of the semantic pre-extraction stage in an embodiment of the visual localization and mapping method based on comparative temporal semantic consistency of the present invention. Figure 4 This is a data flow diagram of a single-pass hybrid semantic rendering submodule in an embodiment of the visual localization and mapping method based on contrastive temporal semantic consistency of the present invention. Figure 5 This is a flowchart illustrating the calculation of temporal contrast consistency loss based on a sliding window, according to an embodiment of the visual localization and mapping method based on contrast temporal semantic consistency of the present invention. Figure 6 This is a schematic diagram illustrating the optimization mechanism of the visual localization and mapping method based on the comparative temporal semantic consistency of the present invention. Figure 7This is a flowchart illustrating the quantitative evaluation of the semantic temporal effect of the visual localization and mapping method based on comparative temporal semantic consistency of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used in the present invention should have the ordinary meaning understood by those skilled in the art.

[0017] Example 1 like Figures 1-7 As shown, this embodiment provides a visual localization and mapping method based on comparative temporal semantic consistency. It should be understood that the specific parameters, models and protocols mentioned in this embodiment are merely examples to help those skilled in the art understand the present invention, and are not intended to limit the present invention.

[0018] The visual localization and mapping method based on contrastive temporal semantic consistency of the present invention includes the following steps: S1. Receive the raw RGB-D image sequence, extract frame-by-frame object masks using instance segmentation, complete cross-frame object identity ID timing tracking through mask intersection-union ratio (IoU), and simultaneously generate dynamic object motion masks based on optical flow / mask displacement. The unified instance mask sequence and dynamic motion mask across frames are synchronously transmitted to the S2 semantic feature extraction branch and the S5 deep geometric loss calculation module.

[0019] This step uses IoU mask matching to achieve lightweight temporal object tracking without a dedicated video segmentation model. Unlike basic instance segmentation schemes that detect objects independently frame by frame without identity association, this step establishes a unique cross-frame identifier for objects, accurately divides dynamic pixel regions, and provides a basis for object association for subsequent depth loss to shield dynamic interference and temporal semantic comparison.

[0020] The execution of instance mask extraction, cross-frame ID matching, and motion mask generation includes the following sub-steps: S101. Use a deep learning instance segmentation model to parse the RGB image frame by frame and output the binary instance mask of all objects; calculate the total area of ​​each mask pixel, filter out tiny invalid masks with pixel area below a preset threshold, and remove imaging noise and tiny fragment interference. S102. For each valid instance mask in the current frame, iterate through all tracked instance masks in the previous frame and calculate the Intersection over Union (IoU) of the two sets of masks one by one. If the maximum IoU exceeds a preset matching threshold, inherit the unique ID of the corresponding object in the previous frame; if there is no matching mask, assign a new object ID and generate a complete instance mask sequence with unified IDs across frames. The formula is: ; In the formula, The mask for the object in the current frame; This is the mask for historical objects in the previous frame; The intersection and union of two masks; S103. Based on the RGB dense optical flow or instance mask spatial offset of adjacent frames, mark dynamic pixels, assign 1 to pixels in dynamic object regions and 0 to pixels in static regions, and generate frame-by-frame binary motion masks. .

[0021] Synchronously store cross-frame instance ID mapping table and frame-by-frame dynamic motion mask matrix; retain only valid object instances with continuous temporal association, and filter out isolated small objects that appear instantaneously.

[0022] Output a consistent instance mask sequence across frames and a dynamic motion mask. , which serves as the pre-constraint input for S2 semantic dimensionality reduction, S5 depth loss, and S5 contrastive temporal loss; S2 receives the RGB image sequence output by S1, calls the CLIP vision-language pre-trained encoder to extract pixel-by-pixel high-dimensional dense semantic feature maps, and uses incremental principal component analysis (incremental PCA) to fit the projection matrix in batches to compress the high-dimensional semantic features to a preset low-dimensional space; the pixel-by-pixel semantic target feature maps after dimensionality reduction are passed to the S4 rendering ground truth matching and S5 semantic consistency loss calculation modules.

[0023] This step uses batch incremental PCA to reduce the dimensionality of semantic features in long videos. Unlike the traditional PCA compression scheme that loads all frame features at once, this step iteratively fits the feature distribution in batches, avoiding memory overflow caused by loading massive high-dimensional features into memory at the same time. Meanwhile, half-precision storage significantly reduces disk usage.

[0024] Dense semantic extraction, incremental PCA fitting, and global feature projection are performed separately, including the following sub-steps: The S201 and CLIP visual encoders process RGB images frame by frame and output the original 768-dimensional pixel-by-pixel dense semantic feature map, which fully preserves the pixel-level semantic information of objects in the image. S202, Incremental PCA Batch Fitting: Randomly sample 10% of keyframes from the complete image sequence and divide them into multiple batches. Load the batches into memory and iteratively fit the PCA projection matrix and feature mean vector, without storing all the original high-dimensional features. S203, Global Feature Dimensionality Reduction Projection: Substitute the original high-dimensional semantic features of all image frames into the fitted projection matrix and uniformly map them to the preset low-dimensional dimension D; use FP16 half-precision floating-point numbers to store the dimensionality reduction feature map to reduce storage overhead.

[0025] The compression dimension D is fixed and its mapping relationship with the original CLIP feature is established. The PCA projection matrix is ​​saved for unified dimensionality reduction of the entire sequence. The original 768-dimensional high-dimensional feature map is discarded, and only the low-dimensional semantic target feature map is retained.

[0026] Output a pixel-wise low-dimensional semantic target feature map with uniform output dimension, which serves as the standard reference data for S4 rendering semantic ground truth and S5 semantic loss; S3 receives the instance mask sequence output by S1 and the low-dimensional semantic feature dimension D output by S2, constructs a four-dimensional Gaussian radiation field model, configures a complete set of learnable parameters for geometry, vision, motion, and semantics for each Gaussian ellipsoid, and registers all parameters to the optimizer; the fully initialized 4D Gaussian ellipsoid parameter set is passed to the S4 single-pass hybrid semantic rendering module.

[0027] This step uses learnable low-dimensional semantic feature vectors as native parameters of the Gaussian ellipsoid. Unlike traditional 3D / 4D Gaussian modeling schemes that only optimize RGB geometric parameters and bind semantic labels, this scheme achieves integrated coupling of semantic features and geometric units, ensuring that semantic gradients and geometric gradients are transmitted synchronously during backpropagation.

[0028] The initialization of multiple parameters of the four-dimensional Gaussian ellipsoid is performed, including the following sub-steps: S301. Assign a three-dimensional spatial position, covariance matrix, and opacity parameter to each Gaussian ellipsoid, and initialize the spatial distribution based on the RGB-D depth ground truth. S302. Initialize the spherical harmonic color coefficients sh for view-dependent RGB rendering; build a deformation network MLP as temporal motion parameters to model the displacement and rotation transformations of dynamic objects over time. S303, a semantic feature vector with configuration dimension equal to the compression dimension D of S2, is initialized as a Gaussian random vector with variance of 0.01 and marked as a learnable variable; S304. Synchronously register all parameters of geometry, spherical harmonic color, deformation network, and semantic features to the Adam optimizer to participate in subsequent joint gradient updates.

[0029] All Gaussian unit semantic vector dimensions are strictly consistent with the S2 output D; the deformation network is only enabled for Gaussian units of dynamic objects with instance masking.

[0030] Output a complete set of initialized 4D Gaussian ellipsoid parameters as the basic scene representation for S4 hybrid rendering; S4 receives the complete set of parameters of the 4D Gaussian ellipsoid and the camera's line-of-sight vector output by S3, calculates the RGB tensor of the spherical harmonic function and concatenates the Gaussian built-in semantic feature tensor channels into a mixed color tensor, and sends it to the differentiable Gaussian rasterizer to perform a single rendering, simultaneously outputting the RGB rendering image, the rendering depth map, and the rendering semantic feature map; the three types of rendering images are simultaneously transmitted to all multimodal loss calculation modules in S5.

[0031] This step adopts a single rasterization rendering architecture after RGB-semantic channel splicing. Traditional solutions require two separate rasterizations to render RGB and semantic features separately, which results in redundant computational overhead and gradient conflicts in the dual rendering paths. This step outputs multi-channel results synchronously in a single rendering, reducing computational load and eliminating differential graph conflicts, thus ensuring stable gradient propagation.

[0032] The process of spherical harmonic RGB calculation, hybrid tensor stitching, single raster rendering, and channel splitting is executed in stages, including the following sub-steps: S401. Based on the spherical harmonic coefficients of the Gaussian ellipsoid and the camera's line of sight, the RGB color of each Gaussian unit is calculated using the following formula: ; In the formula, For the entire Gaussian unit RGB color matrix, ; This represents the total number of Gaussian ellipsoids. This is a function for converting spherical harmonic coefficients to RGB. This is a truncation function that constrains pixel colors to the range of 0 to 1. The spherical harmonic color coefficient; This is the vector representing the line of sight.

[0033] S402. Extract the built-in semantic feature vectors of all Gaussian units to construct a semantic matrix. D represents the semantic feature dimension after compression by S2 incremental principal component analysis; the RGB matrix and semantic matrix are concatenated along the channel dimension, and the mixture tensor formula is: ; In the formula, For the color mixing tensor, ; This is a channel-level splicing operation; This is the semantic feature matrix of the Gaussian unit.

[0034] S403: Input the blending tensor into the GPU-accelerated differentiable Gaussian rasterizer renderer and output a multi-channel blended image in one go. ; In the formula, Render images for multiple channels; Input RGB-D image height and width; the first 3 channels correspond to the RGB rendering result, and the last... The channel corresponds to the semantic feature rendering result, and the renderer synchronously outputs a pixel-by-pixel rendering depth map. .

[0035] S404, to Split by channel dimension to obtain independent RGB rendered images. Rendering semantic feature maps Simultaneously extract rasterized output depth map .

[0036] The rendered image resolution is perfectly aligned with the original input RGB-D image; the hybrid tensor participates in automatic differential tracking throughout the process, ensuring that the gradient of semantic feature parameters can be backpropagated.

[0037] Output RGB rendering image Rendering depth map Rendering semantic feature maps , serving as the true input value for the rendering side of the S5 four-class loss function; S5 receives the RGB rendering image, rendering depth map, and rendering semantic feature map output by S4, and receives the dynamic motion mask output by S1. The system uses a cross-frame object ID mapping table, receives low-dimensional semantic target feature maps output by S2, and sequentially calculates RGB photometric loss, depth geometric loss, semantic consistency loss, and sliding window InfoNCE comparison temporal consistency loss. The weighted summation yields the total loss, and backpropagation synchronously updates all 4D Gaussian ellipsoid parameters. The updated Gaussian parameters are then fed back to S4 for iterative rendering optimization.

[0038] This step constructs a geometric-semantic coupled quadruple composite loss system. Traditional semantic Gaussian SLAM only uses single-frame semantic pixel supervision and has no cross-frame temporal constraints. This step introduces sliding window momentum InfoNCE contrastive loss to constrain the convergence of semantic features of the same object across frames. At the same time, it uses motion masks to shield the geometric errors of dynamic objects, achieving triple synchronous optimization of geometric localization, rendering quality, and semantic temporal sequence.

[0039] The four independent losses are calculated step by step, the total loss is weighted and fused, and the gradient is updated in reverse. The complete formulas for each loss are as follows: RGB photometric loss , is represented as: ; In the formula, For RGB luminance loss; The SSIM structural similarity loss weights are used in this embodiment. ; is the L1 pixel norm; SSIM is the image structure similarity function.

[0040] We employ L1 loss to constrain pixel grayscale error and SSIM to constrain the global image structure, and then use a weighted balance between pixel accuracy and visual consistency.

[0041] Depth geometric loss is expressed as: ; In the formula, For depth geometry loss; Input the actual depth map; This is a pixel-by-pixel multiplication operation.

[0042] mask By setting the pixel loss of dynamic objects to 0, depth error constraints are applied only to static scene areas to avoid trajectory drift caused by dynamic objects.

[0043] Semantic consistency loss is expressed as: ; In the formula, For pixel-wise semantic consistency loss; This step S4 outputs a rendered semantic feature map; This is a low-dimensional semantic target feature map; For rendering effective pixel masks with an opacity greater than 0.1; For pixel-by-pixel L2 squared error; This represents the total number of valid pixels. To prevent the division of zero into minimal constants.

[0044] Semantic constraints are applied only to pixels with high Gaussian opacity and effective visibility, filtering out noise in invalid transparent areas.

[0045] The time-series consistency loss for sliding window InfoNCE comparison is expressed as: The basic loss formula is: ; In the formula, To compare the timing consistency loss; A set of positive sample pairs of the same object across frames. This represents the total number of positive samples. A set of negative samples from different objects; Normalize the semantic features of the object at the anchor point of the current frame; Historical features of the same object are cached for the momentum encoder; For negative sample features of foreign objects; The cosine similarity function; In this embodiment, the temperature coefficient is used. .

[0046] Extract the pixels at the intersection of the high-opacity region in the S4 rendered semantic map and the instance mask in S1, and use spatial average pooling to obtain the global semantic features of a single object. Perform L2 normalization; Configure a fixed-capacity sliding window with W=10 frames per window. Push normalized object features of new frames into the window and automatically remove features from the oldest frames stored in the window. The momentum feature encoder uses an exponential moving average cache of historical object semantic features with a momentum coefficient of 0.99, and uses the momentum cached features as positive sample targets when calculating the loss. InfoNCE contrast loss calculation is performed only when the sliding window stores ≥2 frames, bringing features of the same object closer and pushing features of different objects further away.

[0047] The total loss fusion formula is expressed as: ; In the formula, This represents the total multimodal loss. The weighted coefficients for RGB luminance loss, depth geometry loss, semantic consistency loss, and contrast temporal consistency loss are determined by tuning based on an indoor dynamic scene dataset.

[0048] The gradients generated by RGB luminance loss and depth geometry loss are used to back-update the geometric parameters (position, covariance, opacity) of the 4D Gaussian ellipsoid; the gradients generated by semantic consistency loss and contrast temporal consistency loss are used to directly update the built-in semantic feature vector of the Gaussian unit; all parameters share the Gaussian ellipsoid carrier, realizing synchronous coupling optimization of semantic gradient and geometric gradient.

[0049] Each keyframe iteration completes a total loss calculation and parameter update; the updated set of 4D Gaussian parameters is sent back to the S4 rendering module for iterative optimization.

[0050] Output the set of parameters of the 4D Gaussian ellipsoid after gradient update, which will be used for the next round of hybrid rendering and loss constraints.

[0051] S6. Receive all iteratively optimized 4D Gaussian scene parameters and full-frame object temporal normalized semantic feature cache. Fix Gaussian geometry and semantic parameters, only optimize spherical harmonic color and exposure parameters to complete global color refinement. Based on the semantic similarity of objects matched in adjacent frames within the sliding window, calculate three quantitative evaluation indicators: STC semantic temporal consistency, Drift ratio, and MeanSim average similarity. Finally, output the complete camera trajectory, the 4D Gaussian scene model with semantic features, and the three semantic temporal evaluation reports to the terminal server.

[0052] This step adds three new types of exclusive semantic temporal quantification metrics. Traditional 4D Gaussian SLAM only relies on trajectory error and image PSNR / SSIM geometric rendering metrics to evaluate mapping quality, which cannot measure the cross-frame stability of semantic modeling. This solution fully quantifies semantic temporal consistency from three dimensions: fluctuation degree, drift ratio, and overall similarity, forming a complete semantic mapping evaluation system.

[0053] Global color refinement and calculation of three semantic metrics are performed separately, including the following sub-steps: S601: Fix all geometric parameters, deformation network parameters, and learnable semantic feature vectors of the 4D Gaussian ellipsoid, and set only the spherical harmonic color coefficient and camera exposure parameters as optimizable variables, and iterate 2000 times to optimize the rendering color fidelity. S602. Traverse all adjacent frames within the sliding window and filter the set of matching objects that exist simultaneously in both frames. Calculate the cosine similarity of semantic features of objects pairwise. ; S603, Solving for three types of semantic time-series indices, specifically: The STC semantic temporal consistency index is expressed as: ; In the formula, It serves as a semantic-temporal consistency indicator. It is the variance function; For the first Frame Object Normalized semantic features; For the first The entire set of objects in the frame; The value represents the size of the sliding window; the lower the value, the smaller the fluctuation of the semantic features of objects across frames and the stronger the temporal stability.

[0054] The Drift semantic drift ratio metric is expressed as: ; In the formula, This refers to the semantic drift ratio; In this embodiment, to preset a similarity threshold, The numerator is the total number of object pairs with similarity below the threshold and severe semantic drift; the denominator is the total number of matched object pairs in adjacent frames; the higher the index, the more severe the semantic drift in the scene.

[0055] The MeanSim average semantic similarity index is expressed as: ; In the formula, This is the average semantic similarity index; It is a mean function; the higher the index value, the better the overall semantic consistency of all objects in the scene across frames.

[0056] S604, export the optimized camera pose trajectory file and the 4D Gaussian scene storage file with semantic features, simultaneously outputting... , , The semantic temporal evaluation report.

[0057] The global refinement process does not change the geometric and semantic feature parameters, but only optimizes the color rendering effect; the three indicators are used together to fully measure the temporal stability of semantic mapping in dynamic scenes.

[0058] Output the final camera trajectory, semantic 4D Gaussian scene model, and STC / Drift / MeanSim semantic temporal evaluation metrics, which are used for scene reconstruction quality assessment and downstream robot navigation and AR virtual-real anchoring task deployment.

[0059] Therefore, the present invention adopts the above-mentioned visual localization and mapping method based on contrastive temporal semantic consistency. This method simultaneously improves the localization accuracy, rendering efficiency, and semantic temporal stability of dynamic scenes, and constructs a complete semantic quantitative evaluation system.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A visual localization and mapping method based on contrastive temporal semantic consistency, characterized in that, Includes the following steps: S1. Acquire RGB-D image sequences of indoor dynamic scenes using RGB-D sensors, extract frame-by-frame object masks using instance segmentation, complete cross-frame object ID temporal tracking by relying on mask IoU matching, and generate a unified instance mask sequence; combine RGB optical flow or mask displacement to generate dynamic region motion masks, and synchronously send the instance masks, motion masks, and original RGB images to S2 and S5. S2 receives the RGB image from S1, extracts dense semantic feature maps through CLIP vision-language encoder, and uses incremental principal component analysis to fit the projection matrix in batches to compress the high-dimensional semantic features into low-dimensional pixel-by-pixel semantic target feature maps. The reduced feature maps and feature dimension parameters are then sent to S3, S4, and S5 simultaneously. S3. Receive feature compression dimension parameters, build a 4D Gaussian radiation field model, configure three-dimensional coordinates, covariance, spherical harmonic color, opacity, deformation network and learnable semantic vector matching compression dimension for each Gaussian ellipsoid, and send the fully initialized Gaussian parameter set to S4. S4. Receive Gaussian parameters and camera line vector, concatenate the RGB tensor and Gaussian built-in semantic tensor channels obtained by spherical harmonic conversion into a hybrid tensor, input it into the differentiable Gaussian rasterizer renderer for a single rendering, and simultaneously output RGB, depth and semantic three-class rendering images to S5. S5. Gather the rendering results of S4, motion mask and object ID sequence of S1, and low-dimensional semantic truth map of S2. Calculate RGB luminance loss, dynamic mask constraint depth loss, opacity mask constraint semantic loss, and InfoNCE contrast temporal loss of sliding window + momentum encoder architecture in sequence. Weighted fusion is used to obtain the total loss. Backpropagation is used to update all Gaussian parameters synchronously and backpropagate to S4 for iterative optimization. S6. After the full sequence iteration is completed, read the optimized Gaussian parameters and object temporal semantic cache, fix the geometric and semantic parameters and only refine the spherical harmonic color and exposure; calculate the semantic cosine similarity of adjacent matching objects, solve the three evaluation indicators of STC semantic temporal consistency, Drift semantic drift ratio, and MeanSim average similarity, and finally output the camera trajectory, the semantic 4D Gaussian scene model and semantic evaluation indicators.

2. The visual localization and mapping method based on contrastive temporal semantic consistency according to claim 1, characterized in that, In S1, cross-frame object identity ID timing tracking, the object mask matching determination satisfies: ； In the formula, The mask for the object in the current frame; This is the mask for historical objects in the previous frame; The intersection and union of two masks; If the maximum intersection-union ratio is greater than the preset matching threshold, the current object inherits the object ID from the previous frame; otherwise, a new object ID is assigned. At the same time, tiny invalid masks with pixel areas smaller than the preset threshold are filtered out.

3. The visual localization and mapping method based on contrastive temporal semantic consistency according to claim 2, characterized in that, In S4, the RGB color tensor, semantic feature tensor, and blend color tensor satisfy the following calculation formula: ； In the formula, For the entire Gaussian unit RGB color matrix, ; This represents the total number of Gaussian ellipsoids. This is a function for converting spherical harmonic coefficients to RGB. This is a truncation function; The spherical harmonic color coefficient; The line-of-sight vector; ； In the formula, This is the semantic feature matrix of the Gaussian unit; The semantic feature dimensions after compression are determined by incremental principal component analysis; ； In the formula, For the color mixing tensor, ; This is a channel-dimensional splicing operation.

4. The visual localization and mapping method based on contrastive temporal semantic consistency according to claim 3, characterized in that, In S5, the RGB photometric loss, depth geometric loss, and semantic consistency loss are respectively expressed as: ； In the formula, For RGB luminance loss; To render RGB images; The weights for the SSIM structural similarity loss; It is the L1 pixel norm; SSIM is an image structure similarity function; ； In the formula, For depth geometry loss; To render a depth map; Input the actual depth map; This is a pixel-by-pixel multiplication operation; For dynamic motion masks; ； In the formula, For pixel-wise semantic consistency loss; To render semantic feature maps; This is a low-dimensional semantic target feature map; A high-opacity effective pixel mask; For pixel-by-pixel L2 squared error; Total number of valid pixels; To prevent the division of zero into minimal constants.

5. The visual localization and mapping method based on contrastive temporal semantic consistency according to claim 4, characterized in that, In S5, the InfoNCE loss for time-consistent comparison satisfies the following formula: ； In the formula, To compare the timing consistency loss; A set of positive sample pairs of the same object across frames. This represents the total number of positive samples. A set of negative samples from different objects; Normalize the semantic features of the object at the anchor point of the current frame; Historical features of the same object are cached for the momentum encoder; For negative sample features of foreign objects; The cosine similarity function; Temperature coefficient; The momentum encoder uses an exponential moving average to update historical features by caching temporal object features through a fixed-capacity sliding window.

6. The visual localization and mapping method based on contrastive temporal semantic consistency according to claim 5, characterized in that, In S5, the formula for multimodal total loss fusion is: ； In the formula, This represents the total multimodal loss. The weighted coefficients are, in order, the RGB luminance loss, depth geometry loss, semantic consistency loss, and contrast temporal consistency loss.

7. The visual localization and mapping method based on contrastive temporal semantic consistency according to claim 6, characterized in that, In S6, the formulas for calculating the three semantic temporal evaluation metrics are as follows: ； In the formula, It serves as a semantic-temporal consistency indicator. It is the variance function; For the first Frame Object Normalized semantic features; For the first The entire set of objects in the frame; To adjust the sliding window size; ； In the formula, This refers to the semantic drift ratio; Set a preset similarity threshold; ； In the formula, This is the average semantic similarity index; It is a mean function.

8. A computer device, characterized in that, include: A processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1-7.

9. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1-7.

Images