Methods, devices, and software products for high-precision visual positioning and high-fidelity object-level mapping in indoor dynamic environments

Abstract

This disclosure provides a method, apparatus, and program product for high-precision visual positioning and high-fidelity object-level map construction in indoor dynamic environments. The method includes: segmenting and extracting features from an image based on a large semantic segmentation model; generating an extended mask for active dynamic objects; obtaining robust inactive dynamic points; initially optimizing the pose using a Gaussian map; calculating the distance between the robust inactive dynamic points corresponding to the object and the center of the Gaussian ellipsoid; combining the semantic loss change of the corresponding pixel region in the keyframe sequence to obtain the final dynamic object mask; optimizing by backpropagating the loss of static pixel regions; and densifying the Gaussian ellipsoid in the multi-view intersection region to obtain high-precision visual positioning results and a high-fidelity object-level map. This disclosure can accurately and comprehensively remove dynamic objects in any indoor dynamic environment, achieving high-precision visual positioning and constructing a high-fidelity object-level map.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and artificial intelligence, and in particular to a method, apparatus and program product for high-precision visual positioning and high-fidelity object-level map construction for indoor dynamic environments. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] Visual Simultaneous Localization and Mapping (VSLAM) is a task that uses visual sensor data to reconstruct 3D maps in unknown environments while simultaneously estimating camera pose. Traditional VSLAM methods have significant limitations in dynamic environments, particularly in handling the impact of dynamic objects on camera pose estimation. Furthermore, existing methods struggle to reconstruct high-fidelity object-level maps, which to some extent restricts the practical application of VSLAM technology. With the continuous development of 3D reconstruction technology, high-fidelity map reconstruction has gradually become a research hotspot. 3D Gaussian Splatting for Real-Time Radiance Field Rendering (3DGS) technology reconstructs high-fidelity scene maps from input images and possesses powerful real-time rendering capabilities. Its efficient rendering performance allows 3DGS to be seamlessly integrated with the VSLAM mapping backend, providing innovative solutions for cutting-edge fields such as autonomous driving and virtual reality.

[0004] Currently, VSLAM based on 3DGS has become a research hotspot, but most current systems still lack robustness in dynamic environments. To address this issue, some researchers have attempted to combine multiple methods to handle dynamic objects in the scene, thereby improving localization accuracy and system robustness. However, these methods have not fully considered edge noise between color and depth images, have failed to effectively handle passive dynamic objects in the scene, and have not effectively eliminated artifacts generated by dynamic objects. There is still significant room for improvement in localization accuracy and map reconstruction quality, requiring further optimization and refinement. Summary of the Invention

[0005] In view of this, the purpose of this disclosure is to propose a method, device and program product for high-precision visual positioning and high-fidelity object-level map construction for indoor dynamic environments, which at least to some extent solves some technical problems in the prior art.

[0006] To achieve the above objectives, the first aspect of this exemplary embodiment provides a method for high-precision visual positioning and high-fidelity object-level map construction for indoor dynamic environments, the method comprising:

[0007] Collect visual sensor data, including color images and depth images. Segment and extract features from the color images based on a semantic segmentation model. Obtain an active dynamic object extension mask and semantic image through deep clustering and feature encoding. Obtain robust non-active dynamic points based on the active dynamic object extension mask, semantic image, color image and depth image.

[0008] A preliminary optimized pose is obtained based on robust inactive dynamic points and the Gaussian ellipsoid in the Gaussian map. Based on the preliminary optimized pose, the distance between the center of the robust inactive dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map is calculated to obtain potential dynamic semantic features. Based on the semantic loss change of the corresponding pixel region in the keyframe sequence based on the potential dynamic semantic features, passive dynamic semantic features are obtained. Based on the passive dynamic semantic features and the active dynamic object expansion mask, the final dynamic object mask is obtained. Based on the passive dynamic semantic features and the robust inactive dynamic points, static points are obtained.

[0009] Based on the final dynamic object mask, the loss of the input image and the rendered image in the static pixel region is obtained. The loss is backpropagated using the differentiable rendering property of the Gaussian ellipsoid to obtain a high-precision visual positioning result.

[0010] Based on the high-precision visual positioning results, the static points are inserted into the Gaussian map, and the parameters of the Gaussian ellipsoid in the Gaussian map are optimized based on the loss to obtain a high-fidelity map.

[0011] Based on the loss, high-loss objects are obtained. The ray intersection regions of the high-loss objects in the keyframe sequence are calculated to obtain multi-view ray intersection regions. Based on the multi-view ray intersection regions, a Gaussian ellipsoid is compacted to obtain a high-fidelity object-level map.

[0012] Based on the same inventive concept, a second aspect of the exemplary embodiments of this disclosure provides a high-precision visual positioning and high-fidelity object-level map building apparatus for indoor dynamic environments, including:

[0013] A visual sensor data preprocessing module for indoor dynamic environments is configured to collect visual sensor data, including color images and depth images, segment and extract features from the color images based on a semantic segmentation model, obtain an active dynamic object extended mask and semantic image through deep clustering and feature encoding, and obtain robust non-active dynamic points based on the active dynamic object extended mask, semantic image, color image and depth image.

[0014] The dynamic object detection and deletion module is configured to obtain a preliminary optimized pose based on robust non-active dynamic points and a Gaussian ellipsoid in a Gaussian map; calculate the distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map based on the preliminary optimized pose to obtain potential dynamic semantic features; obtain passive dynamic semantic features based on the semantic loss change of the corresponding pixel region in the keyframe sequence based on the potential dynamic semantic features; obtain the final dynamic object mask based on the passive dynamic semantic features and the active dynamic object extended mask; and obtain static points based on the passive dynamic semantic features and the robust non-active dynamic points.

[0015] The high-precision visual positioning module is configured to obtain the loss of the input image and the rendered image in the static pixel region based on the final dynamic object mask, and backpropagate the loss using the differentiable rendering characteristics of the Gaussian ellipsoid to obtain the high-precision visual positioning result.

[0016] The high-fidelity map optimization module is configured to insert the static points into a Gaussian map based on the high-precision visual positioning results, and optimize the parameters of the Gaussian ellipsoid in the Gaussian map based on the loss to obtain a high-fidelity map.

[0017] The high-fidelity object-level map densification module is configured to obtain high-loss objects based on the loss, calculate the ray intersection regions of the high-loss objects in the keyframe sequence, obtain multi-view ray intersection regions, and densify the Gaussian ellipsoid based on the multi-view ray intersection regions to obtain a high-fidelity object-level map.

[0018] Based on the same inventive concept, a third aspect of the exemplary embodiments of this disclosure provides a computer program product including computer program instructions that, when run on a computer, cause the computer to perform the method as described in the first aspect.

[0019] As can be seen from the above, the embodiments of this disclosure provide the following: collecting visual sensor data, including color images and depth images; segmenting and extracting features from the color images based on a semantic segmentation model; obtaining an active dynamic object extended mask and semantic image through deep clustering and feature encoding; obtaining robust inactive dynamic points based on the active dynamic object extended mask, semantic image, color image, and depth image; obtaining preliminary optimized poses based on the robust inactive dynamic points and the Gaussian ellipsoid in the Gaussian map; calculating the distance between the center of the robust inactive dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map based on the preliminary optimized poses to obtain potential dynamic semantic features; and obtaining passive dynamic semantic features based on the semantic loss change of the corresponding pixel region in the keyframe sequence according to the potential dynamic semantic features. Based on the passive dynamic semantic features and the active dynamic object extension mask, a final dynamic object mask is obtained. Based on the passive dynamic semantic features and the robust non-active dynamic points, static points are obtained. Based on the final dynamic object mask, the loss of the input image and the rendered image in the static pixel region is obtained. The loss is backpropagated using the differentiable rendering characteristics of the Gaussian ellipsoid to obtain a high-precision visual positioning result. Based on the high-precision visual positioning result, the static points are inserted into the Gaussian map. Based on the loss, the parameters of the Gaussian ellipsoid in the Gaussian map are optimized to obtain a high-fidelity map. Based on the loss, high-loss objects are obtained. The ray intersection regions of the high-loss objects in the keyframe sequence are calculated to obtain multi-view ray intersection regions. Based on the multi-view ray intersection regions, the Gaussian ellipsoid is compacted to obtain a high-fidelity object-level map. This disclosure proposes a method for high-precision visual positioning and high-fidelity object-level map construction in indoor dynamic environments. Through steps such as visual sensor data preprocessing for indoor dynamic environments, dynamic object detection and deletion, high-precision visual positioning, high-fidelity map optimization, and high-fidelity object-level map densification, it overcomes the limitations of traditional VSLAM technology. It can accurately and comprehensively delete dynamic objects in any indoor dynamic environment, achieve high-precision visual positioning, and construct a high-fidelity object-level map. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for high-precision visual positioning and high-fidelity object-level map construction for indoor dynamic environments provided in this embodiment of the disclosure;

[0022] Figure 2 This is a schematic diagram of a high-precision visual positioning and high-fidelity object-level map building device for indoor dynamic environments provided in an embodiment of this disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this disclosure clearer, the principles and spirit of this disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0024] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0025] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The article "a" or "an" preceding an element does not exclude the presence of multiple such elements.

[0026] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments.

[0027] As described in the background section, existing visual positioning technologies often face numerous challenges in dynamic environments. Accurately and comprehensively identifying and removing dynamic objects from images is a crucial step in improving visual positioning accuracy. Existing methods typically remove active dynamic objects based on active dynamic masks segmented from color images, and passive dynamic objects based on depth masks obtained from multi-frame depth reprojection errors. When the depth image accuracy is low, these methods are prone to significant errors. In dynamic environments, large movements can cause the edges of dynamic objects to not be strictly aligned in color and depth images. Furthermore, due to the time synchronization issue between color and depth images, existing methods struggle to remove all pixel regions containing dynamic objects, thus affecting the accuracy of visual positioning and map building tasks. In addition, existing visual simultaneous localization and map building technologies often reconstruct scenes using point clouds. Point clouds lack size and orientation attributes, resulting in numerous holes in the map, severely impacting visual quality.

[0028] To address the aforementioned issues, this disclosure provides a method, apparatus, and program solution for high-precision visual positioning and high-fidelity object-level map construction in indoor dynamic environments, specifically including:

[0029] Visual sensor data, including color and depth images, is collected. The color images are segmented and features extracted using a large-scale semantic segmentation model. Active dynamic object extended masks and semantic images are obtained through deep clustering and feature encoding. Robust inactive dynamic points are obtained based on these active dynamic object extended masks, semantic images, color images, and depth images. Preliminary optimized poses are obtained based on these robust inactive dynamic points and the Gaussian ellipsoid in the Gaussian map. The distance between the center of each robust inactive dynamic point and the center of the Gaussian ellipsoid in the Gaussian map is calculated based on the preliminary optimized poses to obtain latent dynamic semantic features. Passive dynamic semantic features are obtained based on the semantic loss change of the corresponding pixel region in the keyframe sequence according to these latent dynamic semantic features. Semantic features and the extended mask of the active dynamic object are used to obtain the final dynamic object mask. Static points are obtained based on the passive dynamic semantic features and the robust non-active dynamic points. The loss of the input and rendered images in the static pixel region is obtained based on the final dynamic object mask. The loss is backpropagated using the differentiable rendering properties of the Gaussian ellipsoid to obtain a high-precision visual positioning result. The static points are inserted into a Gaussian map based on the high-precision visual positioning result. The parameters of the Gaussian ellipsoid in the Gaussian map are optimized based on the loss to obtain a high-fidelity map. High-loss objects are obtained based on the loss. The ray intersection regions of the high-loss objects in the keyframe sequence are calculated to obtain multi-view ray intersection regions. The Gaussian ellipsoid is compacted based on the multi-view ray intersection regions to obtain a high-fidelity object-level map. The advantage of this scheme is that it performs object-level deletion on both active and passive dynamic objects in the indoor dynamic environment, accurately and comprehensively deleting dynamic objects, and achieving high-precision visual positioning based on the differentiable rendering properties of the Gaussian ellipsoid, thus constructing a high-fidelity object-level map.

[0030] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0031] refer to Figure 1 A method for high-precision visual positioning and high-fidelity object-level map construction in indoor dynamic environments, the method comprising the following steps:

[0032] Step S110: Collect visual sensor data, including color images and depth images. Segment and extract features from the color images based on a semantic segmentation model. Obtain an active dynamic object extension mask and semantic image through deep clustering and feature encoding. Obtain robust non-active dynamic points based on the active dynamic object extension mask, semantic image, color image, and depth image.

[0033] The following section will introduce the methods for generating object-level masks and semantic images:

[0034] In this exemplary embodiment, the object-level mask and semantic image are obtained through the following methods:

[0035] The Grounding DINO model is used to detect instance objects in the current color image. SAM2 is used to obtain object-level masks, and the high-dimensional semantic features of each object are encoded based on a predefined encoding table to obtain a semantic image.

[0036] In practice, an object-level mask is generated, including:

[0037] In a color image, a text cue is input into the Grounding DINO model to detect the object specified by the text cue, obtaining the bounding box of the object in the color image. This bounding box is used as a cue for SAM2 to generate an object-level detection mask for the current frame. Using SAM2's mask propagation function, the object in the current color image is continuously tracked, resulting in an object-level propagated mask for subsequent frames. During object tracking, when the number of objects in the color image decreases to three-quarters of its original number, the objects in the color image are re-detected, obtaining the bounding boxes of the objects in the detected frames. These bounding boxes are used as cue for SAM2 to generate another object-level detection mask. The object-level detection mask and the object-level propagated mask of the color image are correlated to obtain the final object-level mask used for visual localization and high-fidelity object-level map construction.

[0038] In practice, semantic images are generated, including:

[0039] Based on a predefined encoding table, the high-dimensional feature vector output by the Grounding DINO model is encoded, and a low-dimensional semantic feature is assigned to each object. Combined with the object-level mask, a semantic image is obtained for visual localization and high-fidelity object-level map construction.

[0040] In the exemplary embodiments described above, the methods for obtaining object-level masks and semantic images were introduced. Below, the method for extending proactive dynamic object masks will be introduced:

[0041] In this exemplary embodiment, the active dynamic object extension mask is obtained through the following methods:

[0042] Based on the object-level mask and the semantic image, an active dynamic object mask is obtained. The active dynamic object mask is then extended based on the depth image to obtain an active dynamic object extended mask.

[0043] In practical implementation, the active dynamic object mask is obtained, including:

[0044] The semantic features corresponding to objects with active motion capabilities are queried, and the pixel regions corresponding to the semantic features in the semantic image are obtained to obtain the active dynamic object mask.

[0045] In practical implementation, the active dynamic object mask is extended, including:

[0046] The depth image is clustered using the K-Means clustering algorithm to obtain depth clustering results. The pixel coordinates with true values ​​in the active dynamic object mask are queried, and the depth category labels of these pixel coordinates in the depth clustering results are counted to obtain the active dynamic depth category. Each pixel in the depth image is then determined to belong to the active dynamic depth category, resulting in a binary result for each pixel. The binary results are merged to obtain the depth mask. The active dynamic object mask is then morphologically dilated to obtain the dilated mask. The intersection region of the dilated mask and the depth mask is taken to obtain the active dynamic object extended mask.

[0047] In the above exemplary embodiments, a method for obtaining an active dynamic object extension mask was introduced. Below, a method for obtaining robust non-active dynamic points will be introduced:

[0048] In this exemplary embodiment, robust non-active dynamic points are obtained through the following methods:

[0049] Based on the extended mask of the active dynamic object, the depth image, and the intrinsic parameters of the visual sensor, inactive dynamic points are obtained. Based on the coordinates and semantic features of the inactive dynamic points, edge noise in the inactive dynamic points is filtered out to obtain robust inactive dynamic points.

[0050] In practical implementation, non-active dynamic points are obtained, including:

[0051] The active dynamic object extension mask is applied to the semantic image, color image, and depth image, preserving the non-active dynamic pixel regions. The non-active dynamic pixel regions in the semantic image, color image, and depth image are correlated. Using a pinhole camera model, based on the depth image and the intrinsic parameters of the vision sensor, the non-active dynamic points in the camera coordinate system are back-projected pixel coordinates, which can be represented as:

[0052]

[0053] Where K represents the intrinsic parameters of the vision sensor, u,v represent pixel coordinates, D represents the value of pixel (u,v) in the depth image, and X,Y,Z represent the three-dimensional coordinates in the camera coordinate system.

[0054] In practical implementation, robust non-active dynamic points are obtained, including:

[0055] The spatial coordinates and semantic features of the non-active dynamic points are normalized. The normalized spatial coordinates and semantic features are concatenated to obtain a high-dimensional feature vector. The DBSCAN clustering algorithm is used to cluster the high-dimensional feature vector to obtain density clustering results. Information about each cluster in the density clustering results is statistically analyzed to obtain the number of points contained in the cluster and the aspect ratio of the cluster. The number of objects in the current color image is queried, and the clusters are sorted according to the number of points contained in each cluster. The cluster with the largest number of objects is obtained, and a cluster filtering threshold is determined. Clusters with fewer points than the cluster filtering threshold and clusters with aspect ratios greater than a specified aspect ratio threshold are deleted to obtain robust non-active dynamic points.

[0056] Step S120: Obtain a preliminary optimized pose based on robust non-active dynamic points and the Gaussian ellipsoid in the Gaussian map. Calculate the distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map based on the preliminary optimized pose to obtain potential dynamic semantic features. Obtain passive dynamic semantic features based on the semantic loss change of the corresponding pixel region in the keyframe sequence according to the potential dynamic semantic features. Obtain the final dynamic object mask based on the passive dynamic semantic features and the active dynamic object expansion mask. Obtain static points based on the passive dynamic semantic features and the robust non-active dynamic points.

[0057] The following will introduce a preliminary method for optimizing pose:

[0058] In this exemplary embodiment, the preliminary optimized pose is obtained through the following method:

[0059] Calculate the covariance of each point in the robust non-active dynamic points, recover the center coordinates and covariance matrix of each Gaussian ellipsoid in the Gaussian map, use nearest neighbor search to obtain the correspondence between the robust non-active dynamic points and the Gaussian ellipsoids in the Gaussian map, construct the objective function, and use maximum likelihood estimation to obtain the best transformation of the current sample, i.e., the preliminary optimized pose.

[0060] In specific implementation, the covariance of the robust non-active dynamic points is calculated, including:

[0061] Calculate the K nearest neighbors of each point in the robust non-active dynamic points, and use the K nearest neighbors to calculate the covariance matrix of each point. The covariance of the robust non-active dynamic points can be expressed as:

[0062]

[0063] Where Σ f σ represents the covariance of the robust non-active dynamic points. ab The covariance between dimensions a and b in the robust, non-active dynamic point can be expressed as:

[0064]

[0065] Where x ka This represents the coordinates of the k-th point in dimension a. N represents the average coordinates along dimension a, i.e., the centroid. K This indicates that the covariance matrix of each point is calculated using the K nearest neighbors.

[0066] In practice, the center coordinates and covariance matrix of each Gaussian ellipsoid in the Gaussian map are restored, including:

[0067] Extract the center coordinates, axial radius, and rotation of each Gaussian ellipsoid from the Gaussian map. Reconstruct the covariance matrix of each Gaussian ellipsoid from the axial radius and rotation, which can be expressed as:

[0068] Σ m =RSS T R T

[0069] Where R represents the rotation property of the Gaussian ellipsoid, S represents the scale property of the Gaussian ellipsoid, and Σ m Let represent the covariance matrix of the Gaussian ellipsoid.

[0070] In practice, nearest neighbor search is used to obtain the correspondence between robust non-active dynamic points and Gaussian ellipsoids in the Gaussian map, and matching pairs with the same index are obtained.

[0071] In practical implementation, the objective function is constructed, including:

[0072] Each point can be defined as an independent Gaussian distribution, which can be represented as:

[0073]

[0074] in This represents the observed coordinates of point i. Let i represent the covariance matrix of point i. Let i represent the Gaussian distribution at point i.

[0075] Each Gaussian ellipsoid is defined as an independent Gaussian distribution, which can be represented as:

[0076]

[0077] in Represents the coordinate observations of Gaussian ellipsoid i. Let i represent the covariance matrix of the Gaussian ellipsoid i. Let i represent the Gaussian distribution of the Gaussian ellipsoid i.

[0078] Based on the matching pairs with the same index, an error term is defined, which can be expressed as:

[0079]

[0080] in, Describe the Gaussian distribution of the Gaussian ellipsoid i. Let i represent the Gaussian distribution of point i, and T represent the pose transformation between the robust non-active dynamic point and the Gaussian map.

[0081] Assume there exists an optimal pose transformation T. * It is obvious that:

[0082]

[0083] because and Given independent Gaussian random variables, their linear combination It is also a Gaussian random variable, which can be represented as:

[0084]

[0085] Where T * This represents the assumed optimal pose transformation. This represents the distribution of the Gaussian random variable corresponding to the error term.

[0086] In practice, maximum likelihood estimation is used to obtain the optimal transformation for the current sample, including:

[0087] The optimal pose transformation T * This can be seen as needing to start from The probability distribution parameters are estimated from the given probability distribution. The maximum likelihood estimation method is used to find the method that maximizes the probability of the current sample. probability T * , can be represented as:

[0088]

[0089] in This indicates the search for a pose transformation that maximizes the likelihood function. This represents the probability density function of the Gaussian random variable corresponding to the error term. This represents the search for a pose transformation that minimizes the objective function, T.** This represents the optimal pose transformation of the current sample, i.e., the preliminary optimized pose.

[0090] In the above exemplary embodiments, a method for preliminary pose optimization was introduced. Below, a method for obtaining potential dynamic semantic features will be introduced:

[0091] In this exemplary embodiment, the potential dynamic semantic features are obtained through the following methods:

[0092] Based on the preliminary optimized pose, the robust non-active dynamic points are transformed into the map coordinate system. The distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map is compared to obtain the potential dynamic semantic features.

[0093] In specific implementation, based on the preliminary optimized pose, the robust non-active dynamic points are transformed to the map coordinate system, including:

[0094] Using the aforementioned preliminary optimized pose, the robust non-active dynamic points in the current camera coordinate system are transformed to the map coordinate system to obtain robust non-active dynamic points in the map coordinate system.

[0095] In specific implementation, the distance between the robust non-active dynamic point center corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map is compared to obtain the potential dynamic semantic features, including:

[0096] For the semantic features of each object, calculate the center of the corresponding point in the robust non-active dynamic points in the map coordinate system and the center of the corresponding Gaussian ellipsoid in the current Gaussian map, and calculate the distance between the two centers. Mark the semantic features corresponding to objects whose distance exceeds a threshold to obtain the potential dynamic semantic features.

[0097] In the above exemplary embodiments, a method for obtaining latent dynamic semantic features was introduced. Below, a method for obtaining passive dynamic semantic features will be introduced:

[0098] In this exemplary embodiment, passive dynamic semantic features are obtained through the following methods:

[0099] Based on the semantic image of the keyframe sequence, the pixel regions corresponding to the potential dynamic semantic features are obtained, and the semantic loss change of the pixel regions between frames is calculated to obtain the passive dynamic semantic features.

[0100] In specific implementation, the pixel region corresponding to the potential dynamic semantic features is obtained, including:

[0101] Obtain the semantic image of each keyframe in the keyframe sequence, merge the pixel regions in the semantic image whose values ​​are the potential dynamic semantic features, and obtain the pixel regions corresponding to the potential dynamic semantic features in the keyframe sequence.

[0102] In practical implementation, the passive dynamic semantic features obtained include:

[0103] For each pixel region, render the semantic image of the current Gaussian map from the perspectives of the current frame and keyframes. Calculate the loss between the rendered semantic image and the input semantic image in that pixel region to obtain the semantic loss for the current frame and keyframes, which can be expressed as:

[0104]

[0105] Where S represents the input semantic image. N represents rendering a semantic image. s This indicates the number of valid pixels in the semantic image. This represents the L1 loss between the rendered semantic image and each valid pixel j of the input semantic image. This represents the semantic loss.

[0106] Calculate the magnitude of change between the semantic losses, identify the pixel regions whose magnitude of change is greater than a threshold, query the potential dynamic semantic features corresponding to the pixel regions, and obtain the passive dynamic semantic features.

[0107] In the above exemplary embodiments, a method for obtaining passive dynamic semantic features was introduced. Below, a method for obtaining the final dynamic object mask will be described:

[0108] In this exemplary embodiment, the final dynamic object mask is obtained through the following methods:

[0109] Query the pixel regions in the current frame semantic image whose values ​​are the passive dynamic semantic features, merge the pixel regions to obtain the passive dynamic object mask, and merge the passive dynamic object mask and the active dynamic object extended mask to obtain the final dynamic object mask.

[0110] In practical implementation, the passive dynamic object mask is obtained, including:

[0111] Determine whether each pixel value in the current frame semantic image belongs to the passive dynamic semantic feature, and obtain a binary result for each pixel. Combine the binary results to obtain the passive dynamic object mask.

[0112] In practical implementation, the final dynamic object mask is obtained, including:

[0113] The passive dynamic object mask and the active dynamic object extended mask are merged pixel by pixel to obtain the final dynamic object mask.

[0114] Step S130: Based on the final dynamic object mask, obtain the loss of the input image and the rendered image in the static pixel region, and backpropagate the loss using the differentiable rendering characteristics of the Gaussian ellipsoid to obtain a high-precision visual positioning result.

[0115] The following section will introduce high-precision visual positioning methods:

[0116] In this exemplary embodiment, high-precision visual positioning results are obtained through the following methods:

[0117] Based on the final dynamic object mask, the static pixel regions of the input image and the rendered image are preserved. The color, depth, and semantic losses of the static pixel regions are calculated. The losses are backpropagated using the differentiable rendering properties of the Gaussian ellipsoid to obtain high-precision visual positioning results.

[0118] In specific implementation, the static pixel regions of the input image and the rendered image are obtained, including:

[0119] The final dynamic object mask is applied to the input image and the rendered image, the dynamic pixel region values ​​are set to zero, and the static pixel region values ​​remain unchanged, thus obtaining the static pixel regions of the input image and the rendered image.

[0120] In specific implementation, the color, depth, and semantic losses of the static pixel region are calculated, including:

[0121] The calculation of the color, depth, and semantic loss of the static pixel region can be expressed as:

[0122]

[0123] Where C represents the input color image, N represents rendering a color image. r Indicates the number of valid pixels in a color image. This represents the l1 loss between the rendered color image and each valid pixel j of the input color image. This represents the pixel value of the final dynamic object mask at pixel j. This indicates the inverse operation. This refers to the color loss.

[0124]

[0125] Where D represents the input depth image. N represents the depth image being rendered. d This indicates the number of valid pixels in the depth image. This represents the L1 loss between the rendered depth image and each valid pixel j of the input depth image. This represents the pixel value of the final dynamic object mask at pixel j. This indicates the inverse operation. This represents the depth loss.

[0126]

[0127] Where S represents the input semantic image. N represents rendering a semantic image. s This indicates the number of valid pixels in the semantic image. This represents the L1 loss between the rendered semantic image and each valid pixel j of the input semantic image. This represents the pixel value of the final dynamic object mask at pixel j. This indicates the inverse operation. This represents the semantic loss.

[0128] In practice, high-precision visual positioning results are obtained, including:

[0129] A loss function composed of color, depth, and semantic losses of the static pixel region is constructed. Minimizing the loss function yields a high-precision visual localization result, which can be expressed as:

[0130]

[0131] in Let λ1 represent the color loss, and λ1 represent the weight of the color loss. Let λ1 represent the depth loss, and λ2 represent the weight of the depth loss. Let λ represent the semantic loss, and λ3 represent the weight of the semantic loss. This represents the loss function obtained by weighted summation of the color loss, the depth loss, and the semantic loss.

[0132] Step S140: Based on the high-precision visual positioning results, insert the static points into the Gaussian map, optimize the parameters of the Gaussian ellipsoid in the Gaussian map based on the loss, and obtain a high-fidelity map.

[0133] The following will describe the method for inserting the static points into a Gaussian map:

[0134] In this exemplary embodiment, the static points are inserted into the Gaussian map using the following method:

[0135] Based on the passive dynamic semantic features, the passive dynamic points in the robust non-active dynamic points are deleted to obtain static points. Based on the high-precision visual positioning results, the static points are inserted into the Gaussian map.

[0136] In practical implementation, static points are obtained, including:

[0137] Query the points that contain the passive dynamic semantic features in the robust non-active dynamic points to obtain passive dynamic points, delete the passive dynamic points to obtain static points.

[0138] In specific implementation, inserting the static points into the Gaussian map includes:

[0139] Based on the high-precision visual positioning results, the static points in the current camera coordinate system are transformed to the map coordinate system, and the static points in the map coordinate system are inserted into the Gaussian map.

[0140] In the above exemplary embodiments, a method for inserting the static points into a Gaussian map was described. Below, a method for optimizing a high-fidelity map will be described:

[0141] In this exemplary embodiment, the high-fidelity map is optimized using the following methods:

[0142] Based on the loss, the parameters of the Gaussian ellipsoid in the Gaussian map are optimized to obtain a high-fidelity map.

[0143] In practice, a high-fidelity map is obtained, including:

[0144] A loss function consisting of color, depth, and semantic losses of the static pixel region is constructed, and the loss function is minimized to optimize the high-fidelity map.

[0145] Step S150: Based on the loss, obtain high-loss objects, calculate the ray intersection region of the high-loss objects in the keyframe sequence, obtain multi-view ray intersection regions, and compact Gaussian ellipsoids based on the multi-view ray intersection regions to obtain a high-fidelity object-level map.

[0146] The following will introduce the method for obtaining the light intersection region:

[0147] In this exemplary embodiment, the multi-view light intersection region is obtained by the following method:

[0148] Based on the high-loss objects in the current frame, the ray intersection regions of the high-loss objects in the keyframe sequence are calculated to obtain the multi-view ray intersection regions.

[0149] In practice, the resulting multi-view light intersection area includes:

[0150] Based on the loss query, high-loss objects in the current frame image are obtained, and their semantic features are derived. The corresponding pixel regions of these semantic features in the keyframe sequence are acquired, and bounding boxes of the high-loss objects are generated based on these pixel regions in each keyframe. Based on the high-precision visual localization results, the rays on the bounding boxes are back-projected, and the ray intersection regions of the high-loss objects in the current frame and keyframe sequence views are calculated, resulting in multi-view ray intersection regions.

[0151] In the above exemplary embodiments, a method for obtaining multi-view ray intersection regions was introduced. Below, a method for obtaining a high-fidelity object-level map will be described:

[0152] In this exemplary embodiment, a high-fidelity object-level map is obtained through the following methods:

[0153] Based on the multi-view ray intersection region, the Gaussian ellipsoid inside the multi-view ray intersection region is densified to obtain a high-fidelity object-level map.

[0154] In practice, a high-fidelity object-level map is obtained, including:

[0155] The query center selects Gaussian ellipsoids within the multi-view light intersection area, filters Gaussian ellipsoids with gradients greater than a threshold, clones Gaussian ellipsoids with gradients greater than the threshold and scales smaller than the threshold, and splits Gaussian ellipsoids with gradients greater than the threshold and scales larger than the threshold to obtain a high-fidelity object-level map.

[0156] Based on the same inventive concept, corresponding to any of the above embodiments, this disclosure also provides a high-precision visual positioning and high-fidelity object-level map building device for indoor dynamic environments.

[0157] refer to Figure 2 The high-precision visual positioning and high-fidelity object-level map building device for indoor dynamic environments includes:

[0158] The visual sensor data preprocessing module 210 for indoor dynamic environments is configured to collect visual sensor data, including color images and depth images, segment and extract features from the color images based on a semantic segmentation model, obtain an active dynamic object extended mask and semantic image through deep clustering and feature encoding, and obtain robust non-active dynamic points based on the active dynamic object extended mask, semantic image, color image and depth image.

[0159] The dynamic object detection and deletion module 220 is configured to obtain a preliminary optimized pose based on robust non-active dynamic points and a Gaussian ellipsoid in a Gaussian map; calculate the distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map based on the preliminary optimized pose to obtain potential dynamic semantic features; obtain passive dynamic semantic features based on the semantic loss change of the corresponding pixel region in the keyframe sequence based on the potential dynamic semantic features; obtain a final dynamic object mask based on the passive dynamic semantic features and the active dynamic object extended mask; and obtain static points based on the passive dynamic semantic features and the robust non-active dynamic points.

[0160] The high-precision visual positioning module 230 is configured to obtain the loss of the input image and the rendered image in the static pixel region based on the final dynamic object mask, and backpropagate the loss using the differentiable rendering characteristics of the Gaussian ellipsoid to obtain the high-precision visual positioning result.

[0161] The high-fidelity map optimization module 240 is configured to insert the static points into the Gaussian map based on the high-precision visual positioning results, and optimize the parameters of the Gaussian ellipsoid in the Gaussian map based on the loss to obtain a high-fidelity map.

[0162] The high-fidelity object-level map densification module 250 is configured to obtain high-loss objects based on the loss, calculate the ray intersection regions of the high-loss objects in the keyframe sequence, obtain multi-view ray intersection regions, and densify Gaussian ellipsoids based on the multi-view ray intersection regions to obtain a high-fidelity object-level map.

[0163] In this exemplary embodiment, the visual sensor data preprocessing module 210 for indoor dynamic environments is specifically configured as follows:

[0164] The Grounding DINO model is used to detect instance objects in the current color image. SAM2 is used to obtain object-level masks. High-dimensional semantic features of each object are encoded based on a predefined encoding table to obtain a semantic image. Based on the object-level mask and the semantic image, an active dynamic object mask is obtained. The active dynamic object mask is extended based on the depth image to obtain an extended active dynamic object mask. Based on the extended active dynamic object mask, the depth image, and visual sensor intrinsic parameters, inactive dynamic points are obtained. Based on the coordinates and semantic features of the inactive dynamic points, edge noise in the inactive dynamic points is filtered out to obtain robust inactive dynamic points.

[0165] In this exemplary embodiment, the dynamic object detection and deletion module 220 is specifically configured as follows:

[0166] Calculate the covariance of each point in the robust non-active dynamic points, recover the center coordinates and covariance matrix of each Gaussian ellipsoid in the Gaussian map, use nearest neighbor search to obtain the correspondence between the robust non-active dynamic points and the Gaussian ellipsoids in the Gaussian map, construct an objective function, use maximum likelihood estimation to obtain the optimal transformation of the current sample, i.e., the preliminary optimized pose, transform the robust non-active dynamic points to the map coordinate system based on the preliminary optimized pose, compare the distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map to obtain potential dynamic semantic features, obtain the pixel regions corresponding to the potential dynamic semantic features based on the semantic image of the keyframe sequence, calculate the semantic loss change of the pixel regions between frames to obtain passive dynamic semantic features, query the pixel regions in the current frame semantic image with the value of the passive dynamic semantic features, merge the pixel regions to obtain the passive dynamic object mask, merge the passive dynamic object mask and the active dynamic object extended mask to obtain the final dynamic object mask.

[0167] In this exemplary embodiment, the high-precision visual positioning module 230 is specifically configured as follows:

[0168] Based on the final dynamic object mask, the static pixel regions of the input image and the rendered image are preserved. The color, depth, and semantic losses of the static pixel regions are calculated. The losses are backpropagated using the differentiable rendering properties of the Gaussian ellipsoid to obtain high-precision visual positioning results.

[0169] In this exemplary embodiment, the high-fidelity map optimization module 240 is specifically configured as follows:

[0170] Based on the passive dynamic semantic features, the passive dynamic points in the robust non-active dynamic points are deleted to obtain static points. Based on the high-precision visual positioning results, the static points are inserted into the Gaussian map. Based on the loss, the parameters of the Gaussian ellipsoid in the Gaussian map are optimized to obtain a high-fidelity map.

[0171] In this exemplary embodiment, the high-fidelity object-level map densification module 250 is specifically configured as follows:

[0172] Based on the loss query, high-loss objects in the current frame are queried, and the ray intersection regions of the high-loss objects in the keyframe sequence are calculated to obtain multi-view ray intersection regions. Based on the multi-view ray intersection regions, the Gaussian ellipsoids inside the multi-view ray intersection regions are densified to obtain a high-fidelity object-level map.

[0173] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0174] The apparatus described above is used to implement the corresponding high-precision visual positioning and high-fidelity object-level map construction method for indoor dynamic environments in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0175] Based on the same inventive concept, corresponding to the high-precision visual positioning and high-fidelity object-level map construction method for indoor dynamic environments described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to execute the high-precision visual positioning and high-fidelity object-level map construction method for indoor dynamic environments. Corresponding to the execution entity for each step in each embodiment of the high-precision visual positioning and high-fidelity object-level map construction method for indoor dynamic environments, the processor executing the corresponding step can belong to the corresponding execution entity.

[0176] The computer program product of the above embodiments is used to enable the computer and / or the processor to execute the high-precision visual positioning and high-fidelity object-level map construction method for indoor dynamic environments as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0177] Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0178] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0179] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0180] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0181] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

[0182] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for high-precision visual positioning and high-fidelity object-level map construction for indoor dynamic environments, characterized in that, include: Collect visual sensor data, including color images and depth images. Segment and extract features from the color images based on a semantic segmentation model. Obtain an active dynamic object extension mask and semantic image through deep clustering and feature encoding. Obtain robust non-active dynamic points based on the active dynamic object extension mask, semantic image, color image and depth image. A preliminary optimized pose is obtained based on robust inactive dynamic points and the Gaussian ellipsoid in the Gaussian map. Based on the preliminary optimized pose, the distance between the center of the robust inactive dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map is calculated to obtain potential dynamic semantic features. Based on the semantic loss change of the corresponding pixel region in the keyframe sequence based on the potential dynamic semantic features, passive dynamic semantic features are obtained. Based on the passive dynamic semantic features and the active dynamic object expansion mask, the final dynamic object mask is obtained. Based on the passive dynamic semantic features and the robust inactive dynamic points, static points are obtained. Based on the final dynamic object mask, the loss of the input image and the rendered image in the static pixel region is obtained. The loss is backpropagated using the differentiable rendering property of the Gaussian ellipsoid to obtain a high-precision visual positioning result. Based on the high-precision visual positioning results, the static points are inserted into the Gaussian map, and the parameters of the Gaussian ellipsoid in the Gaussian map are optimized based on the loss to obtain a high-fidelity map. Based on the loss, high-loss objects are obtained. The ray intersection regions of the high-loss objects in the keyframe sequence are calculated to obtain multi-view ray intersection regions. Based on the multi-view ray intersection regions, a Gaussian ellipsoid is compacted to obtain a high-fidelity object-level map.

2. The method according to claim 1, characterized in that, The collected visual sensor data includes color images and depth images. The color images are segmented and feature extracted based on a large semantic segmentation model. Active dynamic object extension masks and semantic images are obtained through deep clustering and feature encoding. Based on the active dynamic object extension masks, semantic images, color images, and depth images, robust non-active dynamic points are obtained, including: The Grounding DINO model is used to detect instance objects in the current color image. SAM2 is used to obtain object-level masks. The high-dimensional semantic features of each object are encoded based on a predefined encoding table to obtain a semantic image. Based on the object-level mask and the semantic image, an active dynamic object mask is obtained. The active dynamic object mask is then extended based on the depth image to obtain an active dynamic object extended mask. Based on the extended mask of the active dynamic object, the depth image, and the intrinsic parameters of the visual sensor, non-active dynamic points are obtained; based on the coordinates and semantic features of the non-active dynamic points, edge noise in the non-active dynamic points is filtered to obtain robust non-active dynamic points.

3. The method according to claim 1, characterized in that, The process involves obtaining a preliminary optimized pose based on robust inactive dynamic points and a Gaussian ellipsoid in a Gaussian map. Based on this preliminary optimized pose, the distance between the center of the robust inactive dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map is calculated to obtain latent dynamic semantic features. Based on the semantic loss change of the corresponding pixel region in the keyframe sequence according to these latent dynamic semantic features, passive dynamic semantic features are obtained. Based on these passive dynamic semantic features and the active dynamic object expansion mask, a final dynamic object mask is obtained. Finally, based on the passive dynamic semantic features and the robust inactive dynamic points, static points are obtained, including: Calculate the covariance of each point in the robust non-active dynamic point, recover the center coordinates and covariance matrix of each Gaussian ellipsoid in the Gaussian map, use nearest neighbor search to obtain the correspondence between the robust non-active dynamic point and the Gaussian ellipsoid in the Gaussian map, construct the objective function, use maximum likelihood estimation to obtain the best transformation of the current sample, i.e. the preliminary optimized pose. Based on the preliminary optimized pose, the robust non-active dynamic points are transformed into the map coordinate system. The distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map is compared to obtain the potential dynamic semantic features. Based on the semantic image of the keyframe sequence, the pixel region corresponding to the potential dynamic semantic feature is obtained, and the semantic loss change of the pixel region between frames is calculated to obtain the passive dynamic semantic feature. Query the pixel regions in the current frame semantic image whose values ​​are the passive dynamic semantic features, merge the pixel regions to obtain the passive dynamic object mask, and merge the passive dynamic object mask and the active dynamic object extended mask to obtain the final dynamic object mask.

4. The method according to claim 1, characterized in that, The loss in the static pixel region of the input image and the rendered image is obtained based on the final dynamic object mask. The loss is backpropagated using the differentiable rendering property of the Gaussian ellipsoid to obtain a high-precision visual positioning result, including: Based on the final dynamic object mask, the static pixel regions of the input image and the rendered image are preserved. The color, depth, and semantic losses of the static pixel regions are calculated. The losses are backpropagated using the differentiable rendering properties of the Gaussian ellipsoid to obtain high-precision visual positioning results.

5. The method according to claim 1, characterized in that, The process of inserting the static points into a Gaussian map based on the high-precision visual positioning results, and optimizing the parameters of the Gaussian ellipsoid in the Gaussian map based on the loss function to obtain a high-fidelity map includes: Based on the passive dynamic semantic features, the passive dynamic points in the robust non-active dynamic points are deleted to obtain static points. Based on the high-precision visual positioning results, the static points are inserted into the Gaussian map. Based on the loss, the parameters of the Gaussian ellipsoid in the Gaussian map are optimized to obtain a high-fidelity map.

6. The method according to claim 1, characterized in that, The process involves obtaining high-loss objects based on the aforementioned loss, calculating the ray intersection regions of these high-loss objects in the keyframe sequence to obtain multi-view ray intersection regions, and then compacting a Gaussian ellipsoid based on these multi-view ray intersection regions to obtain a high-fidelity object-level map, including: Based on the high-loss objects in the current frame, calculate the ray intersection region of the high-loss objects in the keyframe sequence to obtain the multi-view ray intersection region. Based on the multi-view ray intersection region, the Gaussian ellipsoid inside the multi-view ray intersection region is densified to obtain a high-fidelity object-level map.

7. A device for high-precision visual positioning and high-fidelity object-level map construction in indoor dynamic environments, characterized in that, include: A visual sensor data preprocessing module for indoor dynamic environments is configured to collect visual sensor data, including color images and depth images, segment and extract features from the color images based on a semantic segmentation model, obtain an active dynamic object extended mask and semantic image through deep clustering and feature encoding, and obtain robust non-active dynamic points based on the active dynamic object extended mask, semantic image, color image and depth image. The dynamic object detection and deletion module is configured to obtain a preliminary optimized pose based on robust non-active dynamic points and a Gaussian ellipsoid in a Gaussian map; calculate the distance between the center of the robust non-active dynamic point corresponding to each object and the center of the Gaussian ellipsoid in the Gaussian map based on the preliminary optimized pose to obtain potential dynamic semantic features; obtain passive dynamic semantic features based on the semantic loss change of the corresponding pixel region in the keyframe sequence based on the potential dynamic semantic features; obtain the final dynamic object mask based on the passive dynamic semantic features and the active dynamic object extended mask; and obtain static points based on the passive dynamic semantic features and the robust non-active dynamic points. The high-precision visual positioning module is configured to obtain the loss of the input image and the rendered image in the static pixel region based on the final dynamic object mask, and backpropagate the loss using the differentiable rendering characteristics of the Gaussian ellipsoid to obtain the high-precision visual positioning result. The high-fidelity map optimization module is configured to insert the static points into a Gaussian map based on the high-precision visual positioning results, and optimize the parameters of the Gaussian ellipsoid in the Gaussian map based on the loss to obtain a high-fidelity map. The high-fidelity object-level map densification module is configured to obtain high-loss objects based on the loss, calculate the ray intersection regions of the high-loss objects in the keyframe sequence, obtain multi-view ray intersection regions, and densify the Gaussian ellipsoid based on the multi-view ray intersection regions to obtain a high-fidelity object-level map.

8. A computer program product, characterized in that, It includes computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images