Multi-target scene visual SLAM (Simultaneous Localization and Mapping) method fusing target semantics and Gaussian splashing

Abstract

The invention discloses a multi-target scene visual SLAM (Simultaneous Localization and Mapping) method fusing target semantics and Gaussian splashing. The method comprises the steps that input data are preprocessed, two-dimensional space masks and semantic information corresponding to the two-dimensional space masks are extracted, global semantic identifiers are distributed to the two-dimensional space masks corresponding to a background and each target, and a target semantic segmentation map is obtained; reading first frame RGBD data as a current frame, acquiring a target semantic segmentation map corresponding to the current frame, and initializing camera pose parameters; respectively establishing an initial target Gaussian splashing model for the background and each target; adopting a Gaussian splash algorithm to render all the target Gaussian splash models to a next frame, and generating an RGB rendering image, a depth rendering image and a target semantic rendering image of the next frame; and designing a loss function between the next frame of rendered image and the corresponding real image, optimizing the pose of the camera, and updating the parameters of the target Gaussian splash model. According to the invention, simultaneous positioning and map construction of multiple target scenes are realized.

Description

Technical Field

[0001] The present invention belongs to the fields of deep learning and computer vision, and particularly relates to a multi-object scene visual SLAM method that fuses object semantics and Gaussian splashing. Background Art

[0002] Simultaneous Localization and Mapping (SLAM), as a core technology in the fields of computer vision and robotics, endows robots with the ability to simultaneously determine their own positions and construct maps of the surrounding environment in unknown environments, laying a solid foundation for the autonomous navigation, environmental perception, and intelligent interaction of robots. Traditional SLAM systems need to integrate various sensing technologies such as cameras, lidar, inertial measurement units (IMUs), and global positioning systems (GPS) to complete tasks, which are bulky and expensive. Visual SLAM, with the characteristics of less dependence and the ability to capture detailed information in the environment, has been gradually widely used. With the development of the field of deep learning computer vision, deep learning SLAM systems, typified by visual SLAM systems that combine Neural Radiance Fields (NeRFs), have become a hot topic in industry research and application due to advantages such as scene prediction and generating globally consistent and dense global maps.

[0003] With the introduction of Neural Radiance Field (NeRF) technology, research in the field of Simultaneous Localization and Mapping (SLAM) based on deep learning has shown an explosive growth trend. Among them, iMAP, as a pioneering achievement of NeRF-based SLAM systems, uses a single multi-layer perceptron (MLP) to represent the entire scene in the SLAM system. After optimization and improvement of the feature grid-based scene representation technology, NICE-SLAM emerged. Subsequently, ESLAM and Co-SLAM made subsequent progress in this regard. ESLAM uses three planes to improve real-time performance and reconstruction accuracy, while Co-SLAM uses joint coordinates and sparse parameter scenes to achieve accurate scene representation. At the same time, vMAP enhances the object reconstruction effect by constructing a separate MLP for the object. Even under limited indoor space conditions, these methods have also shown promising results in achieving high-quality reconstruction tasks.

[0004] As the latest achievement of the Gaussian splatting algorithm as a high-quality real-time new view synthesis method for 3D scenes, it is very different from the implicit scene representation based on the NeRF method. It uses explicit 3D Gaussians to represent the scene. The advantage of explicit parameter representation is that it allows people to manipulate the parameters more intuitively. Through differentiable Gaussian splatting algorithm for rendering, this method shows significant advantages in terms of computing speed, storage efficiency, and reconstruction accuracy, which makes it have better effects in the field of real-time rendering. On the contrary, current NeRF-based SLAM methods often encounter many obstacles in the actual application process because most of them use non-volume or implicit scene representations. The following are the problems existing in the current visual SLAM methods:

[0005] 1. Low storage and computing efficiency. Existing methods have insufficient parsing of scene details under the same conditions, slow training convergence and poor accuracy. Due to the limitations of the storage structure and algorithm, when dealing with large data, they occupy a lot of resources and take a long time, the feature extraction is inaccurate and incomplete, and it is difficult to capture the subtle texture structure changes, resulting in poor training.

[0006] 2. The problem of catastrophic forgetting of implicit fields. The scene representation based on neural implicit fields is difficult to apply to open scene reconstruction. Old information is easily lost during training, resulting in performance degradation, and due to its own characteristics and method limitations, it is difficult to achieve complete and accurate reconstruction for open scenes without prior information.

[0007] 3. Poor reduction of details of a single target object. Most existing methods focus on the reconstruction of the complete scene representation and cannot reconstruct a single target with high precision. They focus on the overall architecture outline and ignore the unique geometry, texture, material, etc. of the target, lacking a targeted mechanism and making it difficult to restore the fine structure texture of a single target. Summary of the Invention

[0008] In view of the deficiencies in the prior art, a multi-target scene visual SLAM method that fuses target semantics and Gaussian splatting is provided.

[0009] The present invention fuses target semantic information with the 3D Gaussian representation method. First, an initial target Gaussian splatting model is established for the background and each target in the multi-target scene. Through the pixel-level loss between the rendered image of the target Gaussian splatting model at the reference viewpoint and the real image, the parameters of the target Gaussian splatting model and the camera pose of the reference viewpoint are learned and optimized, so as to achieve simultaneous localization and mapping of the multi-target scene.

[0010] The technical solutions adopted by the present invention to solve its technical problems include the following steps:

[0011] Step S1: Input data preprocessing, extract the two-dimensional spatial mask and its corresponding semantic information, and assign a global semantic identifier to the two-dimensional spatial mask corresponding to the background and each target to obtain the target semantic segmentation map.

[0012] Step S2: Read the first-frame RGBD data as the current frame. Meanwhile, obtain the corresponding target semantic segmentation map of the current frame and initialize the camera pose parameters;

[0013] Step S3: Establish an initial target Gaussian splash model for the background and each target respectively;

[0014] Step S4: Predict the pose of the next frame according to the current frame pose. Adopt the Gaussian splash algorithm to render all the target Gaussian splash models onto the next frame, generating the RGB rendering map, depth rendering map, and target semantic rendering map of the next frame;

[0015] Step S5: Design the loss function between the next-frame rendering map and the corresponding real image;

[0016] Step S6: Optimize the camera pose of the next frame relative to the current frame;

[0017] Step S7: Update the parameters of the target Gaussian splash model;

[0018] Step S8: Take the next frame as the current frame and jump back to Step S4 until the last frame is processed.

[0019] The beneficial effects of the present invention are as follows:

[0020] The present invention provides a visual SLAM solution.

[0021] The present invention constructs an innovative deep learning framework focusing on multi-object reconstruction training. The present invention extracts the mask of each object in the RGB image through the SAM model and provides a cross-view semantic tracking strategy to assign a globally unique semantic identifier (ID) to each object in the 2D image. And a solution is proposed to fuse the target semantic features into an isotropic 3D Gaussian as the scene representation.

[0022] The present invention proposes an explicit multi-object representation method based on an isotropic 3D Gaussian integrating target semantic information. Compared with the implicit scene representation method and the representation method based on the voxel grid stage distance field, it has lower memory occupancy and faster rendering speed. In addition, the present invention uses a differentiable Gaussian splash algorithm for rendering, which can further optimize the memory and improve the reconstruction effect.

[0023] The present invention proposes a simultaneous localization and mapping (SLAM) system architecture based on underlying multi-object Gaussians. This architecture uses 3D Gaussians that integrate semantic information as the underlying scene representation form, and adopts a dual-main-thread architecture of pose tracking and reconstruction optimization. For each object in the scene, a 3D Gaussian cluster is constructed based on its semantic information. With the help of the differentiable Gaussian splashing algorithm, the learning and optimization of the entire SLAM system are realized based on gradients. After obtaining the optimized pose through the pose tracking thread, the reconstruction optimization thread optimizes the parameters of the target Gaussian splashing model according to the rendered image and the real image of the key frame set through pixel-level loss. Description of the Drawings

[0024] Figure 1 It is a flowchart.

[0025] Figure 2 It is a schematic diagram of the target Gaussian splashing model.

[0026] Figure 3 It is a schematic diagram of the main step flow of the present invention.

[0027] Figure 4 It is an RGB rendering effect diagram of the present invention.

[0028] Figure 5 It is a depth rendering effect diagram of the present invention. Detailed Implementation Manner

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings of the present invention.

[0030] As Figure 1 shown, a multi-object scene visual SLAM method that integrates target semantics and Gaussian splashing specifically includes the following steps:

[0031] Step S1: Input data preprocessing, extract the two-dimensional spatial mask and its corresponding semantic information, and assign a global semantic identifier to the background and each target for the corresponding two-dimensional spatial mask to obtain the target semantic segmentation map. The specific details are as follows:

[0032] The visual simultaneous localization and mapping (SLAM) system in the present invention takes a monocular RGBD data set as input, covering synthetic and real scenes. The data sources include Replica and TUM-RGBD. Among them, the synthetic scene data set Replica has high precision and complete depth maps, and is a commonly used benchmark data in the SLAM field; the TUM-RGBD data set in the real scene has problems with the poor quality of RGB maps and depth maps, and even the local area of the depth map fails. Subsequent targeted optimization is required.

[0033] First, deploy the SAM model (an open-source semantic segmentation model), select the "everything" mode to preprocess the input RGBD images, generate a two-dimensional spatial mask for the segmentation results of each object in each image, and obtain the corresponding semantic segmentation map of the image based on the two-dimensional spatial mask. In the 3D scene, for each 2D mask, a two-dimensional spatial mask with a globally unique semantic identifier (ID) needs to be assigned, and the 2D spatial masks with the same semantic information in different views need to be associated, while obtaining the total number N of objects (including the background) in the 3D scene.

[0034] According to the actual application scenario of this SLAM system, essentially, the input image sequence can be regarded as a set of video frame sequences with continuous view change characteristics. Based on this, a zero-shot tracker in the field of object tracking is introduced and trained. With the help of this tracker, the global association and propagation operations of the two-dimensional spatial masks are realized, and then the total number N of objects in the scene is obtained. The two-dimensional spatial masks corresponding to multiple object semantic identifiers are spliced to obtain the object semantic segmentation map, where the value of the two-dimensional spatial mask of the object is set to the globally consistent semantic identifier number. For the sake of simplifying the processing flow, in the present invention, the background is uniformly regarded as a special object, and the semantic identifier number is set to 0; for the remaining objects, non-background objects are assigned non-zero semantic identifier numbers starting from 1 and increasing to N - 1.

[0035] Step S2: Read the first-frame RGBD data as the current frame, and at the same time obtain the corresponding object semantic segmentation map of the current frame, and initialize the camera pose parameters. The specific details are as follows:

[0036] After completing the preprocessing operations on all inputs in step S1, the RGBD data of the first frame is read as the current frame for input. At the same time, the object semantic segmentation map corresponding to the first-frame image is extracted as the true value of the semantic information for calculating the semantic loss in the subsequent comparison with the semantic rendering map.

[0037] Synchronously initialize the camera pose of the first frame. In the SLAM system, the camera pose is represented by the camera spatial coordinates combined with quaternions. Set the camera position coordinates of the first frame as the origin of the world coordinate system, that is, the translation vector [0, 0, 0] T , and the rotation matrix is the identity matrix, corresponding to the quaternion q = [1, 0, 0, 0]. This is used as the reference state of the camera pose in the SLAM system to provide an initial reference for subsequent operations.

[0038] Step S3: Establish an initial object Gaussian splash model for the background and each object respectively. The specific details are as follows:

[0039] As Figure 2As shown, the present invention innovatively proposes an isotropic 3D Gaussian representation with semantic features, constructing a set of 3D Gaussians for each object in the scene. With the help of the SAM model's understanding of the 2D scene, semantic information is extracted and incorporated into the 3D feature space. On the basis of retaining the inherent attributes of the traditional 3D Gaussian distribution, such as spatial position, color, opacity, size, etc., the semantic information is encoded as a feature and implanted into the original 3D Gaussian, making it accurately correspond to the target object it belongs to. Finally, a set of 3D Gaussians is constructed separately for each object, forming an exclusive target Gaussian splash model, effectively enhancing the target reconstruction effect.

[0040] The present invention improves the original anisotropic 3D Gaussian, converts it into an isotropic 3D Gaussian representation, and simplifies the color information to be consistent in different views. The 3D Gaussian is simplified to be expressible with only 8 parameters: among them, the parameter c = {R, G, B} represents the color information corresponding to the 3D Gaussian; the spatial position is represented by the parameter , representing the specific coordinates of the 3D Gaussian in space; and the radius parameter r representing the Gaussian distribution limits its spatial influence range, and the opacity o ∈ [0, 1] is used for rendering control. Then, the opacity at any point in the 3D Gaussian space is shown as the following formula:

[0041]

[0042] Based on the global semantic identifier obtained in step 1, a new parameter of semantic feature encoding (ID Encoding) is introduced for each 3D Gaussian as a learnable attribute. The semantic feature encoding I is set as a learnable vector with a length of 16, where the length of 16 is a hyperparameter set based on experience, which is sufficient to distinguish different objects in the scene and will not occupy too much computing resources. The learnable Gaussian parameters of the 3D Gaussian fused with the global semantic identifier are represented by S i , where S i = {c i , μ i , r i , o i , I i} represents the parameter list of the i-th 3D Gaussian. Each 3D Gaussian is differentiably rendered through the Gaussian splash algorithm, thereby learning and optimizing the parameter list of the 3D Gaussian. Among them, the 3D Gaussians with the same semantic feature encoding I belong to the same target, forming the target Gaussian splash model of that target.

[0043] Based on all the pixels in the first frame, new Gaussian initialization is performed to complete the initial construction of the underlying target Gaussian splash model. For each pixel p, a 3D Gaussian is constructed, and its learnable Gaussian parameters are initialized as S i = {c i , μ i , ri , o i , I i}, where: color c i takes the same value as the pixel; spatial position μ i is set to the coordinates of the corresponding position of the back-projection of the pixel depth; radius indicates that the Gaussian projection back to the picture only covers one pixel; opacity o i is set to 0.5; semantic feature encoding I i is initialized to be the same as the semantics of the two-dimensional spatial mask where the pixel is located, and in this way, an accurate and frame-initial feature-fitting target Gaussian splash model is constructed, where D GT represents the true depth value corresponding to the pixel, and f represents the camera focal length.

[0044] Step S4: Predict the pose of the next frame according to the current frame pose, and use the Gaussian splash algorithm to render all the target Gaussian splash models to the next frame to generate the RGB rendering, depth rendering, and target semantic rendering of the next frame. The specific details are as follows:

[0045] The present invention uses a differentiable Gaussian splash algorithm. Based on the underlying target Gaussian splash model, it renders and generates the RGB rendering, depth rendering, and target semantic rendering according to the camera pose (camera parameters). Based on the current target Gaussian splash model, using the Gaussian splash algorithm, following the steps below, the corresponding renderings of the next frame are generated:

[0046] 4-1. Initialization of the camera pose of the next frame:

[0047] The pose parameters of the camera of the current frame are initialized using a forward projection method based on a constant velocity. According to the state of the camera of the current frame, the orientation of the camera pose of the next frame is set to the orientation obtained by rotating at the same angular velocity on the basis of the current frame. At the same time, its spatial coordinates are located at the next position reached from the current frame position at a uniform speed. This initialization process can be briefly described by the formula: E t+1 = E t +(E t+1 - E t ), to obtain the camera pose E t+1 .

[0048] 4-2. Obtain the RGB rendering of the next frame scene:

[0049] Through the initialized camera pose E t+1 and the color information c of the underlying target Gaussian splash model i, an RGB image is obtained through the Gaussian splashing algorithm. The specific steps are as follows: First, all the underlying 3D Gaussians are sorted from near to far according to the depth values in the camera coordinate system; then, in the pixel space, the alpha compositing operation is sequentially performed on the projections of each target 3D Gaussian in the pixel space to render the RGB image. The rendered color of each pixel p=(u, v) in the RGB image is expressed as follows:

[0050]

[0051] where f i (p) can be calculated by Equation 1, represents the product of the transparency-related factors from the first Gaussian distribution to the i-th Gaussian distribution, simulating the occlusion effect of the previous Gaussian distributions on the current Gaussian distribution. Among them, μ and r in the formula of f i (p) in the pixel space should be transformed as follows:

[0052]

[0053] where K is the known camera intrinsic matrix, E t+1 is the camera pose of the next frame of the camera, f is the known camera focal length, d=(E t+1 μ) i is the depth of the i-th 3D Gaussian in the camera coordinate system.

[0054] 4-3. Obtain the depth rendering map of the next frame of the scene:

[0055] By means of differentiable rendering, the same operation is performed on the depth component of the 3D Gaussian to obtain the depth prediction value of each pixel, thereby obtaining the rendered depth map. The formula is as follows:

[0056]

[0057] 4-4. Obtain the target semantic rendering map of the next frame of the scene:

[0058] In addition, a semantic mask map for each target needs to be rendered. By replacing the semantic feature encoding I in the 3D Gaussian of the target in the formula, the semantic mask map for each target is rendered. The formula is as follows:

[0059]

[0060] Among them, in the rendered semantic mask map, the semantic feature encoding of each pixel is determined by the visible-weighted summation of the semantic feature encodings of multiple Gaussians covering the pixel in the projection. The semantic mask maps of all targets constitute the target semantic rendering map of the next frame.

[0061] 4-5. Obtain the 3D Gaussian density map:

[0062] Similarly, the 3D Gaussian density GD at the active pixel p is derived for subsequent steps to determine whether the target Gaussian splash model has sufficient 3D Gaussians at the current position to fully represent the scene:

[0063]

[0064] Step S5: Design the loss function between the next frame's rendered image and the corresponding ground truth image, with specific details as follows:

[0065] As Figure 3 shown, thanks to the differentiable nature of the Gaussian splash algorithm, this SLAM system can directly solve the gradients of the underlying scene representation (the target Gaussian splash model) and camera parameters based on the error between the rendered image and the corresponding input data. Then, it iteratively updates the Gaussian distribution and camera parameters to continuously reduce the error, achieving precise fitting of the camera pose and constructing an accurate world map representation.

[0066] Based on the RGB rendered image, depth rendered image, and target semantic rendered image of the next frame obtained in step S4, it is easy to obtain the loss function of the system. According to the role of the loss function, it is refined into two categories: one is the scene reconstruction loss function, which is used for gradient optimization of the differentiable rendered image and RGBD input; the other is the target semantic feature loss function, which optimizes the semantic feature encoding of the target through SAM and two-dimensional semantic perception of global consistency. The following details the design details of the two types of loss functions:

[0067] 5-1. Scene reconstruction loss function:

[0068]

[0069] Among them, the RGB rendered image C(p), depth rendered image D(p), and Gaussian density GD(p) of the next frame are respectively derived according to formulas 2, 4, and 6. For the input of the (t + 1)-th frame, calculate the L1 loss between the RGB rendered image and depth rendered image and the ground truth values, and the color weight λ c is empirically set to take values in the interval [0.4, 0.5]. The Gaussian density GD(p) ∈ [0, 1], and a threshold u GD is set to 0.99, that is, the optimization operation is only performed on pixel positions where the Gaussian density reaches 99%. For pixel positions where the ground truth value of the depth map is invalid, set the L1 loss L1(D(p)) value to 0.

[0070] In addition, to ensure accurate environmental modeling with sufficient 3D Gaussians while avoiding excessive consumption of computing resources, an automatic 3D Gaussian update strategy is configured. Using spatial masking technology, it determines whether it is necessary to add new 3D Gaussians according to specific rules. The detailed determination process is shown in the following formula:

[0071] M(p) = (GD(p) < l GD ) + (D GT (p) < D(p))(L1(D(p)) > λMDE) (8)

[0072] The mask proposed by this formula consists of two parts. One is defined by (GD(p) < l GD ), representing the pixel positions with insufficient map density, where l GD is set to 0.5; the other is the position where there is still a geometry in front of the currently predicted geometry, that is, the true depth value D GT is before the depth prediction value D(p), and the position where the depth error L1(D(p)) is greater than λ times the median depth error MDE, where λ is empirically set to 50. For each pixel covered by this mask, a new 3D Gaussian is added according to the same steps as the initialization of the first frame.

[0073] 5-2. Semantic feature loss function:

[0074] Since the 2D spatial mask obtained by the SAM model based on semantic constraints is 2D and there are only N types of semantic label values, it is necessary to perform corresponding processing on the target semantic rendering map I(p) in formula (5). First, a linear layer T is introduced to restore the feature dimension of I(p) to N, then it is processed using the softmax function, and then the cross-entropy loss L CE is used to perform N-class classification on it. The specific formula is as follows:

[0075] L ID = L CE (softmax(T(I(p)))) (9)

[0076] The total rendering loss L render of the next frame input is the weighted sum of the scene reconstruction loss L t and the semantic feature loss L ID :

[0077] L render = L t+1 + λ ID L ID (10)

[0078] Among them, λ ID represents the weight parameter of the semantic feature loss.

[0079] Step S6: Optimize the camera pose of the next frame relative to the current frame. The specific details are as follows:

[0080] Based on the next-frame scene rendering image obtained in step S4 and the initialized camera pose, according to the scene reconstruction loss function given by formula 7, minimize this loss to obtain the optimized camera pose E for the next frame. t+1 , and the detailed process is as follows:

[0081] On the premise of keeping the parameters of the underlying target Gaussian splash model frozen, use the Adam optimizer to minimize the pixel-level error between the RGB rendering image and the depth rendering image and their true values in a gradient descent manner, and then optimize the camera pose E for the next frame. t+1 During the training process, the relevant settings of the Adam optimizer are as follows: the learning rate is set to lr = 0.004, and eps is set to 1e-15. During the optimization process, first set the maximum number of iterations to 60 times. When this maximum number of iterations is reached, check whether the error loss is less than the preset threshold. If this condition is not met, update the maximum number of iterations to twice the current value and repeat the above optimization operation until the error loss meets the requirements, that is, obtain the optimized camera pose E for the next frame. t+1 .

[0082] Step S7: Update the parameters of the target Gaussian splash model, and the specific details are as follows:

[0083] The core tasks of this step focus on two aspects. One is to determine whether it is necessary to add a new 3D Gaussian representation, and the other is to optimize the parameters of the underlying multi-target Gaussian.

[0084] 7-1. Determine whether to add a new 3D Gaussian representation: After obtaining the camera pose E for the next frame from step S6, t+1 use E t+1 to render a depth map according to formula 4 in combination with the current underlying Gaussian information. At the same time, introduce the true value of the input depth map, and analyze and judge whether each pixel point meets the conditions for adding a Gaussian pixel by pixel according to formula 8. If it needs to be added, the setting method of the parameters of the new Gaussian is the same as the method of adding a Gaussian when processing the first-frame image in step S3 of this process.

[0085] 7-2. Update of Underlying Gaussian Parameters: First, screen the set of key frames for the next frame. Save the input images at intervals of every 4 frames as candidate key frames, and then select 18 frames with the highest overlap with the next frame from all candidate key frames and put them into the set of key frames. The calculation of the overlap is based on the visibility of projecting the point cloud information of the current frame onto each candidate key frame. Subsequently, select the candidate key frame that is closest to the next frame and does not repeat with the selected 18 key frames, and combine this frame with the previously selected 18 key frames and the next frame to form a set of 20 key frames, which is used as the input data for optimizing the underlying Gaussian parameters. Subsequently, fix the camera parameters, and according to the loss functions proposed in Formula 7 and Formula 10, use the Adam optimizer to minimize the error to finely adjust the multi-object Gaussian parameters with semantic information at the underlying layer. When the number of input candidate key frames is less than 20 frames, all input frames are regarded as the set of key frames for the Gaussian parameter optimization process. In the training phase, calculate the pixel-level loss of the loss between the color, depth, and semantic rendering maps of all key frames corresponding to the next frame and the ground truth. The specific configuration of the Adam optimizer is as follows: the learning rate lr = 0.001, eps is set to 1e-15, and the setting strategy of its maximum number of iterations is the same as that in Step S6 to update the parameters of the underlying target Gaussian splash model.

[0086] Step S8: Take the next frame as the current frame and jump back to Step S4 until the last frame is processed. The specific details are as follows:

[0087] On the basis of completing the relevant operations in Steps S4 to S7, continuously take all RGBD frames as the new next frame and input them into the system for processing in turn. As the processing process of Steps S4 to S7 progresses, a multi-object Gaussian model covering the global scope is gradually constructed. Relying on this high-precision representation method and using the Gaussian splash algorithm, it is possible to accurately simulate and generate the RGB color image ( Figure 4 ) and the depth image ( Figure 5 ) of the scene from any desired perspective, and then achieve the goal of high-quality reconstruction of the target scene. The reconstruction results on the Replica dataset are shown in the following table (obtained by taking the average of 8 scenes in the dataset):

[0088] PSNR↑ SSIM↑ LPIPS↓ NICE-SLAM 24.61 0.83 0.22 vMAP 26.33 0.86 0.19 The present invention 34.43 0.95 0.10

[0089] Meanwhile, in the whole processing process, through in-depth mining and collaborative operation of the image data of each frame, the global camera pose information applicable to the entire SLAM system can also be accurately deduced, which reflects the movement trajectory of the camera in the scene. The pose estimation results on the TUM-RGBD dataset are shown in the following table:

[0090] ATERMSE iMAP NICE-SLAM vMAP The present invention fr1 / desk 4.64 2.71 2.67 2.12 fr2 / xyz 2.05 2.19 1.97 1.10 fr3 / office 5.95 3.04 3.10 3.12 Avg. 4.21 2.65 2.58 2.11

[0091] Meanwhile, since the underlying multi-object Gaussian representation also contains rich object semantic and other detailed information, on the one hand, it can provide a high-value data basis for further optimizing object details in the follow-up, making the object presentation more accurate and clear; on the other hand, it also opens up a broad space for scene editing operations, such as flexibly adding or subtracting scene elements and precisely adjusting the scene layout, greatly expanding the application potential of the present invention in the field of complex scene construction and interaction.

[0092] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, they can make several substitutions or modifications to these described embodiments, and these substitution or modification methods should all be regarded as belonging to the protection scope of the present invention.

[0093] The parts not detailed in the present invention belong to the well-known technology in the art.

Claims

1. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing, characterized in that: The steps include: Step S1: preprocessing the input data, extracting the two-dimensional spatial mask and its corresponding semantic information, and assigning a global semantic identifier to the background and each target corresponding to the two-dimensional spatial mask to obtain the target semantic segmentation map; Step S2: read the first frame RGBD data as the current frame, obtain the target semantic segmentation map corresponding to the current frame, and initialize the camera pose parameters; Step S3: establishing an initial target Gaussian splash model for the background and each target respectively; Step S4: predict the pose of the next frame according to the pose of the current frame, use the Gaussian splash algorithm to render all target Gaussian splash models to the next frame, and generate the RGB rendering image, depth rendering image and target semantic rendering image of the next frame; Step S5: design a loss function between the next frame rendering and the corresponding real image; Step S6: Optimizing the camera pose of the next frame relative to the current frame; Step S7: updating the parameters of the target Gaussian splash model; Step S8: Take the next frame as the current frame and jump back to step S4 until the last frame is processed.

2. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 1, characterized in that, Step S1 is specifically implemented as follows: First, deploy the SAM model, select the "everything" mode to preprocess the input RGBD image, generate a two-dimensional spatial mask of the segmentation results of each target object for each image, and obtain the semantic segmentation map corresponding to the image based on the two-dimensional spatial mask; in the 3D scene, assign a two-dimensional spatial mask with a globally unique semantic identifier to each 2D mask, and associate the two-dimensional spatial masks with the same semantic information in different views, and obtain the total number N of background and targets in the 3D scene; According to the actual application scenarios of the SLAM system, a zero-shot tracker in the field of target tracking is introduced and trained. With the help of this tracker, the global association and propagation operations of the two-dimensional spatial mask are realized, and then the total number N of targets in the scene is obtained; the two-dimensional spatial masks corresponding to multiple target semantic identifiers are spliced ​​to obtain a target semantic segmentation map, in which the value of the two-dimensional spatial mask of the target is set to a globally consistent semantic identifier number; in order to simplify the processing flow, the background is uniformly regarded as a special target, and the semantic identifier number is set to 0; for the remaining targets, non-background targets are assigned non-zero semantic identifier numbers starting from 1 and increasing to N-1.

3. A multi-target scene visual SLAM method integrating target semantics and Gaussian splattering according to claim 2, characterized in that, Step S2 is specifically implemented as follows: After the preprocessing operation is completed, the RGBD data of the first frame is read as the current frame input; at the same time, the target semantic segmentation map corresponding to the first frame image is extracted as the true value of the semantic information for the subsequent calculation of the semantic loss in the comparison with the semantic rendering image; Synchronously initialize the first frame camera pose. In the SLAM system, the camera pose is represented by the camera space coordinates combined with quaternions; set the first frame camera position coordinates to the origin of the world coordinate system, that is, the translation vector [0,0,0] T , the rotation matrix is ​​the unit matrix, corresponding to the quaternion q = [1,0,0,0], which is used as the reference state of the camera pose of the SLAM system to provide an initial reference for subsequent operations.

4. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 3, characterized in that, Step S3 is specifically implemented as follows: An isotropic 3D Gaussian representation with semantic features is proposed, and a set of 3D Gaussians is constructed for each target in the scene. With the help of the SAM model's understanding of the two-dimensional scene, the semantic information is extracted and integrated into the three-dimensional feature space. While retaining the spatial position, color, opacity, and size inherent properties of the traditional 3D Gaussian distribution, the semantic information is encoded as features and implanted into the original 3D Gaussian, so that it accurately corresponds to the target object. Finally, a 3D Gaussian set is constructed for each target separately to form an exclusive target Gaussian splash model.

5. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 4, characterized in that, The target Gaussian splash model is implemented as follows: The original anisotropic 3D Gaussian is improved and converted into an isotropic 3D Gaussian representation, and the color information is simplified to be consistent under different views; the 3D Gaussian is simplified to only need 8 parameters to express: where the parameter c = {R, G, B} represents the color information corresponding to the 3D Gaussian; the spatial position is determined by the parameter Indicates the specific coordinates of the 3D Gaussian in space; and the radius parameter r of the Gaussian distribution limits its spatial influence range, and the opacity o∈[0,1] is used for rendering control; then any point in the 3D Gaussian space The opacity at is given by: Based on the global semantic identification, a new parameter of semantic feature encoding is introduced for each 3D Gaussian as a learnable attribute; the semantic feature encoding I is set to a learnable vector of length 16; the learnable Gaussian parameter of the 3D Gaussian fused with the global semantic identification is denoted by S i Representative, where S i ={c i ,μ i ,r i ,o i ,I i } represents the parameter list of the i-th 3D Gaussian; each 3D Gaussian is rendered differently by the Gaussian splash algorithm, so as to learn and optimize the parameter list of the 3D Gaussian; wherein, 3D Gaussians with the same semantic feature encoding l belong to the same target and constitute the target Gaussian splash model of the target; The new Gaussian is initialized based on all the pixels in the first frame to complete the initial construction of the underlying target Gaussian splash model; for each pixel p, a 3D Gaussian is constructed, and its learnable Gaussian parameters are initialized to S i ={c i ,μ i ,r i ,o i ,I i }, where: color c i The value is the same as that of the pixel; the spatial position μ i Set to the coordinates of the pixel depth back projection corresponding to the position; radius Indicates that the Gaussian projection back to the image only covers one pixel; opacity o i Set to 0.5; semantic feature encoding I i Initialized to the same semantics as the two-dimensional space mask where the pixel is located, so as to construct a target Gaussian splash model initialization that is accurate and fits the characteristics of the first frame, where D GT Indicates the true depth value corresponding to the pixel, and f indicates the focal length of the camera.

6. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 5, characterized in that: Step S4 is specifically implemented as follows: 4-1. Initialize the camera pose for the next frame: The camera pose parameters of the current frame are initialized using the forward projection method based on constant speed. According to the state of the current frame camera, the orientation of the next frame camera pose is set to the orientation obtained after rotating at the same angular velocity based on the current frame. At the same time, its spatial coordinates are positioned at the next position reached from the current frame position at a uniform speed. This initialization process can be summarized as the formula: E t+1 =E t +(E t+1 -E t ), get the camera pose E of the next frame t+1 ; 4-2. Get the RGB rendering of the next frame of the scene: By initializing the camera pose E t+1 The color information c of the underlying target Gaussian splash model i , the RGB image is obtained through the Gaussian splash algorithm; the specific steps are: first, sort all the underlying 3D Gaussians from near to far according to the depth value in the camera coordinate system; then, perform alpha synthesis operations on the projection of each target 3D Gaussian in the pixel space in sequence in the pixel space, and render the RGB image; the rendering color of each pixel p = (u, v) in the RGB image is expressed as follows: where f i (p) can be calculated by formula 1, represents the product of the transparency-related factors from the first Gaussian distribution to the i-th Gaussian distribution, simulating the occlusion effect of the previous Gaussian distribution on the current Gaussian distribution; where f in the pixel space i (p) μ and r in the formula should be transformed as follows: Among them, K is the known camera intrinsic parameter matrix, E t+1 is the camera pose of the next frame, f is the known camera focal length, d = (E t+1 μ) i is the depth of the i-th 3D Gaussian in the camera coordinate system; 4-3. Get the depth rendering of the next frame of the scene: Through differentiable rendering, the same operation is performed on the depth component of the 3D Gaussian to obtain the depth prediction value of each pixel, thereby obtaining the rendered depth map. The formula is as follows: 4-4. Get the target semantic rendering of the next frame scene: By replacing the semantic feature encoding I in the 3D Gaussian of the target in the formula, the semantic mask map of each target is rendered; the formula is as follows: Among them, in the rendered semantic mask image, the semantic feature code of each pixel is determined by performing visibility weighted summation on the semantic feature codes of multiple Gaussians projected to cover the pixel; the semantic mask images of all targets constitute the target semantic rendering image of the next frame; 4-5. Get 3D Gaussian density map: Similarly, the 3D Gaussian density GD at the effect pixel p is derived for subsequent steps to determine whether the target Gaussian splash model has enough 3D Gaussians at the current position to fully express the scene:

7. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 6, characterized in that: Step S5 is specifically implemented as follows: Based on the RGB rendering, depth rendering and target semantic rendering of the next frame scene, the loss function of the system is obtained; according to the role of the loss function, it is refined into two categories: one is the scene reconstruction loss function, which is used to optimize the gradient of the differentiable rendering and the RGBD input; the other is the target semantic feature loss function, which optimizes the semantic feature encoding of the target through SAM and globally consistent two-dimensional semantic perception; The scene reconstruction loss function is: Among them, the RGB rendering image C(p), depth rendering image D(p) and Gaussian density GD(p) corresponding to the next frame are derived according to Formula 2, Formula 4 and Formula 6 respectively; for the input of the t+1th frame, the L1 loss of the RGB rendering image and the depth rendering image and the true value is calculated, and the color weight λ c According to practical experience, the value is taken in the interval [0.4, 0.5]; Gaussian density GD(p)∈[0,1], and the threshold u is set GD is 0.99, that is, the optimization operation is performed only for the pixel positions where the Gaussian density reaches 99%; for the pixel positions where the true value of the depth map is invalid, the L1 loss L1(D(p)) value is set to 0; In addition, in order to ensure sufficient 3D Gaussians to accurately model the environment while avoiding excessive consumption of computing resources, an automatic update strategy for 3D Gaussians is configured; using spatial mask technology, it is determined based on specific rules whether it is necessary to add new 3D Gaussians. The detailed determination process is shown in the following formula: M(p)=(GD(p)<l GD )+(D GT (p)<D(p))(L1(D(p))>λMDE) (8) The mask proposed by this formula consists of two parts, one of which is (GD(p) <l GD ) defines the pixel location where the map density is insufficient, where l GD is set to 0.5; the second is that there is still a position of geometry in front of the currently predicted geometry, that is, the true depth value D GT Located before the depth prediction value D(p) and the depth error L1(D(p)) is greater than λ times the median depth error MDE, where λ is empirically set to 50; for each pixel covered by this mask, add a new 3D Gaussian following the same steps as the first frame initialization.

8. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 7, characterized in that: The semantic feature loss function is: Since the two-dimensional spatial mask obtained by the semantic constraint-based SAM model is 2D and there are only N categories of semantic identification values, the target semantic rendering image I(p) in Formula 5 needs to be processed accordingly. First, a linear layer T is introduced to restore the feature dimension of I(p) to N, and then the softmax function is used for processing, and then the cross entropy loss L is used. CE It is classified into N categories, and the specific formula is as follows: L ID =L CE (softmax(T(I(p)))) (9) The total rendering loss L for the next frame input render is the scene reconstruction loss L t and semantic feature loss L ID The weighted sum of: L render =L t+1 +λ ID L ID (10) Among them, λ ID Represents the weight parameter of semantic feature loss.

9. A multi-target scene visual SLAM method integrating target semantics and Gaussian splatting according to claim 8, characterized in that: Step S6 is specifically implemented as follows: Based on the next frame scene rendering obtained in step S4 and the initialized camera pose, according to the scene reconstruction loss function given by formula 7, the optimized next frame camera pose E is obtained by minimizing the loss t+1 , the process is as follows: Under the premise of keeping the parameters of the underlying target Gaussian splash model frozen, the Adam optimizer is used to minimize the pixel-level errors between the RGB rendering and the depth rendering and the true value in a gradient descent manner, and then the camera pose E of the next frame is calculated. t+1 During the training process, the Adam optimizer is set as follows: the learning rate is set to lr = 0.004, and eps is set to 1e-15; during the optimization process, the maximum number of iterations is first set to 60 times. When the maximum number of iterations is reached, check whether the error loss is less than the preset threshold. If this condition is not met, update the maximum number of iterations to twice the current one, and repeat the above optimization operation until the error loss meets the requirements, that is, the optimized camera pose E of the next frame is obtained. t+1 .

10. A multi-target scene visual SLAM method integrating target semantics and Gaussian splashing according to claim 9, characterized in that: Step S7 is specifically implemented as follows: 7-1. Determine whether to add a new 3D Gaussian representation: Get the camera pose E of the next frame from step S6 t+1 After that, use E t+1 Combine the current underlying Gaussian information to render the depth map according to Formula 4; at the same time, introduce the true value of the input depth map, and analyze each pixel pixel by pixel according to Formula 8 to determine whether the conditions for adding a new Gaussian are met; if adding a new Gaussian is required, the parameter setting method of the new Gaussian is consistent with the method of adding a Gaussian when processing the first frame image in step S3 of this process; 7-2. Update of underlying Gaussian parameters: First, filter the key frame set of the next frame; save the input image as candidate key frames at intervals of every 4 frames, then select the 18 frames with the highest overlap with the next frame from all candidate key frames and put them into the key frame set, where the overlap is calculated based on the visibility of projecting the point cloud information of the current frame to each candidate key frame; then, select the candidate key frame that is closest to the next frame and does not overlap with the selected 18 key frames, combine this frame with the previously selected 18 key frames into the next frame, and together form a 20-frame key frame set, which is used as the input data for the optimization of the underlying Gaussian parameters; then, fix the camera parameters According to the loss functions proposed in Formula 7 and Formula 10, the Adam optimizer is used to minimize the error to fine-tune the underlying multi-target Gaussian parameters containing semantic information; when the number of candidate key frames input is less than 20 frames, all input frames are regarded as a set of key frames for the Gaussian parameter optimization process; in the training stage, the color, depth, and semantic rendering images of all key frames corresponding to the next frame are compared with the true value to calculate the pixel-level loss; the specific configuration of the Adam optimizer is: the learning rate lr = 0.001, eps is set to 1e-15, and the setting strategy of the maximum number of iterations is consistent with step S6, and the parameters of the underlying target Gaussian splash model are updated.

Images