Semantic SLAM method based on three-dimensional Gaussian sputtering

Abstract

The invention proposes a semantic SLAM method based on three-dimensional Gaussian sputtering, and relates to the field of three-dimensional Gaussian sputtering and SLAM, and the method comprises the steps: constructing a three-dimensional Gaussian distribution model fusing semantic features, representing a scene through a group of isotropic three-dimensional Gaussian distribution, obtaining the RGB-D data of the scene in real time through an RGB-D sensor, and carrying out the calculation of the RGB-D data. S2, generating a pixel-level semantic segmentation mask of the original data by using a pre-trained semantic segmentation network, projecting the three-dimensional Gaussian distribution model to a two-dimensional plane to generate a rendering data set, and obtaining a multi-modal data set by minimizing a joint loss function formed by RGB loss, depth loss and semantic loss between the rendering data set and the multi-modal data set in S2; and iteratively optimizing the sensor pose parameters and optimizable parameters of the three-dimensional Gaussian distribution model to finally obtain a complete sensor pose track and three-dimensional semantic Gaussian distribution map, so that the semantic consistency and real-time rendering efficiency of three-dimensional scene reconstruction are remarkably improved.

Description

Technical Field

[0001] The present application relates to the fields of three-dimensional Gaussian sputtering and SLAM, and in particular to a three-dimensional Gaussian sputtering semantic SLAM method that introduces semantic information. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology in robotics. By fusing sensor data to estimate the system's position and construct a map of the environment in real time, it provides fundamental support for autonomous navigation and interaction of intelligent systems in unknown scenarios. With the development of artificial intelligence and multimodal perception technologies, semantic SLAM introduces semantic information into the traditional SLAM framework, enabling object-level understanding of maps and further promoting its widespread application in areas such as autonomous driving, drones, and service robots.

[0003] In the autonomous driving sector, semantic SLAM, by integrating multimodal sensors such as lidar and cameras, enables dynamic obstacle recognition and high-precision positioning, providing critical support for path planning and safety decision-making in complex urban scenarios. In the drone sector, traditional GPS-based positioning methods are limited indoors or in densely populated areas. However, semantic SLAM-based solutions significantly enhance drone exploration and obstacle avoidance capabilities in signal-free environments by constructing three-dimensional semantic maps in real time. Service robots, leveraging semantic SLAM technology, implement semantically driven object search and task planning in scenarios such as homes and healthcare, significantly enhancing the intelligence of human-robot collaboration.

[0004] Although existing SLAM systems perform well in sparse map construction, efficiently generating dense maps containing semantic, geometric, and interaction information remains a core challenge in current research. Traditional methods are based on explicit representations of point clouds, voxels, or truncated signed distance fields (TSDFs), which suffer from high memory consumption and slow reconstruction speed, making them difficult to scale to large-scale scenes. In recent years, implicit representation methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting have attracted attention due to their high-fidelity rendering capabilities. However, NeRF-based semantic SLAM faces bottlenecks such as forgetting and cumulative drift in large scenes; while the 3D Gaussian Splatting method improves reconstruction efficiency through explicit parameterization, existing SLAM methods based on 3D Gaussian Splatting only contain color and geometric information and cannot support semantic rendering. Therefore, designing a dense semantic SLAM system that balances efficiency, accuracy, and semantic understanding capabilities is of great significance for promoting robot autonomy and scene intelligence. Summary of the Invention

[0005] The purpose of the invention is to address the above technical problems and provide a semantic SLAM method based on three-dimensional Gaussian sputtering, which can efficiently construct dense maps containing color, geometry and semantic information, and solve the bottlenecks of traditional methods in reconstruction accuracy, semantic fusion and adaptability to large scenes.

[0006] Technical solution, in order to achieve the above-mentioned invention object, the present invention proposes a semantic SLAM method based on three-dimensional Gaussian sputtering, the method comprising the following steps:

[0007] Step S1: Construct a 3D Gaussian distribution model that integrates semantic features. The scene is represented by a set of isotropic 3D Gaussian distributions. Each Gaussian distribution includes the following parameters: opacity α, center position μ, radius r, RGB color parameter c, and 3D semantic feature vector s. A differentiable rasterizer is designed for the 3D Gaussian distribution model that integrates semantic features, and a differentiable mapping of the 3D Gaussian distribution model to a 2D projection plane is achieved.

[0008] Step S2: RGB-D data of the scene is acquired in real time using an RGB-D sensor. A pre-trained semantic segmentation network is used to generate pixel-level semantic segmentation masks of the original data. A mapping matrix is then designed to map the semantic mask to a semantic feature map. A semantic consistency verification module is then designed to verify the contextual consistency of the semantic feature data, filter out semantically ambiguous areas, and ultimately construct a reliable multimodal data set.

[0009] Step S3: Based on the rasterizer renderer in step S1, the three-dimensional Gaussian distribution model is projected onto a two-dimensional plane to generate a rendering data set. By minimizing the joint loss function consisting of RGB loss, depth loss, and semantic loss between the rendering data set and the multimodal data set in S2, the sensor pose parameters and the optimizable parameters of the three-dimensional Gaussian distribution model are iteratively optimized to finally obtain a complete sensor pose trajectory and a three-dimensional semantic Gaussian distribution map.

[0010] Furthermore, the specific method of step S1 is as follows:

[0011] First, for the 3D map construction of the RGB scene, we use the same method as other SLAM methods based on 3D Gaussian sputtering technology: a set of 3D Gaussian distributions is used to construct the underlying map of the scene, and each Gaussian distribution affects a point in the 3D space. At the same time, in order to ensure the real-time performance of the SLAM system, these Gaussian distributions are constrained to be isotropic. Therefore, we have:

[0012]

[0013] where σ∈[0,1] represents opacity, represents the center position of Gaussian, r represents the radius;

[0014] In view of the lack of representation ability of the existing Gaussian distribution for semantic information, a set of three-dimensional feature vectors s is designed based on the above i =[ch1 i ,ch2 i ,ch3 i ] T As semantic labels, we embed Gaussian distribution parameters to obtain a three-dimensional Gaussian distribution model containing semantic information. At this time, each Gaussian distribution includes the following optimizable parameters: {σ, μ, c, s}, where c i =[r i ,g i ,b i ] T is the RGB color parameter;

[0015] In order to optimize the parameters of the semantic Gaussian distribution model to reconstruct three-dimensional RGB, depth and semantic scenes, a rasterizer is designed for the three-dimensional Gaussian distribution model containing semantic information, which is used to render the Gaussian distribution model into a two-dimensional data set. First, for RGB data and depth data, the existing differentiable rasterizer is used to obtain RGB renderings and depth renderings. Then a semantic rasterizer for semantic information is designed to obtain a semantic rendering. The core method is that, given a set of three-dimensional Gaussian distribution models and a set of sensor poses, the above-mentioned Gaussian influence function f(x) is used to sort and integrally project the three-dimensional Gaussian distribution along the depth dimension under the semantic pixel map coordinates, so as to realize the mapping of the three-dimensional Gaussian distribution into semantic rendering data of a two-dimensional plane. For the semantic rendering image, the pixel coordinates of a point p(x,y) are:

[0016]

[0017] where s i is the semantic feature vector value of the i-th Gaussian distribution, f 2D The calculation method of (p) is the same as f(x) above, but the three-dimensional space parameters μ and r in f(x) need to be converted into two-dimensional projection space parameters through the standard point rendering formula:

[0018]

[0019] Where K is the camera intrinsic parameter matrix, E t is the external parameter matrix of the sensor at the tth frame, f is the focal length of the sensor, d is the depth of the i-th Gaussian distribution in camera coordinates, and the vector s i For the semantic rendering graph, the pixel coordinates of a point p(x,y) are:

[0020]

[0021] Furthermore, the specific method of S2 is as follows:

[0022] The RGB-D data stream of the scene is synchronously acquired through the RGB-D sensor, which specifically includes RGB data and depth data Then, based on the pre-trained Encoder-Decoder structure semantic segmentation network model, the semantic mask data of the sensor raw data is obtained, and finally the semantic probability map is obtained. Where K is the number of semantic labels in the dataset during pre-training of the semantic segmentation network. Finally, the one-hot encoded semantic mask S is obtained based on the semantic probability map:

[0023]

[0024] In order to embed the semantic features into the three-dimensional Gaussian distribution model containing semantic information designed in S1, the mapping matrix is designed. One-hot encode semantic label s in K dimensions o Compressed into a three-dimensional feature vector s:

[0025] s=W×s o (8),

[0026] In order to make adjacent one-hot encoded semantic labels evenly mapped to the three-dimensional feature vector, for the column vector w of W i Design the following mapping rules:

[0027]

[0028] Finally, the semantic feature map S' is obtained from the one-hot encoded semantic mask map S.

[0029] In view of the fact that SLAM methods are often used for large-scale and multi-angle movement in complex scenes, a semantic consistency module is designed to prevent the semantic segmentation network from generating unstable semantic mask images during movement, thereby fusing incorrect semantic information into the Gaussian distribution model. In the semantic consistency module, the contextual consistency of the semantic feature data is verified. Specifically, the edge alignment score is first designed. The higher the edge alignment score, the more consistent the semantic edge is with the RGB edge, and the higher the weight is given in the semantic context consistency verification. The edge detection algorithm is used to extract the RGB image I rgb and the edge graph of the semantic graph S', respectively denoted as E rgb and E sem . Then define the edge alignment score:

[0030]

[0031] Where N(x,y) is the N×N local area around (x,y). Then, a context consistency verification method for semantic masks is designed. First, the position (x,y) of the pixel point (x,y) of the current frame in the previous k frame is calculated. k ,y k ):

[0032] x k =xv x ·(tk)·△t

[0033] y k =yv y ·(tk)·△t (11),

[0034] where v x ,v y is the pixel motion speed calculated by the optical flow algorithm, and Δt is the frame interval. Next, we set the time window size T and obtain the semantic label of the corresponding position for each historical surrounding frame k∈[tT,t]. Finally, we combine the edge alignment score and perform a weighted vote on the semantic graph within the time window, selecting the label with the highest score as the correction result:

[0035] Where S'(x,y,k) represents the semantic feature label of the pixel (x,y) in the first k frames, l is the candidate semantic feature label, λ is the edge consistency weight, δ(a,b) is the indicator function: 1 when a=b, otherwise 0, w d is the time decay weight. In addition, for the non-integer coordinate value (x' k ,y' k ), bilinear interpolation is used to calculate the label weight of the position:

[0036]

[0037] Among them, λ i,j Is the coordinate (x k ,y k ) is the weight of the Euclidean distance distribution with four adjacent integer coordinate points in two-dimensional space. Finally, I rgb 、D raw With S cor Composed of original data sets.

[0038] Furthermore, the specific method of S3 is as follows:

[0039] Based on the semantic Gaussian distribution model in S1 and the corresponding rasterizer, a complete semantic SLAM system is constructed.

[0040] For the posture tracking task, the core method is: for the sensor posture E at time t t, using the original data set and the rendered data set obtained by rasterizer rendering, a joint optimizer is constructed. By back-propagating the gradient, the RGB, depth, and semantic joint loss between the original data set and the rendered data set obtained by rasterizer rendering is minimized, and the pose E is iteratively optimized. t :

[0041]

[0042] in and are the values of pixel p in the original data set and the rendered data set, respectively, c ,λ D and λ S are the weights of RGB loss, depth loss, and semantic loss respectively.

[0043] During pose tracking, keyframes are selected for 3D map reconstruction. First, candidate RGB, depth, and semantic frames are synchronously selected at a constant time interval and stored in the corresponding candidate keyframe sequence. In parallel, the candidate frame queue is filtered along the semantic dimension. First, for each frame in the candidate queue, its semantic feature frequency vector is counted:

[0044] v t =[n c1 ,n c2 ...,n cC ] (15),

[0045] Where C is the number of semantic categories in the pre-trained semantic segmentation network, n ck is the candidate semantic key frame S at time t t The number of pixels with category ck is then normalized to obtain the frequency vector Finally, compare the current semantic candidate frame S t and the previous semantic candidate frame S t-1 Cosine similarity of:

[0046]

[0047] Compare the cosine similarity with the set threshold θ. t When >θ, it means that the semantic similarity between the current candidate frame and the previous candidate frame is too large. At this time, the RGB, depth, and semantic candidate frames at time t in the candidate frame queue are synchronously filtered out. Otherwise, all candidate frames at time t are retained as key frames.

[0048] For the map construction task, the core is to synchronously construct a three-dimensional RGB map, a depth map, and a semantic map by using a three-dimensional Gaussian distribution model with semantics. Specifically, first given the first frame data, all pixels of the first frame data are constructed with a corresponding three-dimensional Gaussian distribution to obtain an initialized Gaussian distribution model. Subsequently, each time a new frame of data is obtained, a new Gaussian distribution is initialized in the area of the map where the Gaussian distribution density is lower than the set threshold. For the Gaussian distribution introduced into the three-dimensional model, a joint optimizer is constructed based on the original data set at time t and the rendered data set obtained by rendering the current three-dimensional Gaussian distribution model through the rasterizer. The map reconstruction loss function is:

[0049]

[0050] Among them L C and L S They are the SSIM weighted losses for RGB images and semantic images respectively:

[0051]

[0052] Similar to the pose tracking task, by backpropagating the gradient, minimizing the map reconstruction loss, and iteratively optimizing all variable parameters of the Gaussian distribution, we can finally obtain the converged 3D RGB map, depth map, and semantic map simultaneously.

[0053] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0054] (1) This paper designs a 3D Gaussian distribution model that incorporates semantic information. By embedding semantic feature vectors into a Gaussian parameter system, it achieves a synchronized 3D joint representation of geometric, color, and semantic information. This overcomes the limitations of traditional Gaussian sputtering methods in semantic modeling and significantly improves the semantic consistency and real-time rendering efficiency of 3D scene reconstruction.

[0055] (2) This paper designs a semantic SLAM system framework based on three-dimensional Gaussian sputtering. It extracts semantic masks through a pre-trained semantic segmentation network and combines semantic context consistency verification to effectively filter semantic noise in dynamic interference scenarios. At the same time, a joint loss function is designed to synchronously optimize sensor pose and Gaussian distribution parameters. In addition, key frames are dynamically selected based on semantic features, effectively reducing the computational load of redundant data and significantly improving system processing efficiency. While ensuring the accuracy of semantic modeling, it achieves real-time pose estimation and map updates in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a system flow chart of a semantic SLAM method based on three-dimensional Gaussian sputtering. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] like Figure 1 As shown, the present invention proposes a semantic SLAM method based on three-dimensional Gaussian sputtering, which includes the following steps:

[0059] Step S1: First, for the 3D map construction of the RGB scene, a method consistent with other SLAM methods based on 3D Gaussian sputtering technology is used: a set of 3D Gaussian distributions is used to construct the underlying map of the scene, and each Gaussian distribution affects a point in the 3D space. At the same time, in order to ensure the real-time performance of the SLAM system, these Gaussian distributions are constrained to be isotropic. Therefore, we have:

[0060]

[0061] where σ∈[0,1] represents opacity, Indicates the center position of the Gaussian, r represents the radius. In addition, each Gaussian distribution has a set of RGB color parameters c i =[r i ,g i ,b i ] T .

[0062] In view of the lack of representation ability of the existing Gaussian distribution for semantic information, a set of three-dimensional feature vectors s is designed based on the above i =[ch1 i ,ch2 i ,ch3 i ] T The semantic labels are embedded in the Gaussian distribution parameters to obtain a three-dimensional Gaussian distribution model containing semantic information. At this time, each Gaussian distribution includes the following optimizable parameters: {σ, μ, c, s}.

[0063] In order to optimize the parameters of the semantic Gaussian distribution model to reconstruct three-dimensional RGB, depth and semantic scenes, a rasterizer is designed for the three-dimensional Gaussian distribution model containing semantic information, which is used to render the Gaussian distribution model into a two-dimensional data set. First, for RGB data and depth data, the existing differentiable rasterizer is used. The core method is that, given a set of three-dimensional Gaussian distribution models and a set of sensor poses, the three-dimensional Gaussian distribution is sorted and integrally projected along the depth dimension under the pixel coordinates using the above-mentioned Gaussian influence function f(x), so as to map the three-dimensional Gaussian distribution into RGB and depth rendering data of a two-dimensional plane. For the RGB rendering image, the pixel coordinates of a point p(x,y) are:

[0064]

[0065] where c i is the color parameter value of the i-th Gaussian distribution, f 2D The calculation method of (p) is the same as f(x) above, but the three-dimensional space parameters μ and r in f(x) need to be converted into two-dimensional projection space parameters through the standard point rendering formula:

[0066]

[0067] Where K is the camera intrinsic parameter matrix, E t is the external parameter matrix of the sensor at the tth frame, f is the focal length of the sensor, and d is the depth of the i-th Gaussian distribution in camera coordinates. Similarly, for the depth rendering image, the pixel coordinates of a point p(x,y) are:

[0068]

[0069] where d i Is the depth parameter value of the i-th Gaussian distribution. Based on this, the same projection method is used to construct the semantic feature vector s for the above embedded three-dimensional Gaussian distribution using f(x) i For the semantic rendering graph, the pixel coordinates of a point p(x,y) are:

[0070]

[0071] Step S2: synchronously obtain the RGB-D data stream of the scene through the RGB-D sensor, which specifically includes RGB data and depth data Then, based on the pre-trained semantic segmentation network model, the semantic mask data of the sensor raw data is obtained. Specifically, based on the SegFormer network, the core process is as follows: First, the original data I is converted into rgb Encoded into multi-scale feature maps:

[0072] F1,F2,F3,F4=MixTransformer(I rgb )(6),

[0073] in is the feature map of layer i. Then, through the decoder part of the SegFormer network, MLP is used to unify all F i The dimension is then spliced, up-sampled and processed by multi-layer MLP to obtain the semantic probability map. Where K is the number of semantic labels in the dataset during pre-training of the semantic segmentation network. Finally, the one-hot encoded semantic mask S is obtained based on the semantic probability map:

[0074]

[0075] In order to embed the semantic features into the three-dimensional Gaussian distribution model containing semantic information designed in S1, the mapping matrix is designed. One-hot encode semantic label s in K dimensions o Compressed into a three-dimensional feature vector s:

[0076] s=W×s o (8),

[0077] In order to make adjacent one-hot encoded semantic labels evenly mapped to the three-dimensional feature vector, for the column vector w of W i Design the following mapping rules:

[0078]

[0079] Finally, the semantic feature map S' is obtained from the one-hot encoded semantic mask map S.

[0080] Given that SLAM methods are often used for large-scale, multi-angle movement in complex scenarios, a semantic consistency module is designed. In the semantic consistency module, the contextual consistency of the semantic feature data is verified. Specifically, the edge alignment score is first designed:

[0081]

[0082] Among them E rgb and E sem Use edge detection algorithm to extract RGB image I rgb And the edge map of the semantic map S', N(x,y) is the N×N local area around the pixel coordinate (x,y). Then the context consistency verification method of the semantic mask is designed. First, the position (x) of the pixel point (x,y) of the current frame in the previous k frame is calculated. k ,y k ):

[0083] x k =xv x ·(tk)·△t

[0084] y k =yv y ·(tk)·△t (11),

[0085] where v x ,v yis the pixel motion speed calculated by the optical flow algorithm, and Δt is the frame interval. Next, we set the time window size T and obtain the semantic label of the corresponding position for each historical surrounding frame k∈[tT,t]. Finally, we combine the edge alignment score and perform a weighted vote on the semantic graph within the time window, selecting the label with the highest score as the correction result:

[0086] Where S'(x,y,k) represents the semantic feature label of the pixel (x,y) in the first k frames, l is the candidate semantic feature label, λ is the edge consistency weight, δ(a,b) is the indicator function: 1 when a=b, otherwise 0, w d is the time decay weight. In addition, for the non-integer coordinate value (x' k ,y' k ), bilinear interpolation is used to calculate the label weight of the position:

[0087]

[0088] Among them, λ i,j Is the coordinate (x k ,y k ) is the weight of the Euclidean distance distribution with four adjacent integer coordinate points in two-dimensional space. Finally, I rgb 、D raw With S cor Composed of original data sets.

[0089] Step S3: Based on the semantic Gaussian distribution model in S1 and the corresponding rasterizer, a complete semantic SLAM system is constructed.

[0090] For the pose tracking task, the specific method is as follows: given the first frame data, record the sensor pose E0 at this time. At the next time t, estimate the sensor pose E corresponding to the RGB-D view of the tth frame at this time t , whose initial posture is obtained by the previous frame posture E t-1 It is determined by superimposing a displacement of constant speed forward. Afterwards, a joint optimizer is constructed for the original data set at time t and the rendered data set obtained by rasterizer rendering. By backpropagating the gradient, minimizing the RGB, depth, and semantic joint loss, the pose E is iteratively optimized. t :

[0091]

[0092] in and are the values of pixel p in the original data set and the rendered data set, respectively, c ,λ D and λ Sare the weights of RGB loss, depth loss, and semantic loss respectively.

[0093] During pose tracking, keyframes are selected for 3D map reconstruction. First, candidate RGB, depth, and semantic frames are synchronously selected at a constant time interval and stored in the corresponding candidate keyframe sequence. In parallel, the candidate frame queue is filtered along the semantic dimension. First, for each frame in the candidate queue, its semantic feature frequency vector is counted:

[0094] v t =[n c1 ,n c2 ...,n cC ] (15),

[0095] Where C is the number of semantic categories in the pre-trained semantic segmentation network, n ck is the candidate semantic key frame S at time t t The number of pixels with category ck is then normalized to obtain the frequency vector Finally, compare the current semantic candidate frame S t and the previous semantic candidate frame S t-1 Cosine similarity of:

[0096]

[0097] Compare the cosine similarity with the set threshold θ. t >θ, the RGB, depth, and semantic candidate frames at time t in the candidate frame queue are synchronously filtered out; otherwise, all candidate frames at time t are retained as key frames.

[0098] For the map construction task, the core is to synchronously construct a three-dimensional RGB map, a depth map, and a semantic map by using a three-dimensional Gaussian distribution model with semantics. Specifically, first given the first frame data, all pixels of the first frame data are constructed with a corresponding three-dimensional Gaussian distribution to obtain an initialized Gaussian distribution model. Subsequently, each time a new frame of data is obtained, a new Gaussian distribution is initialized in the area of the map where the Gaussian distribution density is lower than the set threshold. For the Gaussian distribution introduced into the three-dimensional model, a joint optimizer is constructed based on the original data set at time t and the rendered data set obtained by rendering the current three-dimensional Gaussian distribution model through the rasterizer. The map reconstruction loss function is:

[0099]

[0100] Among them L C and L S They are the SSIM weighted losses for RGB images and semantic images respectively:

[0101]

[0102] Similar to the pose tracking task, by backpropagating the gradient, minimizing the map reconstruction loss, and iteratively optimizing all variable parameters of the Gaussian distribution, we can finally obtain the converged 3D RGB map, depth map, and semantic map simultaneously.

[0103] The above description is merely the content and implementation methods of the present invention. Although the description is relatively specific and detailed, it should not be construed as limiting the scope of the patent. It should be noted that a person of ordinary skill in the art may make a number of variations and improvements without departing from the concept of the present application, and these variations and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the patent application shall be based on the appended claims.

Claims

1. A semantic SLAM method based on three-dimensional Gaussian sputtering, characterized in that: The steps include: Step S1: Construct a 3D Gaussian distribution model that integrates semantic features. The scene is represented by a set of isotropic 3D Gaussian distributions. Each Gaussian distribution includes the following parameters: opacity α, center position μ, radius r, RGB color parameter c, and 3D semantic feature vector s. A differentiable rasterizer is designed for the 3D Gaussian distribution model that integrates semantic features, and a differentiable mapping of the 3D Gaussian distribution model to a 2D projection plane is achieved. Step S2: Acquire the RGB-D data of the scene and use the pre-trained semantic segmentation network to generate a pixel-level semantic segmentation mask of the original data. Then, design a mapping matrix to map the semantic mask into a semantic feature map. Design a semantic consistency verification module to verify the contextual consistency of the semantic feature data, filter out semantically ambiguous areas, and finally construct a reliable multimodal data set. In step S3, based on the rasterizer renderer in step S1, the three-dimensional Gaussian distribution model is projected onto a two-dimensional plane to generate a rendering data set. By minimizing the joint loss function consisting of RGB loss, depth loss, and semantic loss between the rendering data set and the multimodal data set in step S2, the sensor pose parameters and the optimizable parameters of the three-dimensional Gaussian distribution model are iteratively optimized, and finally a complete sensor pose trajectory and a three-dimensional semantic Gaussian distribution map are obtained.

2. A semantic SLAM method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The specific method of step S1 is as follows: (1) First, for the 3D map construction of the RGB scene, a set of 3D Gaussian distributions is used to construct the underlying map of the scene. Each Gaussian distribution will affect a point in the 3D space. At the same time, in order to ensure the real-time performance of the SLAM system, these Gaussian distributions are constrained to be isotropic, so: where σ∈[0,1] represents opacity, represents the center position of Gaussian, r represents the radius; (2) Based on the above, a set of three-dimensional feature vectors s is designed. i =[ch1 i ,ch2 i ,ch3 i ] T As semantic labels, we embed Gaussian distribution parameters to obtain a three-dimensional Gaussian distribution model containing semantic information. At this time, each Gaussian distribution includes the following optimizable parameters: {σ, μ, c, s}, where c i =[r i ,g i ,b i ] T is the RGB color parameter; (3) Design a differentiable rasterizer for a three-dimensional Gaussian distribution model containing semantic information.

3. A semantic SLAM method based on three-dimensional Gaussian sputtering according to claim 2, characterized in that, The specific method of step (3) is as follows: first, for RGB data and depth data, use the existing differentiable rasterizer to obtain RGB rendering and depth rendering, then design a rasterizer for semantic information to obtain a semantic rendering. Given a set of three-dimensional Gaussian distribution models and a set of sensor poses, use the above-mentioned Gaussian influence function f(x) to sort the three-dimensional Gaussian distribution along the depth dimension under the semantic pixel map coordinates and integrally project it to achieve mapping the three-dimensional Gaussian distribution to semantic rendering data on a two-dimensional plane. For the semantic rendering, the pixel coordinates of a point p(x, y) are: where s i is the semantic feature vector value of the i-th Gaussian distribution, f 2D The calculation method of (p) is the same as f(x) above, but the three-dimensional space parameters μ and r in f(x) need to be converted into two-dimensional projection space parameters through the standard point rendering formula: Where K is the camera intrinsic parameter matrix, E t is the extrinsic parameter matrix of the sensor at the tth frame, f is the focal length of the sensor, and d is the depth of the i-th Gaussian distribution in camera coordinates.

4. A semantic SLAM method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The specific method of step S2 is as follows: (1) Synchronously obtain the RGB-D data stream of the scene through the RGB-D sensor, which specifically includes RGB data and depth data (2) Based on the pre-trained semantic segmentation network model, obtain the semantic mask data of the sensor raw data; (3) In view of the fact that SLAM methods are often used for large-scale and multi-angle movements in complex scenes, a semantic consistency module is designed to verify the contextual consistency of semantic feature data.

5. A semantic SLAM method based on three-dimensional Gaussian sputtering according to claim 4, characterized in that, The step (2) first uses the existing semantic segmentation network to process the original RGB image to obtain a semantic probability map Where K is the number of semantic labels in the dataset during pre-training of the semantic segmentation network, and then the one-hot encoded semantic mask map S is obtained based on the semantic probability map: Next, in order to embed the semantic features into the three-dimensional Gaussian distribution model containing semantic information designed in step S1, the mapping matrix is designed One-hot encode semantic label s in K dimensions o Compressed into a three-dimensional feature vector s: s=W×s o (8), For the column vector w of W i Design the following mapping rules: Finally, the semantic feature map S' is obtained from the one-hot encoded semantic mask map S.

6. A semantic SLAM method based on three-dimensional Gaussian sputtering according to claim 4, characterized in that, Step (3) involves first designing an edge alignment score: Among them E rgb and E sem Use edge detection algorithm to extract RGB image I rgb And the edge map of the semantic map S', N(x,y) is the N×N local area around the pixel coordinate (x,y), and then the context consistency verification method of the semantic mask is designed. First, the position (x k ,y k ): x k =x-v x ·(t-k)·△t y k =y-v y ·(t-k)·△t (11), where v x ,v y is the pixel motion speed calculated by the optical flow algorithm, △t is the frame interval time, and then the time window size T is set. For each historical surrounding frame k∈[tT,t], the semantic label of the corresponding position is obtained. Finally, combined with the edge alignment score, a weighted vote is performed on the semantic graph within the time window, and the label with the highest score is selected as the correction result: Where S'(x,y,k) represents the semantic feature label of the pixel (x,y) in the first k frames, l is the candidate semantic feature label, λ is the edge consistency weight, δ(a,b) is the indicator function: 1 when a=b, otherwise 0, w d is the time decay weight. In addition, for the non-integer coordinate value (x' k ,y' k ), bilinear interpolation is used to calculate the label weight of the position: Among them, λ i,j Is the coordinate (x k ,y k ) is the weight of the Euclidean distance distribution with four adjacent integer coordinate points in two-dimensional space, and finally uses I rgb 、D raw With S cor Composed of original data sets.

7. A semantic SLAM method based on three-dimensional Gaussian sputtering according to claim 1, characterized in that, The specific method of step S3 is as follows: Based on the semantic Gaussian distribution model and the corresponding rasterizer in step S1, a complete semantic SLAM system is constructed; (1) For the posture tracking task, the core is the sensor posture E at time t. t , using the original data set and the rendered data set obtained by rasterizer rendering, a joint optimizer is constructed to minimize the RGB, depth, and semantic joint loss between the original data set and the rendered data set obtained by rasterizer rendering by back-propagating gradients, and iteratively optimize the pose E t : in and are the values of pixel p in the original data set and the rendered data set, respectively, c ,λ D and λ S are the weights of RGB loss, depth loss, and semantic loss respectively; (2) During the pose tracking process, the key frames for 3D map reconstruction are selected, and the RGB, depth, and semantic candidate frames are synchronously selected at a constant time interval and stored in the corresponding candidate key frame sequence. The candidate frame queue is then screened in the semantic dimension. First, the semantic feature frequency vector of each frame in the candidate queue is counted: v t =[n c1 ,n c2 ...,n cC ](15), Where C is the number of semantic categories in the pre-trained semantic segmentation network, n ck is the candidate semantic key frame S at time t t The number of pixels with category ck is then normalized to obtain the frequency vector Finally, compare the current semantic candidate frame S t and the previous semantic candidate frame S t-1 Cosine similarity of: Compare the cosine similarity with the set threshold θ. t >θ when the RGB, depth and semantic candidate frames at time t in the candidate frame queue are synchronously deleted, otherwise all candidate frames at time t are retained as key frames; (3) For the map construction task, the core is to simultaneously construct a 3D RGB map, a depth map, and a semantic map by using a 3D Gaussian distribution model with semantics. For the Gaussian distribution introduced into the 3D model, a joint optimizer is constructed based on the original data set at time t and the rendered data set obtained by the current 3D Gaussian distribution model after rasterization. The map reconstruction loss function is: Among them L C and L S They are the SSIM weighted losses for RGB images and semantic images respectively: Then, by backpropagating the gradient, minimizing the map reconstruction loss, iteratively optimizing all variable parameters of the Gaussian distribution, and finally obtaining the converged 3D RGB map, depth map, and semantic map simultaneously.

Images