Visual slam method and system based on 3D gaussian spatter in dynamic environment and readable storage medium

Abstract

The application relates to the technical field of computer vision, in particular to a visual SLAM method based on 3D Gaussian spatter in a dynamic environment, a system and a readable storage medium. By performing instance segmentation on a scene image and extracting a prior dynamic mask, a potential dynamic mask is identified and fused to obtain a comprehensive dynamic mask; the comprehensive dynamic mask is optimized by using residual entropy and depth consistency, and then dynamic features are accurately removed; then, in order to balance the stability of pose tracking and the efficiency of mapping, a two-stage key frame screening strategy is constructed; on this basis, an improved 3D Gaussian spatter technology is further introduced to render and construct a scene graph; for each Gaussian ellipsoid, the volume and opacity weight are estimated, the multi-view contribution weight is calculated, the maximum weight is obtained by using a weighted average method based on the multi-view contribution weight, and pruning is performed. The application aims to solve the problem of how to reduce the computing power consumption in the 3D GS-SLAM scheme while ensuring the rendering quality.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a visual SLAM method, system and readable storage medium based on 3D Gaussian splashing in dynamic environments. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology in robotics, autonomous driving, AR / VR, and other fields, enabling real-time estimation of camera pose and construction of a global map in unknown environments. Traditional SLAM methods are mostly based on static scene assumptions, focusing on robustness and accuracy optimization in static environments. However, dynamic targets that are common in real-world scenarios often lead to pose drift or even tracking failure.

[0003] Currently, some research has combined 3D Gaussian Splatting (3DGS) technology with SLAM. By fitting a spatially anisotropic Gaussian kernel function, it can model the continuous probability density distribution of a scene. It has the advantages of being differentiable, fast rendering, and low storage overhead, and supports multi-resolution mapping and progressive refinement. It has attracted much attention in the fields of vision and graphics.

[0004] However, traditional 3DGS-SLAM combined schemes suffer from a surge in computational overhead due to the rapid expansion of the number of Gaussian elements during operation, making them unsuitable for large-scale or dynamic scenes. Therefore, this application proposes a visual SLAM method based on 3D Gaussian sputtering in dynamic environments, aiming to reduce the computational overhead of 3DGS-SLAM schemes while ensuring rendering quality. Summary of the Invention

[0005] The main purpose of this application is to provide a visual SLAM method based on 3D Gaussian splashing in dynamic environments, aiming to solve the problem of how to reduce the computational cost in the 3DGS-SLAM scheme while ensuring rendering quality.

[0006] To achieve the above objectives, this application provides a visual SLAM method based on 3D Gaussian spraying in a dynamic environment, the method comprising the following steps:

[0007] S10, the RGB images acquired by the depth camera are segmented into semantic mask sets, dynamic optical flow points between adjacent RGB image frames are filtered, the dynamic optical flow points are projected and matched with the semantic mask sets to obtain a potential dynamic mask set, and a comprehensive dynamic mask is constructed based on the potential dynamic mask set and the prior dynamic object masks in the semantic mask set.

[0008] S20, calculate the residual entropy and depth value of each potential dynamic mask in the integrated dynamic mask, determine the target integrated dynamic mask whose residual entropy and depth value are in the preset range as dynamic feature points in the mask region and remove them to obtain a set of static feature points;

[0009] S30, invoke the first keyframe filtering strategy to filter the first keyframe set in the depth image acquired by the depth camera, and perform pose estimation on the static feature point set based on the first keyframe set;

[0010] S40, invoke the second keyframe filtering strategy to filter the second keyframe set in the first keyframe set, and perform initial Gaussian rendering and map construction based on the second keyframe set;

[0011] S50, Calculate the multi-view contribution weight of the Gaussian ellipsoids based on the volume and opacity of each Gaussian ellipsoid in the map after mapping;

[0012] S60, calculate Gaussian weights based on the contribution weights of each of the multi-views, filter the Gaussian ellipsoids based on the Gaussian weights, and perform secondary Gaussian rendering and map construction based on the selected target Gaussian ellipsoids.

[0013] Optionally, S10 includes:

[0014] S11, perform instance segmentation on the RGB image using YOLOv11 to obtain a semantic mask set M, where the mask representation of prior dynamic objects is as follows: , ;

[0015] S12, perform epipolar geometry error filtering on optical flow point pairs between adjacent RGB image frames to obtain dynamic optical flow points, and perform projection matching between the dynamic optical flow points and each mask in the semantic mask set M to obtain the corresponding latent mask. :

[0016]

[0017] In the formula, and To match the homogeneous coordinates of optical flow point pairs, Based on the matrix, The threshold for epipolar geometric error;

[0018] S13, according to the potential mask and the mask of the prior dynamic object This constitutes a comprehensive dynamic mask. :

[0019]

[0020] In the formula, s represents the number of instance objects.

[0021] Optionally, S20 includes:

[0022] S21, will Mapping to the corresponding depth map yields the average depth value of the i-th instance object. :

[0023]

[0024] In the formula, Let be the total number of pixels with effective depth within the mask of the i-th instance object. This represents the depth value of the j-th pixel of the i-th instance object. ;

[0025] S22, for the i-th mask Assuming each of its edge points Within its radius neighborhood If there are m pixels, then for Each of them The residuals are defined as follows:

[0026]

[0027] In the formula, Represents edge points In the neighborhood Each pixel within, Indicated depth, ;

[0028] S23, Normalized residuals:

[0029]

[0030] Step 2.4: Calculate edge points The residual entropy of the neighborhood:

[0031]

[0032] Set residual entropy threshold and depth consistency threshold , The preset interval is represented as:

[0033]

[0034] S25: When of The pixels within satisfy the following conditions:

[0035]

[0036] In the formula, m represents the edge point. In the neighborhood The total number of pixels within;

[0037] Then the boundary points Include the corresponding mask;

[0038] If not satisfied, then the boundary points will be... Excluded from the corresponding mask; the optimized mask is obtained by sorting. ;

[0039] S26, For dynamic feature points located at the edge of the mask, assume For dynamic feature points A neighborhood with a radius of 3 pixels A certain pixel within, ;

[0040] If the following conditions are met, then determine the dynamic feature point. Belongs to a specific label Corresponding detection objects:

[0041]

[0042] In the formula, Represents pixels Target detection labels, For indicator functions;

[0043] S27, if multiple different labels exist in the neighborhood. And the number of pixels corresponding to each label are respectively Then the dynamic feature points The target detection label is determined to be the one with the highest percentage of pixels. Corresponding object:

[0044]

[0045] S28, will be located The dynamic feature points at the edges of the corresponding mask regions are removed to obtain a set of static feature points. .

[0046] Optionally, in step S30, the first keyframe filtering strategy includes:

[0047] S31, let... and These represent the current time t depth image frames respectively. The set of interior points detected in The number of interior points in the matrix defines the overlap. To measure and the previous keyframe The degree of overlap of the interior points:

[0048]

[0049] Set the threshold for the number of internal points and overlap threshold The conditions for the judgment:

[0050]

[0051] If the aforementioned judgment conditions are met, then Set as the current keyframe Otherwise, select the next frame. Repeat the judgment in step S31 until the judgment condition is met;

[0052] S32, let and The semantic category sets are respectively represented as and , and The set of shared semantic categories between them is represented as , Then the j-th semantic category probability distribution of pixel proportion The formula for calculation is:

[0053]

[0054] In the formula, Represents the j-th semantic category pixels, express The number of pixels;

[0055] Each probability distribution Forming probability distribution vector Calculate the relative entropy of the probability distribution vector:

[0056]

[0057]

[0058] if and If different categories exist, the probability of the unique category that is not in the shared category is calculated as follows, and the final semantic similarity is obtained:

[0059]

[0060]

[0061] Set semantic similarity threshold If the conditions are met Then Selected as Otherwise, proceed to the next step.

[0062] S33, Calculation and relative rotation matrix :

[0063]

[0064] In the formula, and They represent rotation matrix and The inverse of the rotation matrix is ​​obtained by calculating... to measure the norm and Rotational change:

[0065]

[0066] In the formula, F represents the Frobenius norm. express Matrix number Line number The value of the column;

[0067] Normalized rotational change:

[0068]

[0069] set up and The translational change is Then the normalized translation change measure value as follows:

[0070]

[0071] In the formula, , , These represent the values ​​of the corresponding translation changes. This represents the magnitude of the maximum translational change currently recorded.

[0072] Set rotation threshold Translation threshold And determine whether the following conditions are met:

[0073]

[0074] S34, let and The frame numbers in the input image frame sequence are respectively represented as follows: and And the timestamps are respectively and Set frame interval threshold and time interval threshold If the following conditions are met:

[0075]

[0076] Then Set as S35, Repeat steps S31-S34 to obtain the first set of keyframes. .

[0077] Optionally, in S40, the second keyframe filtering strategy includes:

[0078] S41, Calculate the first keyframe set Spectrum of all keyframes and calculate total energy. :

[0079]

[0080] In the formula, and These are the width and height of the keyframe, respectively. This indicates that the frequency domain location of the keyframe after Fast Fourier Transform is... The spectral values ​​of complex numbers;

[0081] S42, Calculate high-frequency energy :

[0082]

[0083] In the formula, This represents the result of the spectrum after centering.

[0084] S43, based on the total energy and the high-frequency energy Calculate the sharpness information entropy respectively Map point information entropy and semantic tag number information entropy Then, by weighted summation, the total information entropy is obtained. :

[0085]

[0086]

[0087]

[0088]

[0089] In the formula, N represents the number of map points, and S represents the number of semantic tags. Indicates the first The weight of each information entropy;

[0090] S44, determine the keyframe furthest from the current time in the internal window. The total information entropy in the first keyframe of the outer window, excluding the inner window, is calculated. The smallest or longest-lasting keyframe and keyframe within the window The replacement is performed to obtain a set of m first keyframes. , as the second set of keyframes.

[0091] Optionally, S50 includes:

[0092] S51, the state vector of the i-th Gaussian ellipsoid at time step (iteration number) t. Set as a two-dimensional vector:

[0093]

[0094] In the formula, This represents the number of Gaussian ellipsoids. and These represent the weights and weight changes of the Gaussian ellipsoid, respectively.

[0095] S52, the state transition model is constructed as follows:

[0096]

[0097]

[0098] In the formula, Here is the state transition matrix. Let be the covariance matrix of the process noise. Indicates the first The Gaussian ellipsoids at time t are estimated based on the prior state at time t-1. Indicates the first Posterior state estimate of a Gaussian ellipsoid at time t-1 Indicates process noise;

[0099] The observation model is constructed as follows:

[0100]

[0101]

[0102] In the formula, Indicates the first The observation values ​​of a Gaussian ellipsoid at time t. Indicates the first Posterior state estimate of a Gaussian ellipsoid at time t For the observation matrix, To observe the noise, The covariance matrix of the observation noise;

[0103] The actual observed vector This can be expressed as a weighted sum as follows:

[0104]

[0105] in, Let be the weight of the i-th component. This represents the volume of the i-th Gaussian ellipsoid. Let be the opacity of the i-th Gaussian ellipsoid;

[0106] S53, for the i-th Gaussian ellipsoid, the iteration at time t is performed as follows:

[0107] (1)

[0108] (2)

[0109] (3)

[0110] (4)

[0111] (5)

[0112] In the formula, Let (1) denote the innovation vector, and let (2) denote the first... A Gaussian ellipsoid in The prior state estimate at time t is given by Posterior state at time step Through the state transition matrix The prediction is a preliminary inference based on historical information; (2) indicates the first ellipsoid The prior covariance matrix at time t reflects the uncertainty of the prior state, derived from... Post-hoc covariance Transition matrix and its transpose After transformation, the noise covariance during the superposition process (Reflecting the error of the state transition model) we get; (3) in which, It is the Kalman gain, used to balance the reliability of prior predictions and observations, including the uncertainty of the projection of prior predictions. and observation noise covariance The greater the gain, the stronger the influence of the observation on the state update; (4) indicates the first ellipsoid The posterior state estimate at time t is the optimal state outcome, derived from the prior estimate. Add Kalman gain and innovation vector The product of these two terms is obtained by combining the information from prediction and observation; (5) represents the product of the two terms. ellipsoid The posterior covariance matrix at time 1 reflects the uncertainty of the state after fusion of observations, and is expressed through the identity matrix. Subtract Kalman gain and observation matrix The product of these factors, and then the prior covariance. Multiplying them together corrects for the prior uncertainty.

[0113] By repeating step S53 iteratively based on Kalman filtering, the state vector is obtained. Corresponding weights ;

[0114] S54 employs a time decay and weighted averaging strategy to calculate the cumulative contribution of each Gaussian ellipsoid with view input, where the multi-view contribution weight of the i-th Gaussian ellipsoid is... The expression is:

[0115]

[0116] In the formula, The ID of the frame number processed by the system at the current moment. Indicates the time decay weight. Let Gaussian ellipsoid visibility index function be used. Contribute values ​​to a single view, where, Represented as:

[0117]

[0118] In the formula, This represents the set of pixels in the projection region of Gaussian point i in the k-th frame. For opacity, This represents the cumulative transmittance of the p-th pixel in the k-th frame.

[0119] Optionally, S60 includes:

[0120] S61, Set a weighting coefficient The multi-view contribution weight value of the i-th Gaussian ellipsoid is represented in the following convex combination form:

[0121]

[0122] make for The discrete set of states, and assume that it contains A finite number of state values:

[0123]

[0124] In the formula, s represents a total of s states;

[0125] Assumption The probability of state i occurring at time t is expressed as: , The state probability vector is , The state transition probability in one step is expressed as:

[0126]

[0127] express Given that the state is i at time t-1, what is the probability that the state transitions to state j at time t? ;

[0128] S62, according to get The transition probability matrix :

[0129]

[0130] in, satisfy: and

[0131] S63, Establish the state probability prediction equation:

[0132]

[0133] In the formula, Indicates the weighting coefficient at the next time step. The state probability; For the current moment The state probability is a known quantity;

[0134] S64, Construct an optimization problem based on the innovation vector at time k+1:

[0135]

[0136]

[0137] Solve the optimization problem to obtain the minimized residual. ;

[0138] S65, calculate the minimum residual respectively. With discrete state set Distance between each state value :

[0139]

[0140] Sure The residual corresponding to the minimum value Discrete state :

[0141]

[0142] S65, at time t+1 Belongs to a certain state in s ,but Given that the state is i at time t, what is the probability that the state will transition to state j at time t+1? The adjustment rules are as follows:

[0143]

[0144] in, , Indicates the weighting coefficient;

[0145] S66, Set weight threshold and gradient threshold After a preset number of iterations, those that meet the requirements are eliminated. and The target Gaussian ellipsoid is obtained by using the Gaussian ellipsoid.

[0146] Optionally, the method is characterized in that, after step S60, the method further includes:

[0147] S70, the RGB image obtained after rendering in step S60. and depth images , and the RGB image captured by the depth camera and the depth image Compare and calculate RGB loss separately. SSIM loss Deep loss and regularization loss :

[0148]

[0149]

[0150]

[0151]

[0152] In the formula, Represents the mean of the image. Represents the variance of the image. This represents the covariance of two images. and To avoid constants with a denominator of 0, and They represent the first The scale of a Gaussian ellipsoid and its mean scale Indicates the number of valid pixels. This represents the optimized mask. Indicates Gaussian visibility. Indicates the number of valid depth values;

[0153] S80, to minimize RGB loss Minimize SSIM loss Minimize depth loss And a regularization loss The weighted sum is determined as the target loss function. :

[0154]

[0155] In the formula, This represents the weight of the i-th loss component. and Let represent the scale of the i-th Gaussian ellipsoid and its mean scale, respectively;

[0156] S90, with the target loss function The parameters are adjusted with the goal of minimization.

[0157] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the visual SLAM method based on 3D Gaussian spray in a dynamic environment as described in any of the preceding claims.

[0158] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the visual SLAM method based on 3D Gaussian spray in a dynamic environment as described in any of the preceding claims.

[0159] This application has at least the following beneficial effects:

[0160] 1. A visual SLAM system based on RGB image-depth image measurement and compact 3D Gaussian environment representation for dynamic environments was constructed, which significantly improved the system's accuracy, mapping and rendering effects and computational cost;

[0161] 2. A dynamic feature detection strategy for "prior + latent" is proposed, which optimizes the comprehensive dynamic mask by residual entropy and depth consistency, and then uses the obtained high-confidence static features to achieve accurate pose estimation.

[0162] 3. A two-stage keyframe selection strategy, which is "localization-oriented" and "mapping-oriented", was constructed to ensure both the stability of pose tracking and the efficiency of mapping.

[0163] 4. A 3D Gaussian ellipsoid weight estimation and pruning mechanism that integrates Kalman filtering and Markov models was designed, which effectively reduces the number of Gaussian ellipsoids and storage burden while ensuring rendering quality. Attached Figure Description

[0164] Figure 1 This is a flowchart illustrating the visual SLAM method based on 3D Gaussian spraying in a dynamic environment, as described in the embodiments of this application.

[0165] Figure 2 This is a schematic diagram illustrating keyframe optimization and selection based on nested sliding windows in an embodiment of this application;

[0166] Figure 3 A comparison diagram of ATE of ORB-SLAM3 and the method of this application on different sequences in the TUM dataset, according to embodiments of this application;

[0167] Figure 4 A comparison diagram of ATE of ORB-SLAM3 and the method of this application on different sequences of the BONN dataset, according to embodiments of this application;

[0168] Figure 5 This is a comparison chart of the rendering effects of different algorithms involved in the embodiments of this application and the method of this application on the TUM and BONN datasets;

[0169] Figure 6This is a comparison diagram of the trajectories of Photo-SLAM involved in the embodiments of this application and the method of this application in a real scene;

[0170] Figure 7 This is a comparison diagram of the rendering effects of Photo-SLAM and the method of this application in a real scene, based on the embodiments of this application.

[0171] Figure 8 This is another flowchart illustrating the visual SLAM method based on 3D Gaussian spraying in a dynamic environment involved in the embodiments of this application;

[0172] Figure 9 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0173] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0174] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0175] First Embodiment

[0176] Reference Figure 1 This embodiment provides a visual SLAM method based on 3D Gaussian spraying in a dynamic environment. The method includes the following steps:

[0177] S10, the RGB images acquired by the depth camera are segmented into semantic mask sets, dynamic optical flow points between adjacent RGB image frames are filtered, the dynamic optical flow points are projected and matched with the semantic mask sets to obtain a potential dynamic mask set, and a comprehensive dynamic mask is constructed based on the potential dynamic mask set and the prior dynamic object masks in the semantic mask set.

[0178] In this embodiment, RGBD images of the surrounding environment are acquired using a depth camera, and YOLOv11 is used to perform instance segmentation on the scene images to obtain a set of all semantic masks. Finally, optical flow epipolar geometry is combined to obtain a comprehensive dynamic mask.

[0179] Using YOLOv11 to perform instance segmentation on the scene image, we obtain a set M of all semantic masks, where the mask representation of prior dynamic objects (such as pedestrians, vehicles, and animals) is as follows: , .

[0180] For potential dynamic objects (such as backpacks, chairs, and rackets), dynamic optical flow points are obtained by performing epipolar geometry error filtering on optical flow point pairs between adjacent frames. These points are then projected and matched with the masks in M ​​to obtain the corresponding potential dynamic masks. .

[0181] in, and To match the homogeneous coordinates of optical flow point pairs, Based on the matrix, This is the threshold for epipolar geometric error.

[0182] Synthetic dynamic mask Represented as

[0183] In the formula, s represents the number of instance objects.

[0184] S20, calculate the residual entropy and depth value of each potential dynamic mask in the integrated dynamic mask, determine the target integrated dynamic mask whose residual entropy and depth value are in the preset range as dynamic feature points in the mask region and remove them to obtain a set of static feature points;

[0185] In this embodiment, residual entropy and depth consistency (quantized by depth value) are used to optimize the comprehensive dynamic mask, and ORB features in the mask are removed to obtain static feature points.

[0186] Specifically, it includes:

[0187] S21, will Mapping to the corresponding depth map yields the average depth value of the i-th instance object. :

[0188]

[0189] In the formula, Let be the total number of pixels with effective depth within the mask of the i-th instance object. This represents the depth value of the j-th pixel of the i-th instance object. ;

[0190] S22, for the i-th mask Assuming each of its edge points Within its radius neighborhood If there are m pixels, then for Each of them The residuals are defined as follows:

[0191]

[0192] In the formula, Represents edge points In the neighborhood Each pixel within, Indicated depth, ;

[0193] S23, Normalized residuals:

[0194]

[0195] Step 2.4: Calculate edge points The residual entropy of the neighborhood:

[0196]

[0197] Set residual entropy threshold and depth consistency threshold , The preset interval is represented as:

[0198]

[0199] S25: When of The pixels within satisfy the following conditions:

[0200]

[0201] In the formula, m represents the edge point. In the neighborhood The total number of pixels within;

[0202] Then the boundary points Include the corresponding mask;

[0203] If not satisfied, then the boundary points will be... Excluded from the corresponding mask; the optimized mask is obtained by sorting. ;

[0204] S26, For dynamic feature points located at the edge of the mask, assume For dynamic feature points A neighborhood with a radius of 3 pixels A certain pixel within, ;

[0205] If the following conditions are met, then determine the dynamic feature point. Belongs to a specific label Corresponding detection objects:

[0206]

[0207] In the formula, Represents pixels Target detection labels, For indicator functions;

[0208] S27, if multiple different labels exist in the neighborhood. And the number of pixels corresponding to each label are respectively Then the dynamic feature points The target detection label is determined to be the one with the highest percentage of pixels. Corresponding object:

[0209]

[0210] S28, will be located The dynamic feature points at the edges of the corresponding mask regions are removed to obtain a set of static feature points. .

[0211] S30, invoke the first keyframe filtering strategy to filter the first keyframe set in the depth image acquired by the depth camera, and perform pose estimation on the static feature point set based on the first keyframe set;

[0212] In this embodiment, we construct a first-stage localization-oriented keyframe filtering method and use the filtered keyframe set and corresponding static feature points to perform pose estimation.

[0213] Specifically, it includes:

[0214] S31, let... and These represent the current time t depth image frames respectively. The set of interior points detected in The number of interior points in the matrix defines the overlap. To measure and the previous keyframe The degree of overlap of the interior points:

[0215]

[0216] Set the threshold for the number of internal points and overlap threshold The conditions for the judgment:

[0217]

[0218] If the aforementioned judgment conditions are met, then it is considered that... and With low overlap, Set as the current keyframe Otherwise, select the next frame. Repeat the judgment in step S31 until the judgment condition is met;

[0219] S32, let and The semantic category sets are respectively represented as and , and The set of shared semantic categories between them is represented as , Then the j-th semantic category probability distribution of pixel proportion The formula for calculation is:

[0220]

[0221] In the formula, Represents the j-th semantic category pixels, express The number of pixels;

[0222] Each probability distribution Forming probability distribution vector Calculate the relative entropy of the probability distribution vector:

[0223]

[0224]

[0225] if and The existence of different categories indicates a significant semantic difference between them. In this case, the probability of the unique category, which is not in the shared category, needs to be calculated as follows to obtain the final semantic similarity:

[0226]

[0227]

[0228] Set semantic similarity threshold If the conditions are met Then it is believed and The similarity is low, Selected as Otherwise, proceed to the next step.

[0229] S33, Calculation and relative rotation matrix :

[0230]

[0231] In the formula, and They represent rotation matrix and The inverse of the rotation matrix is ​​obtained by calculating... to measure the norm and Rotational change:

[0232]

[0233] In the formula, F represents the Frobenius norm. express Matrix number Line number The value of the column;

[0234] Normalized rotational change:

[0235]

[0236] set up and The translational change is Then the normalized translation change measure value as follows:

[0237]

[0238] In the formula, , , These represent the values ​​of the corresponding translation changes. This represents the magnitude of the maximum translational change currently recorded.

[0239] Set rotation threshold Translation threshold And determine whether the following conditions are met:

[0240]

[0241] S34, let and The frame numbers in the input image frame sequence are respectively represented as follows: and And the timestamps are respectively and Set frame interval threshold and time interval threshold If the following conditions are met:

[0242]

[0243] This means and When the frame interval or time interval between frames reaches a certain level, Set as S35, Repeat steps S31-S34 to obtain the first set of keyframes. .

[0244] S40, invoke the second keyframe filtering strategy to filter the second keyframe set in the first keyframe set, and perform initial Gaussian rendering and map construction based on the second keyframe set;

[0245] In this embodiment, a second-stage keyframe selection method for mapping is constructed, and the selected keyframe set is used for 3D Gaussian rendering and mapping.

[0246] Specifically, it includes:

[0247] S41, Calculate the first keyframe set Spectrum of all keyframes and calculate total energy. :

[0248]

[0249] In the formula, and These are the width and height of the keyframe, respectively. This indicates that the frequency domain location of the keyframe after Fast Fourier Transform is... The spectral values ​​of complex numbers;

[0250] S42, Calculate high-frequency energy :

[0251]

[0252] In the formula, This represents the result of the spectrum after centering.

[0253] S43, based on the total energy and the high-frequency energy Calculate the sharpness information entropy respectively Map point information entropy and semantic tag number information entropy Then, by weighted summation, the total information entropy is obtained. :

[0254]

[0255]

[0256]

[0257]

[0258] In the formula, N represents the number of map points, and S represents the number of semantic tags. Indicates the first The weight of each information entropy;

[0259] S44, determine the keyframe furthest from the current time in the internal window. The total information entropy in the first keyframe of the outer window, excluding the inner window, is calculated. The smallest or longest-lasting keyframe and keyframe within the window The replacement is performed to obtain a set of m first keyframes. , as the second set of keyframes.

[0260] It should be noted that in step S44, the keyframe dynamic optimization and selection method based on nested sliding windows, such as... Figure 2 As shown. The top row displays all keyframes from the previous time step arranged sequentially to fill an outer window of length m. The bottom row displays the current keyframe selected at the end of an inner window of length m. Following the FIFO (First-In, First-Out) principle, the oldest keyframe in the inner window is retrieved, and the keyframe with the lowest information entropy or the longest duration in the outer window (excluding the inner window) is replaced. Finally, the dynamically optimized set of m keyframes in the outer window is used for rendering and mapping.

[0261] S50, Calculate the multi-view contribution weight of the Gaussian ellipsoids based on the volume and opacity of each Gaussian ellipsoid in the map after mapping;

[0262] In this embodiment, on the one hand, Kalman filtering is used to comprehensively estimate the volume and opacity weights of each 3D Gaussian ellipsoid; on the other hand, a time decay and weighted averaging strategy is adopted to calculate the cumulative contribution of each 3D Gaussian ellipsoid as the view input is applied.

[0263] Specifically, it includes:

[0264] S51, the state vector of the i-th Gaussian ellipsoid at time step (iteration number) t. Set as a two-dimensional vector:

[0265]

[0266] In the formula, This represents the number of Gaussian ellipsoids. and These represent the weights and weight changes of the Gaussian ellipsoid, respectively.

[0267] S52, the state transition model is constructed as follows:

[0268]

[0269]

[0270] In the formula, Here is the state transition matrix. Let be the covariance matrix of the process noise. Indicates the first The Gaussian ellipsoids at time t are estimated based on the prior state at time t-1. Indicates the first Posterior state estimate of a Gaussian ellipsoid at time t-1 Indicates process noise;

[0271] The observation model is constructed as follows:

[0272]

[0273]

[0274] In the formula, Indicates the first The observation values ​​of a Gaussian ellipsoid at time t. Indicates the first Posterior state estimate of a Gaussian ellipsoid at time t For the observation matrix, To observe the noise, The covariance matrix of the observation noise;

[0275] The actual observed vector This can be expressed as a weighted sum as follows:

[0276]

[0277] in, Let be the weight of the i-th component. This represents the volume of the i-th Gaussian ellipsoid. Let be the opacity of the i-th Gaussian ellipsoid;

[0278] S53, for the i-th Gaussian ellipsoid, the iteration at time t is performed as follows:

[0279] (1)

[0280] (2)

[0281] (3)

[0282] (4)

[0283] (5)

[0284] In the formula, Let (1) denote the innovation vector, and let (2) denote the first... A Gaussian ellipsoid in The prior state estimate at time t is given by Posterior state at time step Through the state transition matrix The prediction is a preliminary inference based on historical information; (2) indicates the first ellipsoid The prior covariance matrix at time t reflects the uncertainty of the prior state, derived from... Post-hoc covariance Transition matrix and its transpose After transformation, the noise covariance during the superposition process (Reflecting the error of the state transition model) we get; (3) in which, It is the Kalman gain, used to balance the reliability of prior predictions and observations, including the uncertainty of the projection of prior predictions. and observation noise covariance The greater the gain, the stronger the influence of the observation on the state update; (4) indicates the first ellipsoid The posterior state estimate at time t is the optimal state outcome, derived from the prior estimate. Add Kalman gain and innovation vector The product of these two terms is obtained by combining the information from prediction and observation; (5) represents the product of the two terms. ellipsoid The posterior covariance matrix at time 1 reflects the uncertainty of the state after fusion of observations, and is expressed through the identity matrix. Subtract Kalman gain and observation matrix The product of these factors, and then the prior covariance. Multiplying them together corrects for prior uncertainties;

[0285] By repeating step S53 iteratively based on Kalman filtering, the state vector is obtained. Corresponding weights ;

[0286] S54 employs a time decay and weighted averaging strategy to calculate the cumulative contribution of each Gaussian ellipsoid with view input, where the multi-view contribution weight of the i-th Gaussian ellipsoid is... The expression is:

[0287]

[0288] In the formula, The ID of the frame number processed by the system at the current moment. Indicates the time decay weight. Let Gaussian ellipsoid visibility index function be used. Contribute values ​​to a single view, where, Represented as:

[0289]

[0290] In the formula, This represents the set of pixels in the projection region of Gaussian point i in the k-th frame. For opacity, This represents the cumulative transmittance of the p-th pixel in the k-th frame.

[0291] S60, calculate Gaussian weights based on the contribution weights of each of the multi-views, filter the Gaussian ellipsoids based on the Gaussian weights, and perform secondary Gaussian rendering and map construction based on the selected target Gaussian ellipsoids.

[0292] In this embodiment, in the final step, the weighted average method of the adaptive Markov model (SAMM) is used to obtain the final Gaussian weights and perform pruning. Then, the filtered set of keyframes for graph building is used to perform rendering and graph building.

[0293] Specifically, it includes:

[0294] S61, Set a weighting coefficient The multi-view contribution weight value of the i-th Gaussian ellipsoid is represented in the following convex combination form:

[0295]

[0296] make for The discrete set of states, and assume that it contains A finite number of state values:

[0297]

[0298] In the formula, s represents a total of s states;

[0299] Assumption The probability of state i occurring at time t is expressed as: , The state probability vector is , The state transition probability in one step is expressed as:

[0300]

[0301] express Given that the state is i at time t-1, what is the probability that the state transitions to state j at time t? ;

[0302] S62, according to get The transition probability matrix :

[0303]

[0304] in, satisfy: and

[0305] S63, Establish the state probability prediction equation:

[0306]

[0307] In the formula, Indicates the weighting coefficient at the next time step. The state probability; For the current moment The state probability is a known quantity;

[0308] S64, Construct an optimization problem based on the innovation vector at time k+1:

[0309]

[0310]

[0311] Solve the optimization problem to obtain the minimized residual. ;

[0312] S65, calculate the minimum residual respectively. With discrete state set Distance between each state value :

[0313]

[0314] Sure The residual corresponding to the minimum value Discrete state :

[0315]

[0316] S65, at time t+1 Belongs to a certain state in s ,but Given that the state is i at time t, what is the probability that the state will transition to state j at time t+1? The adjustment rules are as follows:

[0317]

[0318] in, , Indicates the weighting coefficient;

[0319] S66, Set weight threshold and gradient threshold After a preset number of iterations, those that meet the requirements are eliminated. and The target Gaussian ellipsoid is obtained by using the Gaussian ellipsoid.

[0320] In the technical solution provided in this embodiment, the scene image is segmented into instances and a priori dynamic mask is extracted. Potential dynamic masks are identified and fused to obtain a comprehensive dynamic mask. Residual entropy and depth consistency are used to optimize it, thereby accurately removing dynamic features. Then, in order to balance the stability of pose tracking and the efficiency of mapping, a two-stage keyframe filtering strategy is constructed. On this basis, an improved 3D Gaussian splashing technique is further introduced to render and construct the scene graph. For each Gaussian ellipsoid, its volume and opacity weights are estimated, and its multi-view contribution weights are calculated. The final weights are obtained by weighted averaging based on the multi-view contribution weights, and pruning is performed.

[0321] Second Embodiment

[0322] Based on the first embodiment, in this embodiment, in order to improve the rendering effect, depth rendering is further added. That is, to render the RGB images separately. and depth map RGB image and depth map of the corresponding scene image A comparison is then performed. Following step S60, the method further includes:

[0323] S70, the RGB image obtained after rendering in step S60. and depth images , and the RGB image captured by the depth camera and the depth image Compare and calculate RGB loss separately. SSIM loss Deep loss and regularization loss :

[0324]

[0325]

[0326]

[0327]

[0328] In the formula, Represents the mean of the image. Represents the variance of the image. This represents the covariance of two images. and To avoid constants with a denominator of 0, and They represent the first The scale of a Gaussian ellipsoid and its mean scale Indicates the number of valid pixels. This represents the optimized mask. Indicates Gaussian visibility. Indicates the number of valid depth values;

[0329] S80, to minimize RGB loss Minimize SSIM loss Minimize depth loss And a regularization loss The weighted sum is determined as the target loss function. :

[0330]

[0331] In the formula, This represents the weight of the i-th loss component. and Let represent the scale of the i-th Gaussian ellipsoid and its mean scale, respectively;

[0332] S90, with the target loss function The parameters are adjusted with the goal of minimization.

[0333] Third Embodiment

[0334] As a verification embodiment, this embodiment performs simulation verification on the effect or performance of the methods or models involved in the above embodiments. Specifically, as follows:

[0335] A series of simulation experiments were conducted on the TUM and BONN datasets, and then the performance of the proposed method was further verified in a real-world environment using a mobile robot. The TUM and BONN datasets contain RGB-D image sequences and camera movement trajectories for various dynamic scenes and are widely used for evaluating visual SLAM algorithms. All experiments were performed on a computer equipped with an NVIDIA RTX 4060 graphics card, an Intel i5-12490F CPU, 16GB of RAM, and running Ubuntu 22.04 LTS. The experiments selected the following sequences from the TUM dataset: high-dynamic walking sequences (fr3 / walking_rpy, fr3 / walking_xyz, fr3 / walking_static); low-dynamic sitting sequence (fr3 / sitting_xyz); low-dynamic sequence (fr2 / desk_with_person); large-scene static sequence (fr3 / long_office_household); and high-dynamic walking sequences (person_tracking, person_tracking2) from the BONN dataset; and sequences containing passively moving objects (ballon, ballon2, ballon_tracking, moving_nonobstructing_box2). These sequences cover different camera motion modes and scene structures, and are compared and analyzed with ORB-SLAM3. Absolute trajectory error (ATE) was used as a performance metric during the evaluation. ATE calculates the error between the system's true pose and estimated pose, reflecting the global localization accuracy, and is used to evaluate the overall performance of the SLAM system.

[0336] Table 1. Comparison of RMSE of Absolute Trajectory Error (ATE) in the TUM dataset

[0337]

[0338] Table 2. Comparison of RMSE of Absolute Trajectory Error (ATE) in the BONN dataset

[0339]

[0340] Tables 1 and 2 show the RMSE comparison between our method and ORB-SLAM3 in terms of absolute trajectory error. The results in the tables demonstrate that by removing features of dynamic objects, our method significantly reduces trajectory drift and improves positioning accuracy in dynamic scenes.

[0341] Additionally, refer to Figure 3 , 4The comparison graphs of ORB-SLAM3 and the method of this application on different sequences of the TUM and BONN datasets show that our algorithm significantly reduces trajectory error compared to ORB-SLAM3.

[0342] Reference Figure 5 The comparison diagrams showing the rendering effects of different algorithms and the method of this application on the TUM and BONN datasets demonstrate that the method of this application achieves better rendering and reconstruction results than the other algorithms mentioned above.

[0343] Reference Figure 6 The comparison diagram of the trajectories of Photo-SLAM and the proposed method in a real scene shows that, in a real scene, the proposed method significantly reduces trajectory error compared to Photo-SLAM.

[0344] Reference Figure 7 The comparison diagrams showing the rendering effects of Photo-SLAM and the method of this application in a real scene demonstrate that the method achieves better rendering and reconstruction results than Photo-SLAM.

[0345] In addition, as an implementation scheme, refer to Figure 8 The diagram illustrates another flowchart of the visual SLAM method based on 3D Gaussian splatting in dynamic environments proposed in this application. It introduces YOLOv11 to segment the scene image and extract a priori dynamic mask, which is then fused with a potential dynamic mask identified through optical flow and epipolar geometry to obtain a comprehensive dynamic mask. This mask is then optimized using residual entropy and depth consistency to accurately remove dynamic features. To balance pose tracking stability and mapping efficiency, a two-stage keyframe filtering mechanism, oriented towards both localization and mapping, is constructed. Furthermore, a compact 3D Gaussian splatting (3DGS) technique is introduced to render and construct the scene graph. For each 3D Gaussian ellipsoid, Kalman filtering is used to estimate its volume and opacity weights, and its contribution weights across multiple views are calculated. A weighted average method based on an adaptive Markov model is then constructed to obtain the final weights and perform pruning.

[0346] As one implementation scheme, Figure 9 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0347] like Figure 9As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0348] Those skilled in the art will understand that Figure 9 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0349] like Figure 9 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0350] exist Figure 9 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0351] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0352] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:

[0353] S10, the RGB images acquired by the depth camera are segmented into semantic mask sets, dynamic optical flow points between adjacent RGB image frames are filtered, the dynamic optical flow points are projected and matched with the semantic mask sets to obtain a potential dynamic mask set, and a comprehensive dynamic mask is constructed based on the potential dynamic mask set and the prior dynamic object masks in the semantic mask set.

[0354] S20, calculate the residual entropy and depth value of each potential dynamic mask in the integrated dynamic mask, determine the target integrated dynamic mask whose residual entropy and depth value are in the preset range as dynamic feature points in the mask region and remove them to obtain a set of static feature points;

[0355] S30, invoke the first keyframe filtering strategy to filter the first keyframe set in the depth image acquired by the depth camera, and perform pose estimation on the static feature point set based on the first keyframe set;

[0356] S40, invoke the second keyframe filtering strategy to filter the second keyframe set in the first keyframe set, and perform initial Gaussian rendering and map construction based on the second keyframe set;

[0357] S50, Calculate the multi-view contribution weight of the Gaussian ellipsoids based on the volume and opacity of each Gaussian ellipsoid in the map after mapping;

[0358] S60, calculate Gaussian weights based on the contribution weights of each of the multi-views, filter the Gaussian ellipsoids based on the Gaussian weights, and perform secondary Gaussian rendering and map construction based on the selected target Gaussian ellipsoids.

[0359] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0360] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the visual SLAM method based on 3D Gaussian spraying in a dynamic environment as described in the above embodiments.

[0361] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0362] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0363] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0364] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0365] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0366] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0367] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0368] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0369] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A visual SLAM method based on 3D Gaussian splashing in a dynamic environment, characterized in that, The method includes the following steps: S10, the RGB images acquired by the depth camera are segmented into semantic mask sets, dynamic optical flow points between adjacent RGB image frames are filtered, the dynamic optical flow points are projected and matched with the semantic mask sets to obtain a potential dynamic mask set, and a comprehensive dynamic mask is constructed based on the potential dynamic mask set and the prior dynamic object masks in the semantic mask set. S20, calculate the residual entropy and depth value of each potential dynamic mask in the integrated dynamic mask, determine the target integrated dynamic mask whose residual entropy and depth value are in the preset range as dynamic feature points in the mask region and remove them to obtain a set of static feature points; S30, invoke the first keyframe filtering strategy to filter the first keyframe set in the depth image acquired by the depth camera, and perform pose estimation on the static feature point set based on the first keyframe set; S40, invoke the second keyframe filtering strategy to filter the second keyframe set in the first keyframe set, and perform initial Gaussian rendering and map construction based on the second keyframe set; S50, Calculate the multi-view contribution weight of the Gaussian ellipsoids based on the volume and opacity of each Gaussian ellipsoid in the map after mapping; S60, calculate Gaussian weights based on the contribution weights of each of the multi-views, filter the Gaussian ellipsoids based on the Gaussian weights, and perform secondary Gaussian rendering and map building based on the selected target Gaussian ellipsoids. In step S30, the first keyframe filtering strategy includes: S31, let... and These represent the current time t depth image frames respectively. The set of interior points detected in The number of interior points in the matrix defines the overlap. To measure and the previous keyframe The degree of overlap of the interior points: ； Set the threshold for the number of internal points and overlap threshold The conditions for the judgment: ； If the aforementioned judgment conditions are met, Set as the current keyframe Otherwise, select the next frame. Repeat the judgment in step S31 until the judgment condition is met; S32, let and The semantic category sets are respectively represented as and , and The set of shared semantic categories between them is represented as , Then the j-th semantic category probability distribution of pixel proportion The formula for calculation is: ； In the formula, Represents the j-th semantic category pixels, express The number of pixels; Each probability distribution Forming probability distribution vector Calculate the relative entropy of the probability distribution vector: ； ； if and If different categories exist, the probability of the unique category that is not in the shared category is calculated as follows, and the final semantic similarity is obtained: ； ； Set semantic similarity threshold If satisfied Then Selected as Otherwise, proceed to the next step. S33, Calculation and relative rotation matrix : ； In the formula, and They represent rotation matrix and The inverse of the rotation matrix is ​​obtained by calculating... to measure the norm and Rotational change: ； In the formula, F represents the Frobenius norm. express Matrix number Line number The value of the column; Normalized rotational change: ； set up and The translational change is Then the normalized translation change measure value as follows: ； In the formula, , , These represent the values ​​of the corresponding translation changes. This represents the magnitude of the maximum translational change currently recorded. Set rotation threshold Translation threshold And determine whether the following conditions are met: ； S34, let and The frame numbers in the input image frame sequence are respectively represented as follows: and And the timestamps are respectively and Set frame interval threshold and time interval threshold If the following conditions are met: ； Then Set as S35, Repeat steps S31-S34 to obtain the first set of keyframes. ; In S40, the second keyframe filtering strategy includes: S41, Calculate the first keyframe set Spectrum of all keyframes and calculate total energy. : ； In the formula, W and H These are the width and height of the keyframe, respectively. This indicates that the frequency domain location of the keyframe after Fast Fourier Transform is... The spectral values ​​of complex numbers; S42, Calculate high-frequency energy : ； In the formula, This represents the result of the spectrum after centering. S43, based on the total energy and the high-frequency energy Calculate the sharpness information entropy respectively Map point information entropy and semantic tag number information entropy Then, by weighted summation, the total information entropy is obtained. : ； ； ； ； In the formula, N represents the number of map points, and S represents the number of semantic tags. Indicates the first The weight of each information entropy; S44, determine the keyframe furthest from the current time in the internal window. The total information entropy in the first keyframe of the outer window, excluding the inner window, is calculated. The smallest or longest-lasting keyframe in the outer window excluding the inner window The replacement is performed to obtain a set of m first keyframes. , as the second set of keyframes.

2. The method as described in claim 1, characterized in that, S10 includes: S11, perform instance segmentation on the RGB image using YOLOv11 to obtain a semantic mask set M, where the mask representation of prior dynamic objects is as follows: , ; S12, perform epipolar geometry error filtering on optical flow point pairs between adjacent RGB image frames to obtain dynamic optical flow points, and perform projection matching between the dynamic optical flow points and each mask in the semantic mask set M to obtain the corresponding latent mask. : ； In the formula, and To match the homogeneous coordinates of optical flow point pairs, Based on the matrix, The threshold for epipolar geometric error; S13, according to the potential mask and the mask of the prior dynamic object This constitutes a comprehensive dynamic mask. : ； In the formula, s represents the number of instance objects.

3. The method as described in claim 2, characterized in that, S20 includes: S21, will Mapping to the corresponding depth map yields the average depth value of the i-th instance object. : ； In the formula, Let be the total number of pixels with effective depth within the mask of the i-th instance object. This represents the depth value of the j-th pixel of the i-th instance object. ; S22, for the i-th mask Assuming each of its edge points Within its radius neighborhood If there are m pixels, then for Each of them The residuals are defined as follows: ； In the formula, Represents edge points In the neighborhood Each pixel within, Indicated depth, ; S23, Normalized residuals: ； S24: Calculate edge points The residual entropy of the neighborhood: ； Set residual entropy threshold and depth consistency threshold , The preset interval is represented as: ； S25: When of The pixels within satisfy the following conditions: ； In the formula, m represents the edge point. In the neighborhood The total number of pixels within; Then the edge points Include the corresponding mask; If not satisfied, then the edge point Excluded from the corresponding mask; the optimized mask is obtained by sorting. ; S26, For dynamic feature points located at the edge of the mask, assume For dynamic feature points A neighborhood with a radius of 3 pixels A certain pixel within, ; If the following conditions are met, then determine the dynamic feature point. Belongs to a specific label Corresponding detection objects: ； In the formula, Represents pixels Target detection labels, For indicator functions; S27, if multiple different labels exist in the neighborhood. And the number of pixels corresponding to each label are respectively Then the dynamic feature points The target detection label is determined to be the one with the highest percentage of pixels. Corresponding object: ； S28, will be located The dynamic feature points at the edges of the corresponding mask regions are removed to obtain a set of static feature points. .

4. The method as described in claim 1, characterized in that, The S50 includes: S51, the state vector of the i-th Gaussian ellipsoid at time step t. Set as a two-dimensional vector: ； In the formula, g This represents the number of Gaussian ellipsoids. and These represent the weights and weight changes of the Gaussian ellipsoid, respectively. S52, the state transition model is constructed as follows: ； ； In the formula, Here is the state transition matrix. Let be the covariance matrix of the process noise. Indicates the first The Gaussian ellipsoids at time t are estimated based on the prior state at time t-1. Indicates the first Posterior state estimate of a Gaussian ellipsoid at time t-1 Indicates process noise; The observation model is constructed as follows: ； ； In the formula, Indicates the first The observation values ​​of a Gaussian ellipsoid at time t. Indicates the first Posterior state estimate of a Gaussian ellipsoid at time t For the observation matrix, To observe the noise, The covariance matrix of the observation noise; The actual observed vector This can be expressed as a weighted sum as follows: ； in, Let be the weight of the i-th component. This represents the volume of the i-th Gaussian ellipsoid. Let be the opacity of the i-th Gaussian ellipsoid; S53, for the i-th Gaussian ellipsoid, the iteration at time t is performed as follows: （1）； （2）； （3）； （4）； （5）； In the formula, Let (1) denote the innovation vector, and let (2) denote the first... A Gaussian ellipsoid in The prior state estimate at time t is given by Posterior state at time step Through the state transition matrix The prediction is a preliminary inference based on historical information; (2) indicates the first ellipsoid The prior covariance matrix at time t reflects the uncertainty of the prior state, derived from... Post-hoc covariance Transition matrix and its transpose After transformation, the noise covariance during the superposition process The result is used to represent the error of the state transition model; in (3), It is the Kalman gain, used to balance the reliability of prior predictions and observations, including the uncertainty of the projection of prior predictions. and observation noise covariance The greater the gain, the stronger the influence of the observation on the state update; (4) indicates the first ellipsoid The posterior state estimate at time t is the optimal state outcome, derived from the prior estimate. Add Kalman gain and innovation vector The product of these two terms is obtained by combining the information from prediction and observation; (5) represents the product of the two terms. ellipsoid The posterior covariance matrix at time 1 reflects the uncertainty of the state after fusion of observations, and is expressed through the identity matrix. Subtract Kalman gain and observation matrix The product of these factors, and then the prior covariance. Multiplying them together corrects for prior uncertainties; By repeating step S53 iteratively based on Kalman filtering, the state vector is obtained. Corresponding weights ; S54 employs a time decay and weighted averaging strategy to calculate the cumulative contribution of each Gaussian ellipsoid with view input, where the multi-view contribution weight of the i-th Gaussian ellipsoid is... The expression is: ； In the formula, The ID of the frame number processed by the system at the current moment. Indicates the time decay weight. Let Gaussian ellipsoid visibility index function be used. Contribute values ​​to a single view, where, Represented as: ； In the formula, This represents the set of pixels in the projection region of Gaussian point i in the k-th frame. For opacity, This represents the cumulative transmittance of the p-th pixel in the k-th frame.

5. The method as described in claim 4, characterized in that, The S60 includes: S61, Set a weighting coefficient The multi-view contribution weight value of the i-th Gaussian ellipsoid is represented in the following convex combination form: ； make for The discrete set of states, and assume that it contains A finite number of state values: ； In the formula, s represents a total of s states; Assumption The probability of state i occurring at time t is expressed as: , The state probability vector is , The state transition probability in one step is expressed as: ； express Given that the state is i at time t-1, what is the probability that the state transitions to state j at time t? ; S62, according to get The transition probability matrix : ； in, satisfy: and ; S63, Establish the state probability prediction equation: ； In the formula, Indicates the weight coefficient at the next time step. The state probability; For the current moment The state probability is a known quantity; S64, Construct an optimization problem based on the innovation vector at time k+1: ； ； Solve the optimization problem to obtain the minimized residual. ; S65, calculate the minimum residual respectively. With discrete state set Distance between each state value : ； Sure The residual corresponding to the minimum value Discrete state : ； S66, at time t+1 Belongs to a certain state in s ,but Given that the state is i at time t, what is the probability that the state will transition to state j at time t+1? The adjustment rules are as follows: ； in, , Indicates the weighting coefficient; S67, Set weight threshold and gradient threshold After a preset number of iterations, those that meet the requirements are eliminated. and The target Gaussian ellipsoid is obtained by using the Gaussian ellipsoid.

6. The method according to any one of claims 1 to 5, characterized in that, After step S60, the method further includes: S70, the RGB image obtained after rendering in step S60. and depth images , and the RGB image captured by the depth camera and the depth image Compare and calculate RGB loss separately. SSIM loss Deep loss and regularization loss : ； ； ； ； In the formula, Represents the mean of the image. Represents the variance of the image. This represents the covariance of two images. and To avoid constants with a denominator of 0, and They represent the first The scale of a Gaussian ellipsoid and its mean scale Indicates the number of valid pixels. This represents the optimized mask. Indicates Gaussian visibility. Indicates the number of valid depth values; S80, to minimize RGB loss Minimize SSIM loss Minimize depth loss And a regularization loss The weighted sum is determined as the target loss function. : ； In the formula, This represents the weight of the i-th loss component. and Let represent the scale of the i-th Gaussian ellipsoid and its mean scale, respectively; S90, with the target loss function The parameters are adjusted with the goal of minimization.

7. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the visual SLAM method based on 3D Gaussian spray in a dynamic environment as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the visual SLAM method based on 3D Gaussian spray in a dynamic environment as described in any one of claims 1 to 6.

Images