3D Gaussian SLAM method based on dynamic suppression and increment optimization

Abstract

The invention discloses a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization. The 3D Gaussian SLAM method comprises the following steps: acquiring an RGB-D image frame, constructing a dynamic object mask by using an integrated lightweight semantic segmentation network, synchronously rejecting a dynamic object in the acquired image frame in color and depth domains, and performing 3D Gaussian element initialization based on the image frame after dynamic pollution suppression; carrying out the iterative optimization estimation of the camera pose of a subsequent input image through constructing a depth-color combined residual constraint; and adding a new Gaussian element to each pixel in an area which is indicated by the mask and needs to be densified, and refining the scene representation of the area which is not fully rendered. The dynamic pollution suppression mechanism provided by the invention provides an accurate static scene basis for Gaussian element increment optimization, and the Gaussian element increment optimization improves the integrity and quality of map construction in turn, thereby jointly improving the positioning and mapping performance of the whole system.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision and robot autonomous navigation technology, in particular to a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization. BACKGROUND

[0002] In the field of mobile robots, SLAM (Simultaneous Localization and Mapping) technology is crucial, which enables robots to self-localize and perceive the surrounding environment in an unknown environment, thereby mapping and estimating their own position and pose. Traditional SLAM systems assume that the environment is static, i.e. landmarks used for localization are generally considered not to change. This assumption cannot usually be established in real life because dynamic objects (such as pedestrians or vehicles) are widespread, which can easily cause data association errors, and in turn have a serious impact on the accuracy of localization and the global consistency of the established map.

[0003] In the intelligent warehousing scenario, this challenge is more prominent. Intelligent warehousing is a modern warehousing ecosystem that integrates Internet of Things, big data, and artificial intelligence and other cutting-edge technologies, aiming to achieve efficient storage, sorting, and automation and intelligentization of warehouse management. However, the bins storing goods in the warehouse scenario are frequently moved, rearranged, or added, forcing the robot to constantly update its positioning information. In addition, the layout of various objects in the warehouse is adjusted, or even new obstacles appear, and the semiconductor intelligent mobile robot needs to have the ability to reposition and update the map in real time. At the same time, due to the existence of a large number of other dynamic AMR robots and manual operations (such as forklift operations or worker activities) in the warehouse, the AMR robot must be able to effectively interact with these dynamic elements while avoiding collision accidents.

[0004] To this end, the patent application with publication number CN118172514A discloses a robot dynamic SLAM method and system for intelligent warehouse dynamic environment, which designs a dynamic object detection network based on SLAM to detect segmented dynamic objects, and proposes a scene completion network to re-complete the areas removed by the segmentation network, to build a complete static scene, so as to improve the accuracy of positioning. However, this method still has some technical defects: (1) the completeness and information richness of the map construction are limited: this method is to complete the scene after removing the dynamic objects, and then construct the map based on the traditional SLAM network (such as ORB-SLAM2). The map constructed in this way is still a sparse point cloud map, which contains limited information, wastes a lot of information in the image frames, and is difficult for people to understand intuitively. (2) the adaptability to dynamic scenes is still insufficient: in a dynamic scene, although this method processes dynamic objects through dynamic object detection and completion network, it still may be affected by dynamic objects when processing complex dynamic environment, resulting in the decline of the accuracy and global consistency of map construction. (3) the utilization rate of sensor data is not high: this method mainly focuses on the processing of dynamic objects and the construction of static scenes, and does not fully utilize the rich data (such as depth information, color information, etc.) input by the visual sensor, resulting in low integration degree and utilization efficiency of information in the process of map construction and positioning.

[0005] Based on this, it is necessary to propose a new robot dynamic SLAM method for intelligent warehouse scenarios to meet the high requirements of the dynamic and uncertain warehouse environment on the autonomous positioning and mapping of intelligent mobile robots. SUMMARY

[0006] The present application aims to at least solve one of the technical problems existing in the related art to some extent.

[0007] The present application aims to at least solve one of the technical problems existing in the related art to some extent.

[0008] In order to achieve the above-mentioned purpose, the present application provides a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, comprising the following steps:

[0009] S1, acquire an image frame of RGB-D, construct a dynamic object mask by using an integrated lightweight semantic segmentation network, and establish a depth-color cross-modal mask synchronization mechanism, synchronize and remove the dynamic object in the acquired image frame in the color and depth domains, complete the dynamic pollution suppression of the image frame, and then perform 3D Gaussian element initialization based on the image frame after dynamic pollution suppression;

[0010] S2, after initialization, the subsequent input image is iteratively optimized and estimated for camera pose by constructing a depth-color joint residual constraint;

[0011] S3, after completing camera tracking and obtaining accurate camera pose estimation, the color value and depth value of the insufficient rendering area are repaired by historical frames and local adjacent area information, and based on the repaired color and depth information, a new Gaussian element is added to each pixel in the area that needs to be densified, and the scene representation of the insufficient rendering area is refined.

[0012] The further preferred technical solution of the application is that in step S1, the dynamic object mask is constructed by using an integrated lightweight semantic segmentation network, and the specific steps are as follows:

[0013] Deploy a real-time semantic segmentation network to input an RGB image Perform dynamic object instance segmentation to generate a pixel-level dynamic region binary mask , and further segment the binary mask of the movable object area affected by the dynamic object by fusing depth information , and add to obtain the dynamic object mask of the final RGB image , which is represented as:

[0014] ;

[0015] Masked RGB image The masked RGB image is represented by pixel-by-pixel and channel masking operation as follows:

[0016] ;

[0017] Wherein, represents the height of the input image, represents the width of the input image, represents the red, green and blue color channels of the input image; represents the pixel position belongs to the dynamic area, represents the pixel belongs to the static area; represents the pixel point in the planar image, is the coordinate of the pixel , is the abscissa, is taken as the ordinate.

[0018] As preferred, the depth-color cross-modal mask synchronization mechanism is established in step S1 to synchronize the dynamic objects in the acquired image frames in the color and depth domains, and the dynamic objects are removed, specifically:

[0019] After setting the mask for the RGB image, the RGB image and the depth image are associated according to the timestamp, and for the RGB image and the depth image with a timestamp interval less than 0.02s, the depth value of the corresponding position in the depth image in the region with a pixel value of zero in the RGB image is set to zero, which is represented as:

[0020] ;

[0021] wherein, is the RGB image the depth image with a timestamp interval less than 0.02s, and are real images obtained from the sensor; is the depth image after the depth value of the dynamic region is forcibly set to zero.

[0022] As preferred, the camera pose of the subsequent input image is iteratively optimized and estimated in step S2 by constructing a depth-color joint residual constraint, specifically:

[0023] The RGB and depth image information of the input image are used for joint registration, and the residual error between the rendered RGB image, depth image and contour image and the real depth image is used to optimize the pose estimation result; and a rendering quality threshold strategy is set, and only the pixel region with sufficient rendering optimization obtained from the contour image is used for pose gradient calculation to minimize the following loss to update the camera pose:

[0024] ;

[0025] wherein, represents the RGB image rendered by the 3D Gaussian method, represents the depth image rendered by the 3D Gaussian method, represents the rendered contour image; and are the weighted coefficients of the depth image rendering error and the weighted coefficients of the color rendering error, respectively, for measuring the importance of different information sources in the optimization process; represents that the rendering loss calculation is performed only in the pixel region with good optimization effect of the contour image, represents the loss function.

[0026] As preferred, after the camera tracking is completed and the accurate camera pose estimation is obtained in step S3, a reliable representation of the position distribution of the Gaussian cells in the scene is established in combination with the geometric information provided by the depth image, and an adaptive densification mechanism is started for the areas in the scene that are not sufficiently covered by the Gaussian model.

[0027] As preferred, the adaptive densification mechanism first sets a mask for determining which areas corresponding to the pixels need to be densified, denoted as:

[0028] ;

[0029] wherein, denotes the RGB image rendered by the 3D Gaussian method, denotes the depth image rendered by the 3D Gaussian method, denotes the rendered contour image; is the RGB image after dynamic masking, is the real depth image;

[0030] The mask indicates that new 3D Gaussian cells need to be added for densification in the following three cases, respectively:

[0031] (1) the area where the rendered contour image is insufficiently rendered;

[0032] (2) the area where the pixel value of the RGB image after dynamic masking is 0;

[0033] (3) the area where the depth value of the real depth image is less than the depth value of the rendered depth image, and the error of the depth value is greater than 50 times the median depth error MDE.

[0034] As preferred, for the area that is long-term shielded by the dynamic object mask, i.e. , by fusing the multi-frame static scene features and the local adjacent area information, the static content that conforms to the physical law is repaired in the area covered by the dynamic object mask , and then the repaired color and depth information are used for densification.

[0035] As preferred, in the area that needs to be densified, a new Gaussian cell is added for each pixel, the color c of the Gaussian cell is the repaired RGB value of the pixel, the center position μ is the coordinate based on the depth back-projection of the repaired pixel to the three-dimensional space, the opacity o is set to 0.5, and the radius r is the ratio of the depth real value to the focal length of the camera.

[0036] Another aspect of the present application provides a non-transitory computer readable storage medium having stored thereon computer instructions that cause a computer to execute the 3D Gaussian SLAM method based on dynamic inhibition and incremental optimization described above.

[0037] Still another aspect of the present application provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete the communication among each other through the communication bus, and the processor invokes the logic instructions in the memory to execute the 3D Gaussian SLAM method based on dynamic inhibition and incremental optimization described above.

[0038] Still another aspect of the present application provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, which, when executed by a processor, causes a computer to execute the 3D Gaussian SLAM method based on dynamic inhibition and incremental optimization described above.

[0039] Beneficial effects: The present application adopts a dynamic pollution inhibition mechanism based on a Gaussian initialization link. In the initialization phase of the 3D Gaussian element construction, dynamic feature active isolation is implemented. By integrating a lightweight semantic segmentation network to construct a dynamic object mask and establishing a depth-color cross-modal mask synchronization mechanism, the depth value of the dynamic region is forced to zero for dynamic pollution inhibition; fusion of depth information and semantic information dual discrimination ensures that the initialization phase of constructing the basic Gaussian model is not affected by dynamic pollution.

[0040] The present application adopts a robust pose optimization mechanism based on rendering residual constraint. In the iterative optimization phase of pose estimation, joint residual constraint of color-depth is implemented to enhance the stability and accuracy of camera pose estimation. By constructing a depth-color dual-channel residual loss function and designing corresponding weights according to the scene to measure the importance of different information sources; set up a rendering quality threshold strategy, only use the outline image to get the pixel area that is fully rendered and optimized for pose gradient calculation, to ensure the robustness and accuracy of camera pose optimization.

[0041] The present application adopts a Gaussian element incremental optimization mechanism for insufficient rendering areas. In the iterative process of 3D Gaussian element density optimization, the rendering hole problem caused by dynamic mask is solved. Through the information of historical frames and local adjacent areas, the insufficient rendering area is repaired for static scene, and the repaired color and depth information are used to incrementally optimize the Gaussian element in the scene. In the area indicated by the mask that needs to be densified, a new Gaussian element is added to each pixel, which has similar properties to the setting when initializing the Gaussian element. By adding new Gaussian elements, the scene representation of the insufficient rendering area is refined.

[0042] In summary, the 3D Gaussian SLAM method based on dynamic inhibition and incremental optimization of the application significantly improves the positioning robustness, consistency and integrity of scene reconstruction in dynamic environment. The dynamic pollution inhibition mechanism proposed in the application provides an accurate static scene basis for Gaussian element incremental optimization, which in turn improves the integrity and quality of map construction, thereby improving the positioning and mapping performance of the entire system. Therefore, the application is particularly suitable for complex working conditions such as dynamic goods handling in intelligent warehousing and AGV cluster operation, and provides high-precision environment model support for subsequent path planning, multi-machine cooperation and other upper-layer applications. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is a flow chart of the 3D Gaussian SLAM method based on dynamic inhibition and incremental optimization.

[0044] Figure 2 It is a comparison diagram of sparse point cloud map and 3D Gaussian map in scene representation; Figure 2 In figure (a), it is a comparison diagram of sparse point cloud map and 3D Gaussian map in fr1_desk sequence, Figure 2 In figure (b), it is a comparison diagram of sparse point cloud map and 3D Gaussian map in fr3_long_office_household sequence.

[0045] Figure 3 It is an effect diagram of dynamic pollution inhibition mechanism in Gaussian initialization link; Figure 3 In figure (a), it is a 3D Gaussian map without dynamic pollution inhibition mechanism, Figure 3 In figure (b), it is a 3D Gaussian map after adding dynamic pollution inhibition mechanism.

[0046] Figure 4 It is a comparison diagram of rendering scene results under different loss weights; Figure 4 In figure (a), it is a rendering scene diagram, In figure (b), it is a rendering scene diagram, Figure 4 In figure (c), it is a rendering scene diagram, In figure (d), it is a rendering scene diagram. Figure 4 Figure 4 In figure (e), it is a rendering scene diagram.

[0047] Figure 5 It is an effect diagram of Gaussian element incremental optimization mechanism for insufficient rendering area; Figure 5 In figure (a), it is a 3D Gaussian map without Gaussian element incremental optimization mechanism, Figure 5 In figure (b), it is a 3D Gaussian map after adding Gaussian element incremental optimization mechanism.

[0048] Figure 6 ​​​Fig. 3 is a comparison diagram of rendering results of the original method and the method of the present application in a dynamic scene; Figure 6 Fig. 3a is a comparison of rendering of the original method (left) and the method of the present application (right) in the fr3_walking_xyz sequence, Figure 6 Fig. 3b is a comparison of rendering of the original method (left) and the method of the present application (right) in the fr3_walking_static sequence, Figure 6 Fig. 3c is a comparison of rendering of the original method (left) and the method of the present application (right) in the fr3_walking_halfsphere sequence, Figure 6 Fig. 3d is a comparison of rendering of the original method (left) and the method of the present application (right) in the fr3_sitting_static sequence.

[0049] Figure 7 Fig. 4 is a comparison diagram of effects of the sparse point cloud map and the 3D Gaussian map established in an actual intelligent warehouse environment; Figure 7 Fig. 4a is an actual intelligent warehouse environment map, Figure 7 Fig. 4b is a sparse point cloud map established, Figure 7 Fig. 4c is a 3D Gaussian map established.

[0050] Figure 8 Fig. 5 is a comparison diagram of rendering results of the original method and the method of the present application in an actual dynamic intelligent warehouse environment; Figure 8 Fig. 5a is a rendering result obtained by the original method, Figure 8 Fig. 5b is a rendering result obtained by the method of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. They should not be understood as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. In the description of the present application, it should be understood that the terms used are only for the purpose of description, and should not be understood as indicating or implying relative importance.

[0052] The following will be described in conjunction with Figures 1-8 The present application provides a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, a non-transitory computer readable storage medium, an electronic device and a computer program product.

[0053] Embodiment 1: The present embodiment provides a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, as shown in the following steps: Figure 1 Fig. 1.

[0054] S1. The system inputs RGB-D image frames and implements a dynamic contamination suppression mechanism during initialization. To address the initialization defects of dynamic interference scenarios, this embodiment proposes a cross-modal dynamic mask suppression mechanism, which specifically includes:

[0055] Deploy a real-time semantic segmentation network on the input RGB image Perform dynamic object instance segmentation and generate pixel-level dynamic area binary masks , and fuse the depth information to further segment the binary mask of the movable object area affected by the dynamic object , add to get the dynamic object mask of the final RGB image , expressed as:

[0056] ;

[0057] Masked RGB image It is represented by pixel-by-pixel and channel-by-channel mask operations:

[0058] ;

[0059] in, represents the height of the input image, represents the width of the input image, Represents the red, green, and blue color channels of the input image; Indicates pixel position Belongs to the dynamic area, Represents pixels Belongs to the static area; Represents a pixel point in a plane image, It's a pixel The coordinates of is the horizontal axis, Is the vertical axis.

[0060] After setting the mask for the RGB image, the RGB image and the depth image are associated according to the timestamp. For the RGB image and the depth image with a timestamp interval less than 0.02s, the depth value of the corresponding position in the depth image at the area with zero pixel value in the RGB image is synchronously set to zero, which is expressed as:

[0061] ;

[0062] in, For RGB images Depth images with timestamp intervals less than 0.02s, and These are real images obtained from the sensor; a depth image with depth values of a dynamic region forced to zero.

[0063] After the dynamic contamination of the image frame is completed, the 3D Gaussian element initialization is performed based on the image frame after the dynamic contamination.

[0064] In recent years, with the emergence of 3D Gaussian technology, it is a new solution to apply it to the field of visual SLAM to deal with dynamic problems in the intelligent warehouse environment. The traditional visual SLAM system usually uses a sparse point cloud to represent the environment, such as ORB-SLAM2. Such a map is sparse, wasting a lot of information in the image frame, and it is difficult for people to understand. In contrast, the 3D Gaussian method uses anisotropic Gaussian elements to represent the environment, making the constructed map contain more information. At the same time, using 3D Gaussian technology can build a dense map of the scene, and through an adaptive density adjustment strategy to add or remove 3D Gaussian elements, the scene is easier for people to understand. However, for static occasions, directly using 3D Gaussian technology to construct a map will not cause the scene to be abnormal, but in a dynamic scene, directly using the 3D Gaussian mapping method will cause a part of the 3D Gaussian elements to fall on the dynamic object during initialization, and the information of this part of the Gaussian element will change with the movement of the dynamic object in the image frame, making the system unable to construct a globally consistent map.

[0065] In this embodiment, 3D Gaussian technology is used to construct a map of the scene, and a Gaussian element parameterization model containing an 8-dimensional feature vector is constructed, including 3 parameters representing the color information of RGB , 3 parameters representing the center position information of the Gaussian element , and the radius of the Gaussian element , the opacity of the Gaussian element . For a point in three-dimensional space, the Gaussian element distribution function is represented as:

[0066] ;

[0067] Through the form of differentiable rendering, the underlying Gaussian map is rendered into a high-fidelity color map, a depth map and an outline map under any camera reference frame. The gradient of the error between the underlying scene Gaussian representation and the camera parameters relative to the rendering result and the provided RGB-D frame is directly calculated, and the Gaussian parameters and the camera parameters are updated accordingly to minimize the error, thereby achieving high-precision fitting of the camera pose and the scene volume representation.

[0068] The RGB image rendering process is as follows:

[0069] Given a set of 3D Gaussian primitives and camera pose, all the primitives are sorted by their depth relative to the camera from near to far. Then the 2D projection of each primitive is blended in order by alpha blending in pixel space to render the RGB image. Each pixel The rendered color can be represented as:

[0070] ;

[0071] where is the RGB color vector of the th primitive, is the contribution weight of the th primitive at pixel , since the pixel is the pixel coordinate of the 3D space point projected to the 2D pixel plane, and and represent the position information and radius of the pixel after projection to the 2D pixel plane, which can be represented as:

[0072] , , ;

[0073] where is the known camera intrinsic matrix, is the extrinsic information including translation and rotation matrix when the camera moves in the th frame, is the known focal length of the camera, is the depth of the Gaussian center in the camera coordinate system, i.e. the z-axis coordinate.

[0074] In addition to rendering the RGB image, to fully utilize the information input from the visual sensor, the system also renders the input depth image in a similar way to color rendering. The rendering result can be directly compared with the input depth value image and return the gradient value, which can be represented as:

[0075] ;

[0076] At the same time, a contour image is rendered to determine whether the information contained in the pixel is visible in the current frame map, which can be represented as:

[0077] ;

[0078] For the first input image, the tracking step is skipped and the camera pose is initialized. Since the rendered silhouette is empty, all pixels are used to initialize new Gaussians. For each pixel, a new Gaussian with the following properties is added: color c is the RGB value of the pixel, center position μ is the coordinate in 3D space based on the depth back-projection of the pixel, opacity o is set to 0.5, and radius r is expressed as:

[0079] ;

[0080] wherein is the depth ground truth value, is the camera focal length, and the ratio is calculated to ensure that the radius is equivalent to 1 pixel when projected to the 2D image.

[0081] In this embodiment, through the cross-modal dynamic mask suppression mechanism, the dynamic objects are simultaneously removed in the color and depth domains, solving the problem of interference of dynamic objects with Gaussian initialization.

[0082] S2, after initialization, the camera pose of the subsequent input image is iteratively optimized and estimated by constructing a depth-color joint residual constraint. Specifically:

[0083] The RGB and depth map information of the input image is used for joint registration, and the residual error between the rendered RGB image, depth image, and silhouette image and the real depth map is used to optimize the pose estimation result; and a rendering quality threshold strategy is set, and only the pixel area that is fully rendered and optimized is used to calculate the pose gradient to minimize the following loss to update the camera pose:

[0084] ;

[0085] wherein, represents the RGB image rendered by the 3D Gaussian method, represents the depth image rendered by the 3D Gaussian method, represents the rendered silhouette image; and are the weighting coefficients of the depth map rendering error and the color rendering error, respectively, and are used to measure the importance of different information sources in the optimization process; represents the loss function. represents that the rendering loss calculation is only performed in the pixel area with good optimization effect, so as to enhance the stability and accuracy of the camera pose estimation. The new input image may contain new perspectives or information that has not appeared in the map, or even content that has not been optimized before, and therefore, through the above loss function design, the accuracy of the overall pose estimation can be improved.

[0086] S3, after completing camera tracking and obtaining accurate camera pose estimation, a Gaussian element incremental optimization mechanism is used for insufficient rendering area.

[0087] After completing camera tracking and obtaining accurate camera pose estimation, combined with the geometric information provided by the depth image, a reliable representation of the scene Gaussian element position distribution has been established. However, for the areas in the scene that have not been fully covered by the existing Gaussian model, an adaptive densification mechanism needs to be started to improve the reconstruction quality. The adaptive densification mechanism first sets a mask for judging which pixels correspond to the area that needs to be densified, denoted as:

[0088] ;

[0089] Mask Indicates that new 3D Gaussian elements need to be added for densification in the following three cases:

[0090] (1) the area of the rendered outline image that is insufficiently rendered;

[0091] (2) the area of the RGB image whose pixel value is 0 after dynamic masking;

[0092] (3) the depth value of the real depth image is less than the depth value of the rendered depth image, and the error of the depth value is greater than 50 times the median depth error MDE.

[0093] However, the area that is long-term shielded by the dynamic object mask, i.e. , cannot trigger the densification operation, resulting in the continuous existence of geometric holes in the dynamic residual area. For these areas, by fusing multi-frame static scene features and local adjacent area information, the static content that conforms to the physical law is repaired in the area covered by the dynamic object mask , and then the repaired color and depth information are used for densification. In the area indicated by the mask that needs to be densified, a new Gaussian element is added for each pixel, and the method is similar to the initialization stage. The color c of this Gaussian element is the repaired RGB value of the pixel, the center position μ is the coordinate based on the depth back-projection of the repaired pixel to the three-dimensional space, the opacity o is set to 0.5, and the radius r is the ratio of the depth real value to the camera focal length.

[0094] The present embodiment realizes high-quality rendering of the scene details in the insufficiently rendered area by adding new Gaussian elements.

[0095] The method of the present application is compared with the method of establishing a sparse point cloud map and the original method without adding dynamic pollution suppression, rendering residual constraint and Gaussian incremental optimization.

[0096] As Figure 2As shown, the difference between the sparse point cloud map and the 3D Gaussian map in scene representation is compared, the sparse point cloud map contains less effective information, wastes a large amount of image frame information obtained from the sensor and is difficult for people to understand, and the 3D Gaussian map can accurately and finely represent the scene, fully utilizes the image information input by the sensor, and is easy for people to understand.

[0097] As shown in the figure, Figure 3 The effect of the dynamic pollution suppression mechanism in the Gaussian initialization link is shown, and in the initialization rendering scene without the dynamic pollution suppression mechanism, the residual image of the dynamic object can be found, and the information belonging to the dynamic object will seriously affect the rendering of the overall map, and a globally consistent map cannot be constructed, and after the dynamic pollution suppression mechanism is added, the information of the dynamic object will not appear in the initialized scene, and the influence of the dynamic object on the map construction is well solved.

[0098] As shown in the figure, Figure 4 The rendering scene results under different loss weights are compared, and it can be seen that the appropriate weight coefficient has a great influence on the final result, and the accuracy of the camera pose estimation is ensured to further improve the rendering effect of the scene map.

[0099] As shown in the figure, Figure 5 The effect of the Gaussian element increment optimization mechanism for rendering insufficient areas is shown, and from the figure it can be seen that when the Gaussian element increment optimization mechanism is not introduced, the final rendering result of the scene has the problem of rendering holes, and the dynamic object shielding area cannot be effectively filled with details; after adding the Gaussian element increment optimization mechanism, the depth-color image data after static repair is used to drive the Gaussian densification, and high-quality rendering of scene details in the insufficient rendering area is realized.

[0100] As shown in the figure, Figure 6 The 3D Gaussian rendering results of the scenes before and after the improvement in the standard TUM dynamic data set are shown, and from the figure it can be seen that through the improvement in scene initialization, Gaussian densification and color and depth loss weight, the method proposed in the application still has very excellent rendering performance in dynamic scenes, fully proving the effectiveness of the method.

[0101] As shown in the figure, Figure 7 The difference between the sparse point cloud map and the 3D Gaussian map in scene representation constructed in the actual intelligent warehouse environment is shown, and from the figure it can be seen that the scene represented by the sparse point cloud map contains too little information and is difficult for people to understand, while the scene rendered by the 3D Gaussian map fully utilizes the information in the input image frame, and the rendered scene is clearer, more detailed and specific, and easy for people to understand.

[0102] As shown in the figure, Figure 8As shown, the comparison of the rendering effects of the original method and the improved method in the actual dynamic intelligent warehouse environment is shown, and from the figure, it can be seen that the improved method proposed in the application well relieves the influence of dynamic objects on the global consistent map construction in the actual dynamic scene, and the rendered scene is clearer.

[0103] The method of the application is compared with the original method and the same type of work on the TUM data set, as shown in Tables 1 and 2, by comparing the rendering quality index peak signal-to-noise ratio (PSNR) and the positioning performance index absolute trajectory error (ATE), the rendering quality of the application is improved by about 30% compared with the original method, and the positioning accuracy is improved by about 90%, and the rendering quality is also improved compared with the same type of work, fully showing the superiority and necessity of the innovative method proposed in the application. The scene map constructed by the original method and the traditional method will be seriously disturbed by dynamic factors, and the rendered scene quality is not high. The dynamic pollution suppression mechanism is used in the application, which well avoids the influence of dynamic factors on the map rendering quality. At the same time, the added Gaussian element increment optimization mechanism makes the details of the rendered scene further improved, and higher scene rendering quality is obtained.

[0104] Table 1 Comparison of positioning and mapping performance of the original method and the method of the application

[0105]

[0106] Table 2 Comparison of rendering quality of the method of the application and the same type of work

[0107]

[0108] The application also compares the influence of different weight designs in the rendering residual constraint on the positioning performance and the mapping quality, as shown in Table 3, a suitable weight design can bring 92.9% and 39.9% performance improvement to the positioning accuracy and the mapping quality of the system respectively, fully showing the importance of the design of the rendering residual constraint.

[0109] Table 3 Analysis of system positioning performance and mapping quality under different loss weights

[0110]

[0111] The dynamic pollution suppression mechanism proposed in the application provides an accurate static scene basis for the Gaussian element increment optimization, and the Gaussian element increment optimization in turn improves the integrity and quality of the map construction, thereby improving the positioning and mapping performance of the whole system. In summary, the application provides an efficient and accurate 3D Gaussian SLAM solution for complex dynamic scenes of intelligent warehouses through multi-dimensional improvement.

[0112] Embodiment 2: The embodiment provides a non-transitory computer-readable storage medium having stored thereon computer instructions that cause a computer to execute a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, the method comprising the following steps:

[0113] S1, acquire an image frame of RGB-D, construct a dynamic object mask by using an integrated lightweight semantic segmentation network, and establish a depth-color cross-modal mask synchronization mechanism, synchronously remove the dynamic object in the acquired image frame in the color and depth domains, complete dynamic pollution suppression of the image frame, and then initialize 3D Gaussian elements based on the image frame after dynamic pollution suppression;

[0114] S2, after initialization, iteratively optimize and estimate the camera pose of the subsequent input image by constructing a joint residual constraint of depth-color;

[0115] S3, after completing camera tracking and acquiring accurate camera pose estimation, repair the color value and depth value of the insufficient rendering area by using historical frame and local adjacent area information, add a new Gaussian element to each pixel in the area indicated by the mask that needs to be densified based on the repaired color and depth information, and refine the scene representation of the insufficient rendering area.

[0116] Embodiment 3: The embodiment provides an electronic device, which can include a processor, a communications interface, a memory and a communications bus, wherein the processor, the communications interface and the memory complete mutual communication through the communications bus. The processor can call logical instructions in the memory to execute a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, the method comprising the following steps:

[0117] S1, acquire an image frame of RGB-D, construct a dynamic object mask by using an integrated lightweight semantic segmentation network, and establish a depth-color cross-modal mask synchronization mechanism, synchronously remove the dynamic object in the acquired image frame in the color and depth domains, complete dynamic pollution suppression of the image frame, and then initialize 3D Gaussian elements based on the image frame after dynamic pollution suppression;

[0118] S2, after initialization, iteratively optimize and estimate the camera pose of the subsequent input image by constructing a joint residual constraint of depth-color;

[0119] S3, after completing camera tracking and obtaining accurate camera pose estimation, static repair is performed on color values and depth values of insufficient rendering areas through historical frames and local adjacent area information, based on repaired color and depth information, a new Gaussian element is added to each pixel in areas indicated by the mask that need to be densified, and scene representation of the insufficient rendering areas is refined.

[0120] In addition, the logic instructions in the memory described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0121] Embodiment 4: The present embodiment provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, and the computer program being executable by a processor, so that a computer can execute a 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, the method comprising the following steps:

[0122] S1, obtaining an image frame of RGB-D, constructing a dynamic object mask using an integrated lightweight semantic segmentation network, and establishing a depth-color cross-modal mask synchronization mechanism, synchronously removing dynamic objects in the color and depth domains in the obtained image frame, completing dynamic pollution suppression of the image frame, and then initializing 3D Gaussian elements based on the image frame after dynamic pollution suppression;

[0123] S2, after initialization, iteratively optimizing and estimating the camera pose of the subsequent input image through the construction of the depth-color joint residual constraint;

[0124] S3, after completing camera tracking and obtaining accurate camera pose estimation, static repair is performed on color values and depth values of insufficient rendering areas through historical frames and local adjacent area information, based on repaired color and depth information, a new Gaussian element is added to each pixel in areas indicated by the mask that need to be densified, and scene representation of the insufficient rendering areas is refined.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A 3D Gaussian SLAM method based on dynamic suppression and incremental optimization, characterized in that: The following steps are involved: S1. Acquire RGB-D image frames, construct dynamic object masks using an integrated lightweight semantic segmentation network, and establish a depth-color cross-modal mask synchronization mechanism to simultaneously remove dynamic objects in the acquired image frames in both the color and depth domains, completing dynamic contamination suppression for the image frames. 3D Gaussian element initialization is then performed based on the image frames after dynamic contamination suppression. S2. After initialization, the camera pose is iteratively optimized and estimated for subsequent input images by constructing a joint depth-color residual constraint. S3. After completing camera tracking and obtaining accurate camera pose estimation, the color and depth values ​​of the insufficiently rendered areas are statically repaired through historical frames and local neighboring area information. Based on the repaired color and depth information, a new Gaussian element is added to each pixel in the area that needs densification to refine the scene representation of the insufficiently rendered areas.

2. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 1, wherein In step S1, the dynamic object mask is constructed using an integrated lightweight semantic segmentation network. The specific steps are as follows: Deploy a real-time semantic segmentation network on the input RGB image Perform dynamic object instance segmentation and generate pixel-level dynamic area binary masks , and fuse the depth information to further segment the binary mask of the movable object area affected by the dynamic object , add to get the dynamic object mask of the final RGB image , expressed as: ； RGB image after dynamic masking It is represented by pixel-by-pixel and channel-by-channel mask operations: ； in, represents the height of the input image, represents the width of the input image, Represents the red, green, and blue color channels of the input image; Indicates pixel position Belongs to the dynamic area, Represents pixels Belongs to the static area; Represents a pixel point in a plane image, It's a pixel The coordinates of is the horizontal axis, Is the vertical axis.

3. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 2, wherein In step S1, a depth-color cross-modal mask synchronization mechanism is established to remove dynamic objects in the acquired image frames synchronously in the color and depth domains. Specifically: After setting the mask for the RGB image, the RGB image and the depth image are associated according to the timestamp. For the RGB image and the depth image with a timestamp interval less than 0.02s, the depth value of the corresponding position in the depth image at the area with zero pixel value in the RGB image is synchronously set to zero, which is expressed as: ； in, For RGB images Depth images with timestamp intervals less than 0.02s, and These are real images obtained from the sensor; The depth image after forcing the depth values ​​of dynamic areas to zero.

4. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 1, wherein In step S2, the camera pose is iteratively optimized and estimated for subsequent input images by constructing a joint depth-color residual constraint, specifically: The RGB and depth map information of the input image are jointly registered, and the pose estimation result is optimized based on the residual between the rendered RGB image, depth image, and silhouette image and the real image. A rendering quality threshold strategy is established, and only the pixel area of ​​the silhouette image that has been fully rendered and optimized is used for pose gradient calculation to minimize the following loss to update the camera pose: ； in, Represents an RGB image rendered using the 3D Gaussian method, Represents the depth image rendered using the 3D Gaussian method, Represents the rendered outline image; and They are the weighted coefficients of depth map rendering error and color rendering error, respectively, used to measure the importance of different information sources in the optimization process; Indicates that rendering loss calculation is performed only in pixel areas where the contour image optimization effect is good. express Loss function.

5. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 1, wherein In step S3, after completing camera tracking and obtaining accurate camera pose estimation, a reliable representation of the scene Gaussian element position distribution is established in combination with the geometric information provided by the depth image, and an adaptive densification mechanism is started for areas in the scene that are not fully covered by the Gaussian model.

6. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 5, characterized in that The adaptive densification mechanism first sets a mask to determine which pixels correspond to the areas that need to be densified, expressed as: ； in, Represents an RGB image rendered using the 3D Gaussian method, Represents the depth image rendered using the 3D Gaussian method, Represents the rendered outline image; is the RGB image after dynamic masking, is the real depth image; Mask It indicates that new 3D Gaussian elements need to be added for densification in the following three cases: (1) The rendered contour image has insufficiently rendered areas; (2) The area where the pixel value of the RGB image after dynamic masking is 0; (3) The depth value of the real depth image is smaller than the depth value of the rendered depth image, and the error of the depth value is greater than 50 times the median depth error MDE.

7. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 6, characterized in that For areas that are long-term masked by dynamic object masks, By fusing multi-frame static scene features with local neighboring area information, dynamic object masks The coverage area repairs the static content of the RGB image and the depth image in accordance with the physical laws, and then densifies it using the repaired color and depth information.

8. The 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to claim 7, characterized in that In step S3, a new Gaussian element is added to each pixel in the area that needs to be densified. The color c of the Gaussian element is the RGB value of the pixel after restoration, the center position μ is the coordinate back-projected into the three-dimensional space based on the depth of the restored pixel, the opacity o is set to 0.5, and the radius r is the ratio of the true depth value to the camera focal length.

9. A non-transitory computer-readable storage medium, characterized in that Computer instructions are stored thereon, and the computer instructions enable the computer to execute the 3D Gaussian SLAM method based on dynamic suppression and incremental optimization as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer executes the 3D Gaussian SLAM method based on dynamic suppression and incremental optimization according to any one of claims 1 to 7.

Images