Map construction methods, devices, and media based on multimodal perception and Gaussian primitives

Abstract

This invention discloses a map construction method, apparatus, and medium based on multimodal perception and Gaussian primitives. The method includes acquiring the initial camera pose of a mobile robot, calculating initial 3D Gaussian primitives based on the initial camera pose, and storing them in an optimization window; periodically acquiring the current frame image and calculating the current camera pose of the mobile robot, while simultaneously optimizing the current camera pose by combining it with the camera pose of the previous keyframe; determining whether the current frame image is a keyframe image based on the optimized current frame camera pose; if so, generating a new set of 3D Gaussian primitives based on the current frame image, and adding the current frame image, the current frame camera pose, and the new 3D Gaussian primitives to the optimization window; and constructing a Gaussian map based on all keyframes and their corresponding 3D Gaussian primitives in the optimization window. This invention can solve the problem in existing technologies where robot map construction cannot simultaneously achieve high-quality scene reproduction and map stability.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a map construction method, apparatus, and medium based on multimodal perception and Gaussian primitives. Background Technology

[0002] Visual SLAM technology is the core of autonomous localization and navigation for mobile robots and AR devices. Currently, the mainstream visual SLAM technologies can be mainly divided into the following two categories: 1) Feature-point-based methods, such as the ORB-SLAM series, construct a sparse 3D point cloud map by extracting and matching feature points in an image. The advantages of this method are high computational efficiency and robustness; however, its map information is sparse and lacks scene texture, appearance, and detail information, making it unsuitable for high-quality scene reproduction. 2) Direct methods generally estimate camera pose and construct semi-dense or dense geometric maps by minimizing photometric errors between images. Although this method can obtain denser maps, the constructed maps usually exist in the form of point clouds or patches, with low rendering fidelity and are more sensitive to changes in lighting, resulting in unstable maps. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a map building method based on multimodal perception and Gaussian primitives, which can solve the problem that existing map building methods in robot navigation cannot simultaneously achieve high-quality scene reproduction and stability.

[0004] The second objective of this invention is to provide a map building device based on multimodal perception and Gaussian primitives, which can solve the problem that existing map building in robot navigation cannot simultaneously achieve high-quality scene reproduction and stability.

[0005] The third objective of this invention is to provide a computer-readable storage medium that can solve the problem that existing map building in robot navigation cannot simultaneously achieve high-quality scene reproduction and stability.

[0006] One of the objectives of this invention is achieved through the following technical solution: A map construction method based on multimodal perception and Gaussian primitives is characterized by the following steps: An initial step involves acquiring initial frame data from the RGB-D camera, IMU sensor, and LiDAR installed on the mobile robot during initialization, preprocessing the initial frame data, initializing the camera pose and 3D Gaussian primitives of the mobile robot based on the preprocessed initial frame data, and adding the initial frame as a keyframe to the system's optimization window; the initial frame data includes RGB color images, depth maps, IMU data, and LiDAR point cloud data. Visual optimization steps: During the movement of the mobile robot, the current frame data is acquired in real time, and the camera pose of the current frame is optimized based on the current frame data and the camera pose of the previous key frame; the current frame data includes RGB color image, depth map, IMU data and LiDAR point cloud data; Judgment steps: Determine whether the current frame is a keyframe based on the optimized camera pose. If so, generate a new set of 3D Gaussian primitives based on the depth map of the current frame, use the current frame as the current keyframe, and add it to the optimization window. Optimization steps: Select multiple keyframes from the optimization window according to preset rules and optimize the camera pose and 3D Gaussian primitive parameters of multiple keyframes, and then update each keyframe in the optimization window. Map building steps: Construct an environment map for the mobile robot's navigation based on the 3D Gaussian primitives of all keyframes within the optimization window.

[0007] Furthermore, the preprocessing of the RGB color image, depth map, IMU data, and lidar point cloud data specifically includes: deblurring the RGB color image based on IMU data, distortion correction of the lidar point cloud based on IMU data, and completion processing of the depth map based on the RGB color image or lidar point cloud data. The specific steps for generating the camera pose of the mobile robot include: when the RGB-D camera is a monocular camera, setting an initial time period and continuously acquiring multiple frames of RGB color images within the initial time period, and calculating the relative camera pose of the mobile robot based on the epipolar geometry principle, and then converting the relative camera pose of the mobile robot into the camera pose of the mobile robot using the triangulation principle; when the RGB-D camera is a binocular camera, acquiring the RGB color image of the current frame through the RGB-D camera and obtaining depth information based on the RGB color image, and then obtaining the camera pose of the mobile robot based on the depth information. The initial size, gravity direction, and velocity of the mobile robot are estimated by pre-integrating the IMU data, and then the camera pose of the mobile robot is estimated based on the initial size, gravity direction, and velocity of the robot. Generating a set of 3D Gaussian primitives specifically includes: back-projecting each point of the depth map using the camera memory matrix to obtain the three-dimensional coordinates of each point, and then initializing a three-dimensional Gaussian primitive based on the three-dimensional coordinates of each point; the parameters of the three-dimensional Gaussian primitive include the position mean, covariance matrix, color coefficient, effective opacity, and semantic feature vector.

[0008] Furthermore, determining whether the current frame is a keyframe specifically includes: the current frame is a keyframe when any of the following conditions are met; The judgment conditions include: Condition 1, the translation distance between the camera pose of the current frame and the previous keyframe is greater than a first preset threshold; Condition 2, the rotation angle between the camera pose of the current frame and the previous keyframe is greater than a second preset threshold; Condition 3, the average overlap area between the RGB color image of the current frame and the RGB color image of the previous keyframe is less than a third preset threshold; Condition 4, the data acquisition time of the current frame and the data acquisition time of the previous keyframe are greater than a fourth preset interval; Condition 5, the proportion of new observation points in the current frame is greater than a fifth preset threshold; Condition 6, the image quality of the current frame is less than a sixth preset threshold; Condition 7, the photometric error of the current frame is less than a seventh preset threshold.

[0009] Furthermore, the map building step includes: firstly, building a map database based on all keyframes within the optimization window, and then projecting the 3D Gaussian primitives of each keyframe in the map database onto a horizontal ground grid to generate an environment map.

[0010] Furthermore, when the camera pose and 3D Gaussian primitive of the current keyframe are added to the optimization window in the judgment step, the loop closure detection process is entered. The loop closure test process specifically includes: Step 1: Extract visual feature data from the RGB color image of the current keyframe and from the RGB color image of each historical keyframe, and calculate the similarity between the visual feature data of the current frame and the visual feature data of each historical keyframe. Step 2: Based on the similarity calculation results, a candidate loop closure frame set is obtained, and each loop closure candidate frame in the candidate loop closure frame set is geometrically verified to determine whether the corresponding loop closure candidate frame is valid; the candidate loop closure frame set includes several candidate loop closure frames, and the visual feature data of each candidate loop closure frame has a similarity to the visual feature data of the current keyframe that is greater than a preset threshold. Step 3: Based on the judgment results, obtain several loop closure frames from the candidate loop closure frame set, and construct a camera pose graph based on the camera pose of the current keyframe and the camera pose of each loop closure frame. Step 4: Optimize each keyframe in the camera pose graph according to the camera pose constraints and the graph optimization library to solve for the camera pose constraints, and optimize the camera pose of each historical keyframe in the optimization window according to the camera pose constraints.

[0011] Furthermore, the visual optimization step specifically includes: estimating the camera pose estimate of the current frame based on the IMU data of the current frame; The predicted image of the current frame is generated by differentiable Gaussian rendering based on the estimated camera pose of the current frame and the 3D Gaussian map. Photometric loss, depth loss, and IMU constraint loss are calculated based on the RGB color image and the predicted image of the current frame. Then, a joint loss function is constructed based on the photometric loss, depth loss, and IMU constraint loss. The Adam optimizer is used to iterate and optimize the joint loss function to optimize the camera pose of the current frame.

[0012] Furthermore, after the judgment step, the method further includes: selecting multiple key frames from the optimization window according to preset rules, optimizing the camera pose and 3D Gaussian primitive parameters of the multiple key frames, and updating the optimized camera pose and 3D Gaussian primitive parameters of the multiple key frames to the optimization window. The optimization of camera pose and 3D Gaussian primitive parameters for multiple keyframes specifically includes: constructing a joint optimization objective function based on photometric factors, IMU pre-integration factors, and geometric prior factors; using the camera pose and 3D Gaussian primitive parameters for multiple keyframes as optimization variables; solving a nonlinear least squares problem on the joint optimization objective function to solve for the optimization variables; and thus optimizing the camera pose and 3D Gaussian primitive parameters for the keyframes.

[0013] Furthermore, optimizing the camera pose and parameters of the 3D Gaussian primitives for multiple keyframes specifically includes: performing corresponding operations on the corresponding 3D Gaussian primitives based on the gradient information and scale of each 3D Gaussian primitive; the operations include splitting, cloning, or culling. When the gradient metric of a 3D Gaussian primitive is greater than the eighth preset threshold and the scale metric is greater than the ninth preset value, the 3D Gaussian primitive is split into two. When the gradient metric of a 3D Gaussian primitive is greater than the eighth preset threshold and the scale metric is less than or equal to the ninth preset threshold, a new 3D Gaussian primitive is generated by cloning the 3D Gaussian primitive. The 3D Gaussian primitive is deleted when its effective transparency is greater than the tenth preset threshold or when it is not observed.

[0014] The second objective of this invention is achieved by the following technical solution: A map building apparatus based on multimodal perception and Gaussian primitives includes a memory and a processor. The memory stores a map building program that runs on the processor. The map building program is a computer program. When the processor executes the map building program, it implements the steps of the map building method based on multimodal perception and Gaussian primitives as one of the objectives of this invention.

[0015] The third objective of this invention is achieved by the following technical solution: A computer-readable storage medium storing a map-building program thereon, the map-building program being a computer program, which, when executed by a processor, implements the steps of a map-building method based on multimodal perception and Gaussian primitives as one of the objectives of this invention.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention uses multiple sensors to perceive the pose of a mobile robot in real time and combines this with 3D Gaussian primitives to construct a 3D Gaussian map, thereby providing a powerful perception and decision-making foundation for the autonomous navigation of the mobile robot, enabling the mobile robot to operate more intelligently and reliably in complex and ever-changing environments. Attached Figure Description

[0017] Figure 1 The flowchart of the map construction method based on multimodal perception and Gaussian primitives provided by the present invention is shown below. Figure 2 for Figure 1 The flowchart for step S3 in the process; Figure 3 for Figure 1 The flowchart for step S5 in the process. Detailed Implementation

[0018] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments. Example 1

[0019] Based on the shortcomings of existing visual SLAM, this invention can combine high-fidelity map representation with real-time online localization and mapping capabilities to improve the performance of mobile robots in mapping, recognition, and localization.

[0020] like Figure 1 As shown, the present invention provides a preferred embodiment of a map construction method based on multimodal perception and Gaussian primitives, comprising: Step S1: When initializing the mobile robot, the initial frame data is obtained from the RGB-D camera, IMU sensor and LiDAR installed on the mobile robot and the initial frame data is preprocessed.

[0021] Specifically, the initial frame data includes RGB color images, depth maps, IMU data, and LiDAR point cloud data. This invention acquires color images, depth maps, point cloud data, and motion data during the movement of a mobile robot by installing multiple cameras or sensor devices, thereby calculating the robot's pose and achieving localization.

[0022] In addition, to ensure the synchronization of multi-mode data, this invention also uses Zhang Zhengyou's checkerboard calibration method to complete the intrinsic parameter calibration of the RGB-D camera, uses Kalibr tools to complete the extrinsic parameter calibration of the RGB-D camera-IMU sensor, and uses a target board to complete the extrinsic parameter calibration of the LiDAR-RGB-D camera. Simultaneously, hardware trigger signals are used to achieve time synchronization of multiple devices to ensure synchronized data acquisition. For example, GPIO trigger signals are used for synchronization with the IMU sensor, and the LiDAR is synchronized with the system clock via the PPS signal. Furthermore, for devices or sensors where hardware signal synchronization is not possible, linear interpolation can be used for software time synchronization.

[0023] The preprocessing of the aforementioned RGB color images, depth maps, point cloud data, and motion data specifically includes: deblurring of RGB color images based on IMU data, distortion correction of LiDAR point cloud data based on IMU data, and completion processing of depth maps based on RGB color images or LiDAR point cloud data. This preprocessing improves the image quality of RGB color images and depth maps, as well as the accuracy of LiDAR point cloud data, providing accurate data for subsequent pose calculations. Furthermore, different devices are used in different scenarios. For example, in practical applications, RGB-D cameras are more suitable for scenes with rich textures and good lighting; conversely, IMU sensors are more suitable for scenes with fast movement, insufficient lighting, or lack of texture. Therefore, this invention installs an RGB-D camera, an IMU sensor, and a LiDAR on the mobile robot to fuse visual and inertial data, solving the problem of inaccurate data from a single dimension in different scenarios. Specifically, when a mobile robot moves quickly, RGB color images may exhibit motion blur. This invention employs an IMU-based motion deblurring method, which uses IMU integration to estimate the camera's motion trajectory within the exposure time to construct a blur model. Then, the blur model is deconvolved to improve the clarity of the RGB color image and enhance image quality.

[0024] In addition, during the rotational scanning process of the lidar, the point cloud data scanned at different times is distorted due to the movement of the mobile robot. Therefore, an IMU data-assisted point cloud distortion correction method is used to transform the points in each lidar point cloud data to the coordinate system at the start of the scan to complete the distortion correction.

[0025] For areas such as glass and mirrors, the depth map acquired by the RGB-D camera may contain holes. To address these holes, this invention employs a guided filtering method, using an RGB color image as a guide image to complete the depth map. Simultaneously, LiDAR point cloud data can be projected onto a two-dimensional plane to generate a sparse depth map, which is then used to generate a dense depth map using guided filtering, thus supplementing the overall depth map and achieving completeness.

[0026] Step S2: Initialize the camera pose and 3D Gaussian primitives of the mobile robot based on the preprocessed initial frame data, and add the initial frame as a key frame into the system's optimization window.

[0027] Specifically, when locating a mobile robot, it is first necessary to obtain the robot's initial position, posture, and orientation, that is, to determine the robot's initial pose. Since the RGB-D camera, IMU sensor, and LiDAR are all fixed to the mobile robot, there is a fixed rigid body transformation relationship between the camera pose and the robot's pose. Therefore, this invention calculates the camera pose using the RGB-D camera, IMU sensor, and LiDAR to obtain the robot's pose. For ease of consistent description, all poses mentioned in this invention refer to the camera pose.

[0028] More preferably, the camera pose can be acquired either from the RGB color image of the RGB-D camera or calculated using IMU data from the IMU sensor. Specifically, when calculating the camera pose using the RGB color image, the camera pose is derived from the current position and orientation of the RGB-D camera in the world coordinate system. When calculating the camera pose using the IMU sensor, the robot's motion state is estimated through pre-integration, and the camera pose is obtained after transformation using a calibrated extrinsic parameter matrix.

[0029] The method for calculating camera pose differs depending on the type of RGB-D camera. Specifically, when the RGB-D camera is a monocular camera: an initial time period is set, and multiple frames of RGB color images are continuously acquired within that period. The relative camera pose of the mobile robot is calculated using epipolar geometry principles, and triangulation is then used to obtain the initial camera pose. When the RGB-D camera is a stereo camera, the current frame image is acquired using the RGB-D camera, and depth information is derived from this image. The initial camera pose of the mobile robot is then obtained based on this depth information.

[0030] When there is insufficient light, the IMU sensor can be activated to obtain IMU data, and the IMU data can be pre-integrated to estimate the initial size, gravity direction and velocity of the mobile robot. Then, the initial camera pose of the mobile robot can be estimated based on the initial size, gravity direction and velocity of the mobile robot.

[0031] Once the initial camera pose of the mobile robot is obtained, a rigid body transformation is performed based on the initial camera pose to obtain the initial pose of the mobile robot.

[0032] Furthermore, this invention also initializes the 3D Gaussian primitives of the 3D Gaussian map using a depth map. The Gaussian map is composed of multiple 3D Gaussian primitives with 3D spatial distribution, providing a means for the mobile robot to understand the world. It primarily involves 3D Gaussian sputtering technology to construct the corresponding environmental model, realizing a new generation of map representation for mobile robots. The depth map is processed through back projection to generate an initial set of 3D Gaussian primitives, i.e., a set of 3D Gaussian primitives, to establish an initial layered spherical Gaussian plot.

[0033] Specifically, any pixel in the initial frame depth map is set. Its depth value Through the camera intrinsic parameter matrix The pixel is back-projected to obtain the coordinates of the 3D point, specifically as follows: (1); in, , , These are the focal lengths of the RGB-D camera. , These are the principal point coordinates of the RGB-D camera, and both are intrinsic parameters of the RGB-D camera; This is the depth value of that pixel.

[0034] The present invention further initializes the above three-dimensional points to obtain a 3D Gaussian primitive, that is, a set of 3D Gaussian primitives can be generated based on a depth map. A 3D Gaussian primitive includes a position mean, a covariance matrix, a color coefficient, and an effective opacity. The position mean is used to describe the three-dimensional center coordinates of the Gaussian primitive, the covariance matrix is ​​used to describe the anisotropy, scale, and orientation of the Gaussian primitive, the color coefficient is used to describe the appearance information of the Gaussian primitive, and the effective opacity is used to describe the contribution intensity of the Gaussian primitive to the pixel color.

[0035] Specifically, setting the first 3D Gaussian primitives Its parameter set is defined as follows: (2); In the formula: 3D Gaussian primitives The position mean is the center coordinate of the 3D Gaussian primitive.

[0036] 3D Gaussian primitives The covariance matrix; where, , For diagonal matrix functions, This represents the depth uncertainty.

[0037] 3D Gaussian primitives The color coefficients, also known as spherical harmonic function coefficients, are derived from the pixel values ​​in the depth map of the initial frame. The color.

[0038] 3D Gaussian primitives Opacity; among which, Generally, the initial value is set to 0.1, and it is gradually increased based on evidence during subsequent optimization.

[0039] After the mobile robot is initialized, the initial camera pose and the corresponding set of 3D Gaussian primitives are calculated based on the initial frame data. The initial frame is then used as a keyframe and stored in the system's optimization window. This optimization window stores keyframes, each of which includes the camera pose and a set of 3D Gaussian primitives. In this way, the keyframes in the optimization window are used for the subsequent construction of Gaussian maps and environment maps.

[0040] Step S3: During the movement of the mobile robot, acquire the current frame data in real time and optimize the camera pose of the current frame based on the current frame data and the camera pose of the previous key frame.

[0041] The current frame data includes an RGB color image, a depth map, IMU data, and LiDAR data. As the robot moves, the camera pose changes. Therefore, when calculating the camera pose for the current frame, in addition to the acquired multi-mode data, the camera pose from the previous keyframe is also used to estimate and optimize the current frame's camera pose, thus achieving front-end pose tracking.

[0042] More preferably, the present invention uses the current frame data as input and optimizes the camera pose of the current frame through differentiable Gaussian rendering and the construction of a joint loss function.

[0043] Specifically, such as Figure 2 As shown, step S3 further includes: Step S31: Estimate the camera pose of the current frame based on the IMU data of the current frame and the phase pose of the previous key frame.

[0044] Specifically, based on the angular velocity in the IMU data of the current frame. and acceleration Pre-integration is performed to obtain the pre-integrated value between the current frame and the previous keyframe. Then, the estimated value of the camera pose of the current frame can be estimated based on the pre-integrated value and the camera pose of the previous keyframe.

[0045] Among them, the first is set At the current moment, the previous keyframe is captured, and the first keyframe is captured. If the current frame is obtained at a certain time, then: (3), (4), (5); In the formula: , , These are the relative rotation increment, velocity increment, and displacement increment of the previous keyframe, respectively, that is: the... The moment to the The rotational changes, velocity changes, and displacement changes measured by the IUM sensor at each moment.

[0046] , , These are the pre-integral values ​​of the relative rotation increment, velocity increment, and displacement increment for the current frame, respectively; that is: the... The moment to the The rotational, velocity, and displacement changes measured by the IUM sensor at each moment; For gyroscope bias For accelerometer bias, This is gyroscope noise. This is acceleration noise.

[0047] for That time The time interval between each moment.

[0048] Based on the relative rotation increment of the current frame Speed ​​increment Displacement increment The estimated camera pose for the current frame can be obtained by combining the camera pose of the previous keyframe with the current camera pose.

[0049] Step S32: Generate the predicted image of the current frame using differentiable Gaussian rendering based on the estimated camera pose value of the current frame and the 3D Gaussian map.

[0050] A 3D Gaussian map is a three-dimensional spatial representation stored in the form of a set of 3D Gaussian primitives. It is the internal data structure of a map. The 3D Gaussian map can be projected onto a two-dimensional image plane through the differential rendering pipeline to generate a visualized two-dimensional image, which is the predicted image of the current frame. It can be used to verify map quality, support camera pose optimization, and provide visual reference for navigation.

[0051] Set the 3D Gaussian map generated by the current system as The estimated camera pose value for the current frame is Then the predicted image for the current frame is: (6); In the formula: The estimated camera pose for the current frame. The predicted image for the current frame; A differentiable Gaussian rendering operator.

[0052] At the same time, for any pixel in the predicted image of the current frame. Its predicted color is: (7), (8); In the formula: The first frame in the current frame's perspective A 3D Gaussian primitive color; For the first A 3D Gaussian primitive at the pixel level Effective opacity at the location; The number of visible 3D Gaussian primitives from the current perspective; This represents the cumulative transmittance.

[0053] Step S33: Calculate the photometric loss, depth loss, and IMU constraint loss based on the RGB color image and the predicted image of the current frame. Then, construct a joint loss function based on the photometric loss, depth loss, and IMU constraint loss, and use the Adam optimizer to optimize and iterate the joint loss function to optimize the camera pose of the current frame.

[0054] The joint loss function consists of three parts: photometric loss, depth loss, and IMU constraint loss, as detailed below: (9); In the formula: , , These are the weighting coefficients for photometric loss, depth loss, and IMU constraint loss, respectively. For photometric loss, For deep loss, For IMU-constrained loss; This is the joint loss function.

[0055] Specifically, light loss It is derived from L1 loss and SSIM loss, specifically as follows: (10); In the formula: L1 loss is the sum of the absolute differences among all pixels in an image, used to measure the overall brightness error. Its specific definition is as follows: (11); It is a structural similarity index used to measure the degree of similarity between brightness, contrast and structure; The RGB color image of the current frame. The predicted image for the current frame. is the weighting coefficient of the SISM term, with a value range of [0, 1].

[0056] Deep loss It is obtained from the L1 error between the predicted map and the predicted depth map of the current frame, specifically: (12); In the formula: The predicted depth for the current frame; This is the depth map of the current frame, which can be obtained from an RGB-D camera.

[0057] IMU constrained loss , is the IMU pre-integration residual, specifically: (13); In the formula: For IMU pre-integration residuals, Let be the covariance matrix of the current frame.

[0058] This invention uses the Adam optimizer to iteratively optimize the joint loss function in order to update the camera pose of the current frame.

[0059] Step S4: Determine whether the current frame is a keyframe based on the optimized camera pose of the current frame. If yes, proceed to step S5.

[0060] Step S5: Generate a new set of 3D Gaussian primitives based on the depth map of the current frame, and add the current frame as the current keyframe to the optimization window.

[0061] All keyframes are stored within the optimized window, and each keyframe includes the camera pose and a set of 3D Gaussian primitives.

[0062] More preferably, the present invention determines whether the current frame is a keyframe based on preset judgment conditions. These judgment conditions mainly include the following: the current frame is determined to be a keyframe when any one of the following conditions is met: Condition 1: The translation distance between the camera pose of the current frame and the camera pose of the previous keyframe is greater than a first preset threshold. Condition 2: The rotation angle between the camera pose of the current frame and the camera pose of the previous keyframe is greater than the second preset threshold. Condition 3: The average overlap area between the RGB color image of the current frame and the RGB color image of the keyframe of the previous frame is less than the third preset threshold. Condition 4: Does the data acquisition time of the current frame exceed the fourth preset threshold compared to the data acquisition time of the previous key frame? Condition 5: The proportion of new observation points in the current frame is greater than the fifth preset threshold; Condition 6: The image quality score of the current frame is lower than the sixth preset threshold, or the photometric error of the current frame is greater than the seventh threshold.

[0063] The aforementioned preset thresholds can be set based on experience or needs in actual applications.

[0064] More preferably, in step S5, when a new keyframe is added to the optimization window, a loop closure detection process is initiated to detect whether there is a revisit relationship between the current keyframe and each historical keyframe in the optimization window.

[0065] This invention optimizes the global pose graph through a loop closure detection algorithm to eliminate accumulated odometry errors and achieve global consistency. Specifically, it detects whether there is a revisit relationship between the current keyframe and each historical keyframe within the optimization window. That is, it determines whether a revisit exists by comparing the current scene with historical scenes. The current scene refers to the visual information within the camera's field of view of the current frame, and the historical scene refers to the visual information within the camera's field of view of each historical keyframe stored in the system.

[0066] That is, such as Figure 3 As shown, the loop closure detection process specifically includes: Step S51: Extract visual feature data from the RGB color image of the current keyframe and extract visual feature data from the RGB color image of each historical keyframe, and calculate the similarity between the visual feature data of the current frame and the visual feature data of each historical keyframe.

[0067] More specifically, the current keyframe refers to the latest frame that has been determined as a keyframe in step S4. Historical keyframes refer to each keyframe that has been stored in the optimization window.

[0068] Step S52: Obtain a candidate loopback frame set based on the similarity calculation results, and perform geometric verification on each candidate loopback frame in the candidate loopback frame set to determine whether the corresponding candidate loopback frame is valid.

[0069] The candidate loopback frame set includes several candidate loopback frames, and the visual feature data of each candidate loopback frame has a similarity to the visual feature data of the current keyframe that is greater than a preset threshold.

[0070] Specifically, geometric verification is performed using the RANSAC algorithm. By matching feature points and estimating the proportion of interior points using the essential matrix, the geometric consistency of the loop closure is verified, thereby confirming the validity of the candidate loop closure frames.

[0071] Step S53: Based on the judgment result, obtain several loop closure frames from the candidate loop closure frame set, and construct a camera pose map based on the current keyframe and each loop closure frame.

[0072] When a candidate loopback frame is valid, it is used as a loopback frame; when a candidate loopback frame is invalid, it is not used as a loopback frame.

[0073] The camera pose graph is constructed based on the current keyframe and each loopback frame. The nodes of the camera pose graph are keyframes or loopback frames, and the edges of the camera pose graph are camera pose constraints between keyframes or between keyframes and loopback frames.

[0074] Step S54: Optimize each node in the camera pose graph according to the camera pose constraints and the graph optimization library, and update the camera pose of each historical keyframe in the optimization window of the system according to each optimized node.

[0075] This invention solves the above objective function and optimizes the camera pose constraints by using the g2o graph optimization library, and then propagates the optimized camera pose constraints to the Gaussian map to achieve global consistency update of the map.

[0076] The camera pose constraint includes odometry constraint and loop closure constraint. The odometry constraint and loop closure constraint are used to optimize the camera pose of key frames, so as to eliminate the cumulative error caused by odometry and achieve global consistency.

[0077] Step S6: Select multiple keyframes from the optimization window according to preset rules, and optimize the camera pose and 3D Gaussian primitive parameters of the multiple keyframes.

[0078] The present invention also sets up a sliding window, and selects multiple keyframes from the optimization window according to preset rules and puts them into the sliding window, so as to jointly optimize the camera pose and 3D Gaussian primitive parameters of each keyframe in the sliding window.

[0079] Specifically, selecting multiple keyframes from the optimization window involves: selecting the most recent keyframes from the optimization window based on their addition time, and optimizing the camera pose and 3D Gaussian primitives for each keyframe to achieve optimized updates of the keyframes within the optimization window. For example, if the sliding window is set to 5, then the 5 most recent keyframes are selected based on their addition time.

[0080] Preferably, optimization is performed on multiple keyframes. This invention employs a factor graph-like model to achieve this. Simultaneously, a joint optimization objective function is constructed based on the photometric factor, IMU pre-integration factor, and geometric prior factor. The camera pose and 3D Gaussian primitive of each keyframe are used as optimization variables. The Levenberg-Marquardt algorithm is used to solve the nonlinear least squares problem of the above optimization objective function to solve for the optimization variables, thereby optimizing the parameters of the camera pose and 3D Gaussian primitive of the keyframe.

[0081] The joint optimization objective function is: (14); In the formula: For the pose variables of the keyframe, For the parameters of the 3D Gaussian primitive corresponding to the keyframe, For photometric residuals, For IMU pre-integration residuals, The geometric prior residuals are given.

[0082] Among them, the geometric prior factor is used to constrain the shape and scale consistency of the 3D Gaussian primitive, which can be further written as: (15); In the formula: For the first The covariance matrix of a 3D Gaussian primitive. For the first The prior covariance matrix of a 3D Gaussian primitive express Norm.

[0083] The Levenberg-Marquardt algorithm is used to solve the above nonlinear least squares problem. The optimization variables include the camera pose of all keyframes within the sliding window and the parameters of the 3D Gaussian primitive.

[0084] More preferably, to ensure the local detail and scale control of the subsequently constructed map, this invention, when jointly optimizing multiple keyframes, also performs corresponding operations on the 3D Gaussian primitives of each keyframe to achieve dynamic updates of the 3D Gaussian primitives of each keyframe. These corresponding operations include splitting, cloning, and culling.

[0085] Specifically, the gradient information and size of each 3D Gaussian primitive in each keyframe are calculated, and the corresponding 3D Gaussian primitive is updated according to the gradient information, size and preset conditions of each 3D Gaussian primitive.

[0086] Let the first The position gradient magnitude of a 3D Gaussian primitive for: (16); in, To jointly optimize the objective function, For the first The gradient of the positional mean of a 3D Gaussian primitive. It is a 2-norm.

[0087] Let the first The scale of a 3D Gaussian primitive Defined as: (17); in, For the first The covariance matrix of a 3D Gaussian primitive. The largest eigenvalue, This is the scaling factor.

[0088] The update operations for 3D Gaussian primitives specifically include: when and At that time, the first A 3D Gaussian primitive is split into two 3D Gaussian primitives.

[0089] when and At that time, in the A 3D Gaussian primitive is cloned and a new 3D Gaussian primitive is generated.

[0090] when When the 3D Gaussian primitive is not observed for a long period of time, the first... Delete a 3D Gaussian primitive.

[0091] in, For the first Effective opacity of a 3D Gaussian primitive The split offset is the opacity. For gradient threshold, For scale threshold, The effective opacity threshold.

[0092] This invention dynamically updates the camera pose and 3D Gaussian primitives of some keyframes within the optimization window, avoiding processing all keyframes, thereby ensuring the quality of the Gaussian map and providing a reference for subsequent camera pose optimization and navigation.

[0093] Step S7: Construct an environment map for mobile robot navigation based on the 3D Gaussian primitives of all keyframes within the optimization window.

[0094] Specifically, a map database is constructed based on all keyframes within the optimization window, and the 3D Gaussian primitives of each keyframe in the map database are projected onto a horizontal ground grid to generate a two-dimensional occupancy grid map, i.e., an environment map, which provides navigation for the mobile robot.

[0095] Among them, the 3D Gaussian map is a three-dimensional spatial representation based on the set of 3D Gaussian primitives, mainly used for high-fidelity rendering and camera pose optimization; the environment map is a two-dimensional occupancy probability map specifically used for robot navigation path planning, representing the communicability of each grid cell on the ground.

[0096] The map generated by this invention is an explicit representation composed of 3D Gaussian primitives, supporting real-time, photorealistic, and lifelike rendering from any viewpoint, far surpassing maps constructed using traditional SLAM methods of sparse point clouds or geometric meshes. Furthermore, leveraging the inherent high rendering efficiency of 3D Gaussian scattering technology, combined with the sliding window optimization and dynamic management strategies of this invention, it can meet real-time requirements while ensuring positioning accuracy. This invention also unifies mobile robot pose estimation and map representation under a single optimization framework, with the two mutually reinforcing each other. This improves both pose accuracy and map quality, while the high-quality map, in turn, corrects pose tracking accuracy, achieving tightly coupled optimization.

[0097] This invention can be applied not only to navigation and obstacle avoidance, but also directly to scenarios requiring high realism, such as AR / VR virtual fusion and digital twins. At the same time, it provides a powerful perception and decision-making foundation for the autonomous navigation of mobile robots, enabling mobile robots to operate more intelligently and reliably in complex and ever-changing environments.

[0098] Example 2 A map building apparatus based on multimodal perception and Gaussian primitives includes a memory and a processor. The memory stores a map building program that runs on the processor. The map building program is a computer program. When the processor executes the map building program, it implements the steps of the map building method based on multimodal perception and Gaussian primitives provided in Embodiment 1 of the present invention.

[0099] Example 3 A computer-readable storage medium storing a map-building program thereon, the map-building program being a computer program, which, when executed by a processor, implements the steps of the map-building method based on multimodal perception and Gaussian primitives as provided in the embodiments of the present invention.

[0100] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A map construction method based on multimodal perception and Gaussian primitives, characterized in that, The map construction method includes: Initial steps: During the initialization of the mobile robot, initial frame data is acquired based on the RGB-D camera, IMU sensor, and LiDAR installed on the mobile robot. The initial frame data is preprocessed, and the camera pose and 3D Gaussian primitives of the mobile robot are initialized based on the preprocessed initial frame data. The initial frame is then added as a keyframe to the system's optimization window. The initial frame data includes RGB color images, depth maps, IMU data, and LiDAR point cloud data. Visual optimization steps: During the movement of the mobile robot, the current frame data is acquired in real time, and the camera pose of the current frame is optimized based on the current frame data and the camera pose of the previous key frame; the current frame data includes RGB color image, depth map, IMU data and LiDAR point cloud data; Judgment steps: Determine whether the current frame is a keyframe based on the optimized camera pose. If so, generate a new set of 3D Gaussian primitives based on the depth map of the current frame, use the current frame as the current keyframe, and add it to the optimization window. Optimization steps: Select multiple keyframes from the optimization window according to preset rules and optimize the camera pose and 3D Gaussian primitive parameters of multiple keyframes, and then update each keyframe in the optimization window. Map building steps: Construct an environment map for the mobile robot's navigation based on the 3D Gaussian primitives of all keyframes within the optimization window.

2. The map construction method based on multimodal perception and Gaussian primitives according to claim 1, characterized in that, The preprocessing of RGB color images, depth maps, IMU data, and LiDAR point cloud data specifically includes: deblurring of RGB color images based on IMU data, distortion correction of LiDAR point clouds based on IMU data, and completion processing of the depth map based on RGB color images or LiDAR point cloud data. The specific steps for generating the camera pose of the mobile robot include: when the RGB-D camera is a monocular camera, setting an initial time period and continuously acquiring multiple frames of RGB color images within the initial time period, and calculating the relative camera pose of the mobile robot based on the epipolar geometry principle, and then converting the relative camera pose of the mobile robot into the camera pose of the mobile robot using the triangulation principle; when the RGB-D camera is a binocular camera, acquiring the RGB color image of the current frame through the RGB-D camera and obtaining depth information based on the RGB color image, and then obtaining the camera pose of the mobile robot based on the depth information. The initial size, gravity direction, and velocity of the mobile robot are estimated by pre-integrating the IMU data, and then the camera pose of the mobile robot is estimated based on the initial size, gravity direction, and velocity of the robot. Generating a set of 3D Gaussian primitives specifically includes: back-projecting each point of the depth map using the camera memory matrix to obtain the three-dimensional coordinates of each point, and then initializing a three-dimensional Gaussian primitive based on the three-dimensional coordinates of each point; the parameters of the three-dimensional Gaussian primitive include the position mean, covariance matrix, color coefficient, effective opacity, and semantic feature vector.

3. The map construction method based on multimodal perception and Gaussian primitives according to claim 1, characterized in that, The determination of whether the current frame is a keyframe specifically includes: the current frame is a keyframe when any of the following conditions are met; The judgment conditions include: Condition 1, the translation distance between the camera pose of the current frame and the previous keyframe is greater than a first preset threshold; Condition 2, the rotation angle between the camera pose of the current frame and the previous keyframe is greater than a second preset threshold; Condition 3, the average overlap area between the RGB color image of the current frame and the RGB color image of the previous keyframe is less than a third preset threshold; Condition 4, the data acquisition time of the current frame and the data acquisition time of the previous keyframe are greater than a fourth preset interval; Condition 5, the proportion of new observation points in the current frame is greater than a fifth preset threshold; Condition 6, the image quality of the current frame is less than a sixth preset threshold; Condition 7, the photometric error of the current frame is less than a seventh preset threshold.

4. The map construction method based on multimodal perception and Gaussian primitives according to claim 1, characterized in that, The map building steps include: first, building a map database based on all keyframes within the optimization window, and then projecting the 3D Gaussian primitives of each keyframe in the map database onto a horizontal ground grid to generate an environment map.

5. The map construction method based on multimodal perception and Gaussian primitives according to claim 1, characterized in that, When the camera pose and 3D Gaussian primitive of the current keyframe are added to the optimization window in the judgment step, the loop closure detection process is entered. The loop closure test process specifically includes: Step 1: Extract visual feature data from the RGB color image of the current keyframe and from the RGB color image of each historical keyframe, and calculate the similarity between the visual feature data of the current frame and the visual feature data of each historical keyframe. Step 2: Based on the similarity calculation results, a candidate loop closure frame set is obtained, and each loop closure candidate frame in the candidate loop closure frame set is geometrically verified to determine whether the corresponding loop closure candidate frame is valid; the candidate loop closure frame set includes several candidate loop closure frames, and the visual feature data of each candidate loop closure frame has a similarity to the visual feature data of the current keyframe that is greater than a preset threshold. Step 3: Based on the judgment results, obtain several loop closure frames from the candidate loop closure frame set, and construct a camera pose graph based on the camera pose of the current keyframe and the camera pose of each loop closure frame. Step 4: Optimize each keyframe in the camera pose graph according to the camera pose constraints and the graph optimization library to solve for the camera pose constraints, and optimize the camera pose of each historical keyframe in the optimization window according to the camera pose constraints.

6. The map construction method based on multimodal perception and Gaussian primitives according to claim 1, characterized in that, The visual optimization steps specifically include: estimating the camera pose of the current frame based on the IMU data of the current frame; The predicted image of the current frame is generated by differentiable Gaussian rendering based on the estimated camera pose of the current frame and the 3D Gaussian map. Photometric loss, depth loss, and IMU constraint loss are calculated based on the RGB color image and the predicted image of the current frame. Then, a joint loss function is constructed based on the photometric loss, depth loss, and IMU constraint loss. The Adam optimizer is used to iterate and optimize the joint loss function to optimize the camera pose of the current frame.

7. The map construction method based on multimodal perception and Gaussian primitives according to claim 1, characterized in that, The judgment step is followed by: selecting multiple key frames from the optimization window according to preset rules, optimizing the camera pose and 3D Gaussian primitive parameters of the multiple key frames, and updating the optimized camera pose and 3D Gaussian primitive parameters of the multiple key frames to the optimization window. The optimization of camera pose and 3D Gaussian primitive parameters for multiple keyframes specifically includes: constructing a joint optimization objective function based on photometric factors, IMU pre-integration factors, and geometric prior factors; using the camera pose and 3D Gaussian primitive parameters for multiple keyframes as optimization variables; solving a nonlinear least squares problem on the joint optimization objective function to solve for the optimization variables; and thus optimizing the camera pose and 3D Gaussian primitive parameters for the keyframes.

8. The map construction method based on multimodal perception and Gaussian primitives according to claim 7, characterized in that, Optimizing the camera pose and parameters of the 3D Gaussian primitives for multiple keyframes further includes: performing corresponding operations on the corresponding 3D Gaussian primitives based on the gradient information and scale of each 3D Gaussian primitive; the operations include splitting, cloning, or culling. When the gradient metric of a 3D Gaussian primitive is greater than the eighth preset threshold and the scale metric is greater than the ninth preset value, the 3D Gaussian primitive is split into two. When the gradient metric of a 3D Gaussian primitive is greater than the eighth preset threshold and the scale metric is less than or equal to the ninth preset threshold, a new 3D Gaussian primitive is generated by cloning the 3D Gaussian primitive. The 3D Gaussian primitive is deleted when its effective transparency is greater than the tenth preset threshold or when it is not observed.

9. A map building apparatus based on multimodal perception and Gaussian primitives, comprising a memory and a processor, wherein the memory stores a map building program that runs on the processor, characterized in that, The map building program is a computer program, and when the processor executes the map building program, it implements the steps of the map building method based on multimodal perception and Gaussian primitives as described in any one of claims 1-8.

10. A computer-readable storage medium storing a map-building program thereon, characterized in that, The map building program is a computer program, and when executed by a processor, it implements the steps of the map building method based on multimodal perception and Gaussian primitives as described in any one of claims 1-8.

Images