Map construction method and device based on visual-inertial fusion 3D Gaussian SLAM

Abstract

The present invention relates to the field of SLAM, and specifically discloses a map construction method and device based on visual-inertial fusion three-dimensional Gaussian SLAM. The method comprises generating three-dimensional coordinate points using an SFM, constructing and initializing a 3D Gaussian model, projecting the three-dimensional coordinate points onto an image plane using camera extrinsics based on the 3D Gaussian model and performing Gaussian rendering to obtain a rendered image, acquiring IMU data, calculating tracking loss, geometric loss, photometric loss, and depth loss, summing these to calculate a loss function, using the loss function to track objects in the rendered image in real time to obtain a real-time loss result, updating the parameters of the 3D Gaussian model through backpropagation based on the real-time loss result, performing downward adaptive expansion mapping, updating the three-dimensional coordinate points, obtaining a real-time tracking image, calculating relative motion increments based on the IMU data, and constructing a three-dimensional map based on the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map. The method aims to improve SLAM accuracy.

Description

Technical Field

[0001] The present invention relates to the field of SLAM, and in particular to a map construction method and device based on visual-inertial fusion three-dimensional Gaussian SLAM. Background Art

[0002] SLAM technology, short for Simultaneous Localization and Mapping, stands for simultaneous localization and mapping. It enables robots to create maps in completely unknown environments and use them for autonomous localization and navigation. The core tasks of SLAM technology include locating the device's current position and building a spatial map of the space. SLAM uses sensors such as cameras, lidar, and inertial measurement units to collect environmental information and integrate this information using algorithms to determine the device's position in the unknown environment and build a map of the environment. This process involves feature extraction, 3D reconstruction, and map model correction. This allows the robot to continuously acquire new landmarks and correct its mental map model as it moves, achieving localization and mapping. SLAM technology is widely used in robot positioning and navigation, autonomous driving, autonomous obstacle avoidance and path planning for drones, and tracking and positioning in VR / AR. It helps devices accurately establish spatial and temporal relationships in unfamiliar environments, answering questions like "Where am I?", "Where is this place?", and "How do I get there?"—thus enabling localization, mapping, and path planning. SLAM systems require real-time performance and accuracy, involving many system integration issues such as high-speed and high-precision perception and posture tracking in hardware, multi-threaded concurrent execution in algorithms, resource allocation, read-write coordination, map data management, optimization and accuracy.

[0003] In existing technologies, SLAM (Simultaneous Localization and Mapping) is typically used solely for map construction. Because Gaussian SLAM relies solely on vision as a single source of information, the system's estimated camera pose can be significantly biased during actual operation. In 3D Gaussian SLAM algorithms, there's no separate loss calculation for depth. This disregard for depth loss results in each pixel contributing the same amount to tracking and mapping. This leads to varying biases and uncertainties depending on the sensor. When these losses accumulate to a certain level, they can cause uncertainty in the overall algorithm. Summary of the Invention

[0004] In view of the above problems, an object of the present invention is to provide a map construction method based on visual-inertial fusion 3D Gaussian SLAM.

[0005] The second object of the present invention is to provide a map construction device based on visual-inertial fusion three-dimensional Gaussian SLAM.

[0006] The first technical solution adopted by the present invention is: S1, using SFM to generate three-dimensional coordinate points, constructing and initializing a 3D Gaussian model;

[0007] S2. Projecting the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and performing Gaussian rendering to obtain a rendered image;

[0008] S3. Obtain IMU data and calculate tracking loss, geometric loss, photometric loss, and depth loss;

[0009] S4, calculating a loss function by summing the tracking loss, the geometric loss, the photometric loss, and the depth loss, and tracking the object in the rendered image in real time using the loss function to obtain a real-time loss result;

[0010] S5. Based on the real-time loss result, update the parameters of the 3D Gaussian model through back propagation, perform downward adaptive expansion mapping, update the three-dimensional coordinate points, and obtain a real-time tracking image;

[0011] S6. Calculate the relative motion increment based on the IMU data, and construct a three-dimensional map according to the real-time tracking image and the relative motion increment to obtain a high-precision three-dimensional map.

[0012] Optionally, the S2 includes:

[0013] S21. According to the 3D Gaussian model, project the three-dimensional coordinate points onto the image plane using a Jacobian matrix to obtain a Gaussian distribution;

[0014] S22, performing depth sorting on the Gaussian distribution, and rendering each pixel color based on a front-back mixed rendering strategy, as shown in the following formula:

[0015]

[0016] Where C represents the color of each pixel after rendering, C i is the initial color of each pixel, σ i is the density;

[0017] The depth rendering corresponding to each pixel is as follows:

[0018]

[0019] Where D is the depth after rendering, d i is the center depth of each pixel.

[0020] Optionally, the S3 includes:

[0021] S31, using the pre-integration to process the IMU data to obtain speed and posture data during the movement;

[0022] S32. Calculate the tracking loss, the geometric loss, the photometric loss, and the depth loss according to the speed and posture data during the movement, wherein the tracking loss includes a translation component loss and a rotation component loss.

[0023] Optionally, the S6 includes:

[0024] The S6 includes:

[0025] S61, presetting several IMU measurement values ​​as the relative motion increment, and linking them with two adjacent key frames generated by Gaussian SLAM;

[0026] S62, performing calculation based on the relative motion increment to obtain an updated relative motion increment;

[0027] S63: Fusing the tracking loss, the geometric loss, the photometric loss, the depth loss, the updated relative motion increment, and the key frame of the real-time tracking image, constructing a three-dimensional map through a 3D Gaussian model to obtain a high-precision three-dimensional map.

[0028] Optionally, the updated relative motion increment is calculated by the following formula:

[0029]

[0030] In the formula, k represents the time, x i-1 is the camera pose corresponding to time i-1, x i is the camera pose corresponding to time i, y k-1 is the noise corresponding to the k-1 moment.

[0031] Optionally, calculating the depth loss in S3 further includes:

[0032] The depth loss of the tracking part is defined as:

[0033]

[0034] in d n The standard deviation, d n is the measured depth of pixel n, u k-1 To weight the uncertainty of the tracking part, N represents N pixels;

[0035] The depth loss of the mapping part is defined as:

[0036]

[0037] where v k-1 Weighting the uncertainty of the mapping part.

[0038] The second technical solution adopted by the present invention is: a map construction device based on visual-inertial fusion 3D Gaussian SLAM, comprising: a model construction module for generating 3D coordinate points using SFM, and constructing and initializing a 3D Gaussian model;

[0039] A rendering module, configured to project the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and perform Gaussian rendering to obtain a rendered image;

[0040] The loss calculation module is used to obtain IMU data and calculate tracking loss, geometric loss, photometric loss, and depth loss;

[0041] a loss comparison module, configured to calculate a loss function based on the sum of the tracking loss, the geometric loss, the photometric loss, and the depth loss, and to track the object in the rendered image in real time using the loss function to obtain a real-time loss result;

[0042] An updating module is used to update the parameters of the 3D Gaussian model through back propagation according to the real-time loss result, perform downward adaptive expansion mapping, update the three-dimensional coordinate points, and obtain a real-time tracking image;

[0043] A mapping module is used to calculate relative motion increments based on the IMU data, and to construct a three-dimensional map according to the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map.

[0044] Optionally, the loss calculation module includes:

[0045] A pre-integration processing module is used to process the IMU data using the pre-integration to obtain speed and posture data during the movement;

[0046] A loss calculation module is used to calculate the tracking loss, the geometric loss, the photometric loss and the depth loss according to the speed and posture data during the movement, wherein the tracking loss includes a translation component loss and a rotation component loss.

[0047] Optionally, the mapping module includes:

[0048] A linking module is used to preset several IMU measurement values ​​as the relative motion increment and link them with two adjacent key frames generated by Gaussian SLAM;

[0049] An RMI calculation module, configured to perform calculation based on the relative motion increment to obtain an updated relative motion increment;

[0050] A fusion module is used to fuse the tracking loss, the geometric loss, the photometric loss, the depth loss, the updated relative motion increment and the key frame of the real-time tracking image, and construct a three-dimensional map through a 3D Gaussian model to obtain a high-precision three-dimensional map.

[0051] Optionally, the updated relative motion increment is calculated by the following formula:

[0052]

[0053] In the formula, k represents the time, x i-1 is the camera pose corresponding to time i-1, x i is the camera pose corresponding to time i, y k-1 is the noise corresponding to the k-1 moment.

[0054] Beneficial effects of the above technical solution:

[0055] The present invention generates three-dimensional coordinate points using SFM, constructs and initializes a 3D Gaussian model, projects the three-dimensional coordinate points onto an image plane using camera extrinsics based on the 3D Gaussian model, and performs Gaussian rendering to obtain a rendered image. IMU data is then acquired, and tracking loss, geometric loss, photometric loss, and depth loss are calculated. A loss function is calculated by summing the tracking loss, geometric loss, photometric loss, and depth loss. The object in the rendered image is tracked in real time using the loss function to obtain a real-time loss result. Based on the real-time loss result, the parameters of the 3D Gaussian model are updated through backpropagation, and adaptive extended mapping is performed downward to update the three-dimensional coordinate points to obtain a real-time tracking image. Relative motion increments are calculated based on the IMU data, and a three-dimensional map is constructed based on the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map. The IMU data is then integrated with Gaussian SLAM to improve SLAM accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of the map construction method based on visual-inertial fusion 3D Gaussian SLAM provided by the present invention;

[0057] Figure 2 A schematic diagram of the structure of a map construction device based on visual-inertial fusion 3D Gaussian SLAM provided by the present invention;

[0058] Figure 3(a) shows the running effect in the living room scene under the apartment sequence Figure 1 ;

[0059] Figure 3(b) shows the running effect in the living room scene under the apartment sequence Figure 2 ;

[0060] Figure 4(a) shows the running effect in the bedroom scene in the apartment sequence Figure 1 ;

[0061] Figure 4(b) shows the running effect in the bedroom scene in the apartment sequence Figure 2 ;

[0062] Figure 5(a) shows the overall operation effect under the apartment sequence Figure 1 ;

[0063] Figure 5(b) shows the overall running effect under the apartment sequence Figure 2 ;

[0064] Figure 6(a) shows the results of running under the ScanNet sequence Figure 1 ;

[0065] Figure 6(b) shows the results of running under the ScanNet sequence Figure 2 ;

[0066] Figure 7 This is a schematic diagram of the overall algorithm flow. DETAILED DESCRIPTION

[0067] The following detailed description of the embodiments of the present invention is provided in conjunction with the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are intended to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention. That is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0068] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more; the terms “first”, “second”, etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0069] Example 1

[0070] An embodiment of the present invention provides a map construction method based on visual inertial fusion 3D Gaussian SLAM, such as Figure 1 and Figure 7As shown, S1, use SFM to generate three-dimensional coordinate points, build and initialize a 3D Gaussian model; S2, according to the 3D Gaussian model, project the three-dimensional coordinate points to the image plane through the camera extrinsic parameters and perform Gaussian rendering to obtain a rendered image; S3, obtain IMU data, and calculate tracking loss, geometric loss, photometric loss and depth loss; S4, calculate a loss function based on the sum of the tracking loss, the geometric loss, the photometric loss and the depth loss, and use the loss function to track the object in the rendered image in real time to obtain a real-time loss result; S5, according to the real-time loss result, update the parameters of the 3D Gaussian model through back propagation, perform adaptive expansion mapping downward, update the three-dimensional coordinate points, and obtain a real-time tracking image; S6, calculate the relative motion increment based on the IMU data, and construct a three-dimensional map based on the real-time tracking image and the relative motion increment to obtain a high-precision three-dimensional map.

[0071] Motion information can be used as an important source of information for state estimation. Since Gaussian SLAM only uses vision as the only information source, the system has a large deviation in estimating the camera pose during actual operation. Therefore, the present invention fuses an inertial measurement unit (IMU) with Gaussian SLAM to improve its accuracy.

[0072] S1. Use SFM to generate three-dimensional coordinate points, build and initialize a 3D Gaussian model.

[0073] Rendering the environment through three-dimensional Gaussian

[0074] First, the 3D scene is modeled as a 3D Gaussian function including opacity and spherical harmonics:

[0075] G={G i :(X i ,Σ i ,Λ i ,Z i )|i=1,…,N}

[0076] Among them G i Represents each 3D Gaussian scene, which contains the 3D point position X i , the three-dimensional covariance matrix Σ i ∈R 3 ×3 , opacity Λ i ∈R and the one-dimensional spherical harmonics Z of each channel i .

[0077] It should be noted that the SFM algorithm (Structure from Motion) is a technology that reconstructs 3D scenes based on image sequences taken from different viewpoints. It infers the 3D structure of the scene and the camera's motion by analyzing images captured by a camera at different viewpoints, thereby creating a 3D model, measuring the camera trajectory, and estimating depth information.

[0078] S2. Projecting the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and performing Gaussian rendering to obtain a rendered image;

[0079] S21. According to the 3D Gaussian model, the three-dimensional coordinate points are projected onto an image plane using a Jacobian matrix to obtain a Gaussian distribution.

[0080] Given a camera pose:

[0081] P={R,t}

[0082] Where P is the camera pose, R is the rotation matrix, and t represents the camera translation component.

[0083] Project the three-dimensional image G onto the two-dimensional image plane and render it using the following formula

[0084] Σ`=JP -1 ΣP -T J T

[0085] Where J is the Jacobian matrix of the radial approximation of the projection function. After projecting the three-dimensional Gaussian distribution onto the image plane, the Gaussian distribution is sorted by depth and the pixel color is plotted.

[0086] S22, performing depth sorting on the Gaussian distribution, and rendering each pixel color based on a front-back mixed rendering strategy, as shown in the following formula:

[0087]

[0088] Where C represents the color of each pixel after rendering, C i is the initial color of each pixel, σ i is the initial density, σ i is the i-th pixel density;

[0089] The depth rendering corresponding to each pixel is as follows:

[0090]

[0091] Where D is the depth after rendering, d i is the center depth of each pixel.

[0092] S3. Obtain IMU data and calculate tracking loss, geometric loss, photometric loss, and depth loss;

[0093] Adding IMU information to the tracking target can provide prior information of motion for the current camera pose.

[0094] IMU data is obtained by an IMU (Inertial Measurement Unit), a device used to measure the motion state of an object. It integrates an accelerometer and a gyroscope, and sometimes also includes a magnetometer and a barometer, and is used to provide data about the object's acceleration, angular velocity, and sometimes also direction and altitude information.

[0095] S31. Process the IMU data using the pre-integration to obtain speed and posture data during the movement.

[0096] A method of integrating the acceleration and angular velocity data in the IMU data to obtain attitude and position information.

[0097] S32. Calculate the tracking loss, the geometric loss, the photometric loss, and the depth loss according to the speed and posture data during the movement, wherein the tracking loss includes a translation component loss and a rotation component loss.

[0098] In order to simplify the calculation, the camera pose of the previous frame relative to the current frame is fixed, and the camera pose corresponding to the k moment is defined as x k =(r k ,t k ), where r∈SO(3) represents the camera rotation component, t∈R 3 Represents the camera translation component. Then the camera pose corresponding to time k is x k , the camera pose corresponding to the k-1 moment is x k-1 .

[0099] The IMU loss consists of two parts: rotation loss and translation loss, where the rotation component loss is as follows:

[0100]

[0101] Where, is the rotational component loss, u k-1 is the uncertainty weight, r k is the rotation component at moment k.

[0102] The rotation component loss is weighted by the uncertainty u k-1 The weight is obtained by linearization Relative to the IMU measurement noise y k-1 , and propagates the covariance matrix of the noise by linearizing Got it.

[0103]

[0104] Where, is the covariance matrix of the IMU noise rotation component, y k-1 is noise, is the rotational component loss.

[0105] Among them, the Jacobian matrix of the loss can be expressed as

[0106]

[0107] Among them, J e The Jacobian matrix belonging to SO(3).

[0108] The loss of the translation component is as follows:

[0109]

[0110] The uncertainty weight of the translation component error is v k-1 is the covariance matrix of the IMU noise translation component Subtract the covariance matrix of the IMU loss rotation component at time k-1 Calculated.

[0111] After obtaining the rotation loss and translation loss components, the final tracking loss of the IMU can be obtained by the following formula:

[0112]

[0113] Where, is the total loss function of IMU in the tracking part, is the translation component loss, is the rotation component covariance matrix.

[0114] In the tracking and mapping part, optimization is performed to minimize the geometric loss, photometric loss, IMU loss, and depth loss.

[0115] The geometric loss is the L1 loss between the observed depth and the predicted depth of M sampling points on the visual ray, as follows:

[0116]

[0117] Where D m is the observation depth of the Mth sampling point, is the predicted depth of the Mth sampling point.

[0118] The photometric loss is the L1 loss of the rendered color and the observed color at M sampling points, as follows:

[0119]

[0120] Where C m is the observed color of the Mth sampling point, is the predicted color of the Mth sampling point.

[0121]

[0122] in d n The standard deviation of n is the measured depth of pixel n, u k-1 Weight the uncertainty of the tracking part, where N represents N pixels.

[0123] S4. Calculate a loss function based on the sum of the tracking loss, the geometric loss, the photometric loss, and the depth loss, and use the loss function to track the object in the rendered image in real time to obtain a real-time loss result.

[0124] In the tracking and mapping parts, we add the losses to get the total loss function of the tracking and mapping parts.

[0125] Tracking section:

[0126]

[0127] Mapping part:

[0128]

[0129] According to the loss of the tracking part and the mapping part, the real-time image is tracked to determine the real-time tracking loss result, that is, the loss between the calculation and the actual image.

[0130] It's important to note that in the 3D Gaussian SLAM algorithm, there's no separate loss calculation for depth. This disregard for depth loss results in each pixel contributing the same amount to both tracking and mapping. This leads to varying deviations and uncertainties depending on the sensor. When these losses accumulate to a certain level, they can cause uncertainty in the overall algorithm. Therefore, depth loss calculation is incorporated into both tracking and mapping to address this issue.

[0131] S5. According to the real-time loss result, the parameters of the 3D Gaussian model are updated through back propagation, and adaptive expansion mapping is performed downward to update the three-dimensional coordinate points to obtain a real-time tracking image.

[0132] According to the real-time loss comparison result, the loss between the current calculation result and the actual image can be known. The loss is returned to the 3D Gaussian model to modify the parameters, and the same steps as S2 are performed to obtain the rendered real-time tracking image.

[0133] S6. Calculate the relative motion increment based on the IMU data, and construct a three-dimensional map according to the real-time tracking image and the relative motion increment to obtain a high-precision three-dimensional map.

[0134] S61, presetting several IMU measurement values ​​as the relative motion increment, and linking them with two adjacent key frames generated by Gaussian SLAM;

[0135] Gaussian SLAM maintains a keyframe group. It groups multiple IMU measurements into a relative motion increment (RMI) and links this group of IMU measurements to two adjacent keyframes generated by Gaussian SLAM. This eliminates the need for preprocessing each IMU measurement and enhances the motion information between keyframes, simplifying the motion constraints of Gaussian SLAM.

[0136] S62, performing calculation based on the relative motion increment to obtain an updated relative motion increment;

[0137] Define the camera's position at time i as x i =(r i ,t i ), the relative motion increment (RMI) of IMU is Δx i =(Δr i ,Δt i ) is used to represent the directional component Δr i and displacement component Δt i Update it with the following formula:

[0138] Δr i =Δr i-1 EXP(u k-1 )

[0139] Δt i =Δt i-1 +Δr i (v k-1 )

[0140] where u k-1 and v k-1 are the uncertainty weights for the rotational component loss and the translational component loss in B1, respectively.

[0141] The updated relative motion increment is calculated by the following formula:

[0142]

[0143] In the formula, k represents the time, x i-1 is the camera pose corresponding to time i-1, x i is the camera pose corresponding to time i, y k-1 is the noise corresponding to the k-1 moment.

[0144] Since the relative motion increment (RMI) is defined between two keyframes, if the system calculates the incremental change between the two frames in time, the RMI between the next keyframe needs to be multiplied by the motion increment between the previous two frames. For example, if the RMI between the 1st and 2nd frames is not calculated in time, the RMI between the 2nd and 3rd frames will be multiplied by the motion increment between the previous two frames. This definition can prevent the system from causing sudden changes in subsequent increments when calculating the current frame increment in a timely manner, thereby causing the IMU data to go astray.

[0145] The RMI loss between two frames is defined as follows:

[0146]

[0147] Where, is the loss of RMI of the rotation component between two frames, is the loss of RMI of the translation component between two frames.

[0148] The final total RMI loss is:

[0149]

[0150] Where, is the total RMI loss between two frames, is the inverse matrix of the covariance matrix of the RMI error between two frames.

[0151] S63: Fusing the tracking loss, the geometric loss, the photometric loss, the depth loss, the updated relative motion increment, and the key frame of the real-time tracking image, constructing a three-dimensional map through a 3D Gaussian model to obtain a high-precision three-dimensional map.

[0152] The effect diagrams are shown in Figure 3(a) to Figure 6(b). It should be noted that the black dotted line is GroundTrue (true trajectory), the red dotted line is the actual trajectory of the algorithm in this paper, and the ab figures with the same label are the comparison of the effects before and after rendering.

[0153] This embodiment uses SFM to generate three-dimensional coordinate points, constructs and initializes a three-dimensional Gaussian model, and obtains a 3D Gaussian model. Based on the 3D Gaussian model, the three-dimensional coordinate points are projected onto the image plane using camera extrinsics and Gaussian rendering is performed to obtain a rendered image. IMU data is acquired, pre-integrated, and then loss is calculated to obtain an error loss. The error loss is added to the loss function of the tracking component. Objects in the rendered image are tracked in real time and compared with the loss function to obtain a real-time loss comparison result. Based on the real-time loss comparison result, the parameters of the 3D Gaussian model are updated through backpropagation, and adaptive downward expansion mapping is performed to update the three-dimensional coordinate points to obtain a real-time tracking image. Relative motion increments are calculated based on the IMU data, and a three-dimensional map is constructed based on the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map. By fusing the IMU data with Gaussian SLAM, the accuracy of SLAM is improved.

[0154] Example 2

[0155] An embodiment of the present invention provides a map construction device 200 based on visual-inertial fusion 3D Gaussian SLAM, such as Figure 2 Shown, including:

[0156] The model construction module 201 is used to generate three-dimensional coordinate points using SFM, and to construct and initialize a 3D Gaussian model;

[0157] A rendering module 202 is configured to project the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and perform Gaussian rendering to obtain a rendered image;

[0158] The loss calculation module 203 is used to obtain IMU data and calculate tracking loss, geometric loss, photometric loss and depth loss;

[0159] a loss comparison module 204 for calculating a loss function by summing the tracking loss, the geometric loss, the photometric loss, and the depth loss, and tracking the object in the rendered image in real time using the loss function to obtain a real-time loss result;

[0160] An updating module 205 is configured to update the parameters of the 3D Gaussian model according to the real-time loss result through back propagation, perform downward adaptive expansion mapping, update the three-dimensional coordinate points, and obtain a real-time tracking image;

[0161] The mapping module 206 is configured to calculate relative motion increments based on the IMU data, and construct a three-dimensional map based on the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map.

[0162] Furthermore, the loss calculation module 203 includes:

[0163] A pre-integration processing module 2031 is used to process the IMU data using the pre-integration to obtain speed and posture data during the movement process;

[0164] The loss calculation module 2032 calculates the tracking loss, the geometric loss, the photometric loss and the depth loss according to the speed and posture data during the movement, wherein the tracking loss includes a translation component loss and a rotation component loss.

[0165] Furthermore, the mapping module 206 includes:

[0166] A linking module 2061 is configured to preset a number of IMU measurement values ​​as the relative motion increment and link them with two adjacent key frames generated by Gaussian SLAM;

[0167] The RMI calculation module 2062 is used to perform calculations based on the relative motion increment to obtain an updated relative motion increment; the fusion module 2063 is used to fuse the tracking loss, the geometric loss, the photometric loss, the depth loss, the updated relative motion increment and the key frame of the real-time tracking image, and construct a three-dimensional map through a 3D Gaussian model to obtain a high-precision three-dimensional map.

[0168] This embodiment is a map construction device based on visual-inertial fusion three-dimensional Gaussian SLAM, including a model construction module, which is used to generate three-dimensional coordinate points using SFM, construct and initialize a 3D Gaussian model; a rendering module, which is used to project the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and perform Gaussian rendering to obtain a rendered image; a loss calculation module, which is used to obtain IMU data and calculate tracking loss, geometric loss, photometric loss and depth loss; a loss comparison module, which is used to calculate a loss function by summing the tracking loss, the geometric loss, the photometric loss and the depth loss, and use the loss function to track objects in the rendered image in real time to obtain a real-time loss result; an update module, which is used to update the parameters of the 3D Gaussian model through back propagation according to the real-time loss result, perform downward adaptive expansion mapping, update the three-dimensional coordinate points, and obtain a real-time tracking image; and a mapping module, which is used to calculate relative motion increments based on the IMU data, and construct a three-dimensional map based on the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map. By fusing IMU data with Gaussian SLAM, the accuracy of SLAM can be improved.

[0169] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A map construction method based on visual-inertial fusion 3D Gaussian SLAM, characterized in that: include: S1. Generate three-dimensional coordinate points using the SFM algorithm, and construct and initialize a 3D Gaussian model. S2. Projecting the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and performing Gaussian rendering to obtain a rendered image; S3. Obtain inertial measurement unit (IMU) data and calculate tracking loss, geometric loss, photometric loss, and depth loss. S4. Add the losses to obtain the total loss function of the tracking and mapping parts; track the real-time image according to the losses of the tracking part and the mapping part, calculate the loss with the actual image, and determine the real-time tracking loss result; wherein, Tracking section: ； Mapping part: ； Where, is the geometric loss; is the loss of luminosity; is the final tracking loss of IMU; represents the total RMI loss; wherein the final tracking loss of the IMU is expressed by the following formula: ； Where, is the rotational component loss; is the translation component loss; is the rotation component covariance matrix; is the covariance matrix of the IMU noise translation component; The total RMI loss is expressed by the following formula: ； Where, is the total RMI loss between two frames, which is obtained based on the RMI loss of the rotation component between the two frames and the RMI loss of the translation component between the two frames; is the inverse matrix of the covariance matrix of the RMI error between two frames; The depth loss of the tracking part is defined as: ； in, for The standard deviation of Pixels n The measurement depth, Weighting the uncertainty of the tracking part, N Representatives include N pixels; The depth loss of the mapping part is defined as: ； in Weighting the uncertainty of the mapping part; S5. Based on the real-time loss result, update the parameters of the 3D Gaussian model through back propagation, perform downward adaptive expansion mapping, update the three-dimensional coordinate points, and obtain a real-time tracking image; S6. Calculating a relative motion increment based on the IMU data, and constructing a three-dimensional map according to the real-time tracking image and the relative motion increment to obtain a high-precision three-dimensional map; The S6 includes: S61, presetting several IMU measurement values ​​as the relative motion increment, and linking them with two adjacent key frames generated by Gaussian SLAM; S62, performing calculation based on the relative motion increment to obtain an updated relative motion increment; S63: Fusing the tracking loss, the geometric loss, the photometric loss, the depth loss, the updated relative motion increment, and the key frame of the real-time tracking image, constructing a three-dimensional map through a 3D Gaussian model to obtain a high-precision three-dimensional map.

2. The map construction method based on visual-inertial fusion 3D Gaussian SLAM according to claim 1 is characterized in that: The S2 includes: S21. According to the 3D Gaussian model, project the three-dimensional coordinate points onto the image plane using a Jacobian matrix to obtain a Gaussian distribution; S22. Perform depth sorting on the Gaussian distribution, and render each pixel color based on a front-back mixed rendering strategy, as shown in the following formula: ； Where, C Represents the color of each pixel after rendering, is the initial color of each pixel, is the density; i is a pixel; N Represents N pixels; The depth rendering corresponding to each pixel is as follows: ； in D is the depth after rendering, is the center depth of each pixel.

3. The map construction method based on visual-inertial fusion 3D Gaussian SLAM according to claim 1 is characterized in that: The S3 includes: S31, processing the IMU data by using pre-integration to obtain speed and posture data during the movement; S32. Calculate the tracking loss, geometric loss, photometric loss, and depth loss based on the speed and posture data during the movement, wherein the tracking loss includes a translation component loss and a rotation component loss.

4. A map construction device based on visual-inertial fusion 3D Gaussian SLAM, characterized in that: include: The model building module is used to generate three-dimensional coordinate points using the SFM algorithm, and to build and initialize a 3D Gaussian model; A rendering module, configured to project the three-dimensional coordinate points onto an image plane using camera extrinsics according to the 3D Gaussian model and perform Gaussian rendering to obtain a rendered image; The loss calculation module is used to obtain the inertial measurement unit (IMU) data and calculate the tracking loss, geometric loss, photometric loss, and depth loss. The loss comparison module is used to sum the losses to obtain the total loss function of the tracking and mapping parts; according to the losses of the tracking part and the mapping part, the real-time image is tracked, the loss with the actual image is calculated, and the real-time tracking loss result is determined; wherein, Tracking section: ； Mapping part: ； Where, is the geometric loss; is the loss of luminosity; is the final tracking loss of IMU; represents the total RMI loss; wherein the final tracking loss of the IMU is expressed by the following formula: ； Where, is the rotational component loss; is the translation component loss; is the rotation component covariance matrix; is the covariance matrix of the IMU noise translation component; The total RMI loss is expressed by the following formula: ； Where, is the total RMI loss between two frames, which is obtained based on the RMI loss of the rotation component between the two frames and the RMI loss of the translation component between the two frames; is the inverse matrix of the covariance matrix of the RMI error between two frames; The depth loss of the tracking part is defined as: ； in, for The standard deviation of Pixels n The measurement depth, Weighting the uncertainty of the tracking part, N Representatives include N pixels; The depth loss of the mapping part is defined as: ； in Weighting the uncertainty of the mapping part; An updating module is used to update the parameters of the 3D Gaussian model through back propagation according to the real-time loss result, perform downward adaptive expansion mapping, update the three-dimensional coordinate points, and obtain a real-time tracking image; A mapping module, configured to calculate relative motion increments based on the IMU data, and construct a three-dimensional map based on the real-time tracking image and the relative motion increments to obtain a high-precision three-dimensional map; The mapping module includes: A linking module is used to preset several IMU measurement values ​​as the relative motion increment and link them with two adjacent key frames generated by Gaussian SLAM; An RMI calculation module, configured to perform calculation based on the relative motion increment to obtain an updated relative motion increment; A fusion module is used to fuse the tracking loss, geometric loss, photometric loss, depth loss, updated relative motion increment and the key frame of the real-time tracking image, and construct a three-dimensional map through a 3D Gaussian model to obtain a high-precision three-dimensional map.

5. The map construction device based on visual-inertial fusion 3D Gaussian SLAM according to claim 4, characterized in that: The loss calculation module includes: A module for processing the IMU data using pre-integration to obtain speed and posture data during movement; A module for calculating the tracking loss, geometric loss, photometric loss and depth loss based on the speed and posture data during the movement, wherein the tracking loss includes a translation component loss and a rotation component loss.

Images