A camera posture solving method, system and terminal based on 3DGS

Abstract

The present invention discloses a camera pose solution method, system and terminal based on 3DGS, the method comprising: obtaining a first preset number of RGBD images, generating point cloud information of the real pose and the current scene according to the first preset number of RGBD images, generating an initial 3D Gaussian model according to the point cloud information, iteratively optimizing the initial 3D Gaussian model according to the real pose, and obtaining a target 3D Gaussian model; obtaining a target RGB image, inputting the target RGB image into a deep learning network, and outputting the initial pose of the camera; generating an opacity map, a depth map and a color map according to the target 3D Gaussian model, iteratively updating the initial pose according to the opacity map, the depth map and the color map, and obtaining a target camera pose estimate. The present invention can optimize the initial pose output by the deep learning network through a 3D Gaussian model, and finally achieve a higher-precision camera pose estimate.

Description

Technical Field

[0001] The present invention relates to the technical field of posture solving, and in particular to a camera posture solving method, system and terminal based on 3DGS. Background Art

[0002] In technical equipment such as intelligent mobile robots, autonomous driving and augmented reality, the corresponding intelligent devices have the ability to sense their own positions in complex environments, that is, the ability to perform self-positioning, which is the basis of many applications. Visual positioning refers to the process of using visual sensors to obtain environmental information and determine one's own position. Currently, the positioning technology based on GNSS (Global Navigation Satellite System) can achieve sub-meter positioning in open outdoor scenes, but it is often difficult to obtain signals that can be used for positioning indoors due to the obstruction of obstacles such as buildings. In contrast, visual sensors are often regarded as the preferred sensors for self-positioning of various intelligent devices due to their simple structure and low cost.

[0003] However, visual sensors cannot directly obtain depth information and are easily affected by environmental factors such as lighting. At the same time, visual odometers have cumulative errors, resulting in inaccurate positioning using visual sensors.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a camera pose solution method, system, terminal and computer-readable storage medium based on 3DGS, aiming to solve the problem that the visual sensor in the prior art cannot directly obtain depth information, is easily affected by environmental factors such as lighting, and has cumulative errors in the visual odometer, resulting in inaccurate positioning using the visual sensor.

[0006] To achieve the above object, the present invention provides a camera pose solving method based on 3DGS, and the camera pose solving method based on 3DGS comprises the following steps:

[0007] Acquire a first preset number of RGBD images, generate point cloud information of a real pose and a current scene according to the first preset number of RGBD images, generate an initial 3D Gaussian model according to the point cloud information, and iteratively optimize the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model;

[0008] Obtain a target RGB image, input the target RGB image into a deep learning network, and output the initial pose of the camera;

[0009] An opacity map, a depth map and a color map are generated according to the target 3D Gaussian model, and an initial pose is iteratively updated according to the opacity map, the depth map and the color map to obtain a target camera pose estimate.

[0010] Optionally, acquiring a first preset number of RGBD images, generating point cloud information of the real pose and the current scene according to the first preset number of RGBD images, and generating an initial 3D Gaussian model according to the point cloud information specifically includes:

[0011] Acquire a first preset number of RGBD images, calculate the first preset number of RGBD images according to a visual 3D reconstruction framework to obtain a true pose, and generate point cloud information of the current scene according to the true pose and depth information of the RGBD images;

[0012] The mean and color of the current scene are obtained according to the point cloud information, the preset covariance matrix and opacity are obtained, and an initial 3D Gaussian model is generated according to the mean, the color, the covariance matrix and the opacity.

[0013] Optionally, the iterative optimization of the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model specifically includes:

[0014] Constructing a training opacity map, a training depth map, and a training color map according to the initial 3D Gaussian model, calculating a Gaussian loss according to the training opacity map, the training depth map, the training color map, and the true pose, and optimizing the initial 3D Gaussian model according to the Gaussian loss;

[0015] The initial 3D Gaussian model is optimized by multiple iterations. When the number of iterations reaches a second preset number, the iteration is terminated, and the initial 3D Gaussian model obtained at the last time is used as the target 3D Gaussian model.

[0016] Optionally, generating an opacity map, a depth map and a color map according to the target 3D Gaussian model specifically includes:

[0017] Calculate an alpha value according to the opacity of the target 3D Gaussian model;

[0018] According to the alpha value and the target 3D Gaussian model, an alpha blending algorithm is used to respectively calculate the opacity map, the depth map and the color map.

[0019] Optionally, iteratively updating the initial pose according to the opacity map, the depth map and the color map to obtain the target camera pose estimate specifically includes:

[0020] Calculate a current loss value according to the initial pose, the opacity map, the depth map, and the color map, and optimize the initial pose according to the current loss value;

[0021] The initial pose is iterated multiple times. When the number of iterations reaches a third preset number, the iteration is terminated, and the initial pose obtained in the last iteration is used as the target camera pose estimate.

[0022] Optionally, the calculating the current loss value according to the initial pose, the opacity map, the depth map and the color map specifically includes:

[0023] Taking pixels in the opacity map that exceed a preset opacity threshold as target pixels;

[0024] Sampling the target pixel in the depth map and the color map to obtain a target depth map and a target color map, and generating a query color map and a query depth map according to the initial pose;

[0025] Calculating the consistency loss between the query depth map and the depth map according to the target depth map to obtain a depth consistency loss, and calculating the consistency loss between the query color map and the color map according to the target color map to obtain a color consistency loss;

[0026] A current loss value is generated according to the depth consistency loss and the color consistency loss.

[0027] Optionally, optimizing the initial pose according to the current loss value specifically includes:

[0028] Solve the optimization gradient according to the current loss value and the current initial pose;

[0029] Optimize the current initial pose according to the optimization gradient.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a camera pose solving system based on 3DGS, wherein the camera pose solving system based on 3DGS comprises:

[0031] a target 3D Gaussian model construction module, used to obtain a first preset number of RGBD images, generate point cloud information of the real pose and the current scene according to the first preset number of RGBD images, generate an initial 3D Gaussian model according to the point cloud information, and iteratively optimize the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model;

[0032] An initial pose generation module is used to obtain a target RGB image, input the target RGB image into a deep learning network, and output the initial pose of the camera;

[0033] The target camera pose estimation generation module is used to generate an opacity map, a depth map and a color map according to the target 3D Gaussian model, and iteratively update the initial pose according to the opacity map, the depth map and the color map to obtain the target camera pose estimation.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a 3DGS-based camera pose solving program stored in the memory and executable on the processor, wherein the 3DGS-based camera pose solving program implements the steps of the 3DGS-based camera pose solving method as described above when executed by the processor.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a 3DGS-based camera pose solving program, and when the 3DGS-based camera pose solving program is executed by a processor, the steps of the 3DGS-based camera pose solving method as described above are implemented.

[0036] In the present invention, a first preset number of RGBD images are obtained, and based on the first preset number of RGBD images, the point cloud information of the real pose and the current scene is generated, an initial 3D Gaussian model is generated based on the point cloud information, and the initial 3D Gaussian model is iteratively optimized based on the real pose to obtain a target 3D Gaussian model; a target RGB image is obtained, and the target RGB image is input into a deep learning network to output the initial pose of the camera; an opacity map, a depth map and a color map are generated based on the target 3D Gaussian model, and the initial pose is iteratively updated based on the opacity map, the depth map and the color map to obtain a target camera pose estimate. The present invention trains a Gaussian model, uses a 3D Gaussian model to explicitly model the geometric information of the scene, reversely renders through a 3D Gaussian model, and optimizes the initial pose output by the deep learning network, ultimately achieving a more accurate camera pose estimate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a preferred embodiment of the camera pose solving method based on 3DGS of the present invention;

[0038] Figure 2 Schematic diagram of a pose optimization framework in the camera pose solving method based on 3DGS of the present invention;

[0039] Figure 3 It is a structural diagram of a preferred embodiment of the camera posture solving system based on 3DGS of the present invention;

[0040] Figure 4 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] In technical equipment such as intelligent mobile robots, autonomous driving and augmented reality, the corresponding intelligent devices need to sense their own positions in complex environments, that is, the ability to perform self-positioning, which is the basis of many applications. Visual positioning refers to the process of using visual sensors to obtain environmental information and determine their own positions. At present, the positioning technology based on GNSS can achieve sub-meter positioning in open outdoor scenes, but it is often difficult to obtain signals that can be used for positioning indoors due to the obstruction of obstacles such as buildings. Positioning methods based on signal sensors are usually limited by environmental interference, resulting in performance degradation, expensive hardware costs, and other problems. In contrast, visual sensors are often regarded as the preferred sensors for self-positioning of various intelligent devices due to their simple structure and low cost. However, visual sensors cannot directly obtain depth information and are easily affected by environmental factors such as light. At the same time, there is a cumulative error in the visual odometer, which leads to inaccurate positioning using visual sensors.

[0043] In response to one or more of the above problems, the present invention obtains a first preset number of RGBD images, generates a real pose and point cloud information of the current scene based on the first preset number of RGBD images, generates an initial 3D Gaussian model based on the point cloud information, iteratively optimizes the initial 3D Gaussian model according to the real pose, and obtains a target 3D Gaussian model; obtains a target RGB image, inputs the target RGB image into a deep learning network, and outputs the initial pose of the camera; generates an opacity map, a depth map, and a color map based on the target 3D Gaussian model, iteratively updates the initial pose based on the opacity map, the depth map, and the color map, and obtains an estimate of the target camera pose.

[0044] The camera pose solving method based on 3DGS described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the camera pose solution method based on 3DGS includes the following steps:

[0045] Step S10: Acquire a first preset number of RGBD images, generate point cloud information of a real pose and a current scene according to the first preset number of RGBD images, generate an initial 3D Gaussian model according to the point cloud information, iteratively optimize the initial 3D Gaussian model according to the real pose, and obtain a target 3D Gaussian model.

[0046] It should be noted that the RGBD image is an image containing color information and depth information, that is, a combination of a three-channel color image and a corresponding depth map; 3DGS (Gaussian Splatting) is 3D Gaussian, and in the present invention, the 3D Gaussian model is a model containing multiple parameters.

[0047] Specifically, in the present invention, a corresponding first preset number of RGBD images is obtained to generate the real pose and point cloud information of the current scene, so as to construct a corresponding initial 3D Gaussian model, and then after optimization, a target 3D Gaussian model can be obtained. The first preset number is set by the user according to the specific scene.

[0048] Further, the acquiring of a first preset number of RGBD images, generating point cloud information of a real posture and a current scene according to the first preset number of RGBD images, and generating an initial 3D Gaussian model according to the point cloud information specifically includes:

[0049] Acquire a first preset number of RGBD images, calculate the first preset number of RGBD images according to a visual 3D reconstruction framework to obtain a true pose, and generate point cloud information of the current scene according to the true pose and depth information of the RGBD images;

[0050] The mean and color of the current scene are obtained according to the point cloud information, the preset covariance matrix and opacity are obtained, and an initial 3D Gaussian model is generated according to the mean, the color, the covariance matrix and the opacity.

[0051] Specifically, in the present invention, after obtaining a first preset number of RGBD images, a visual 3D reconstruction framework is used to calculate the first preset number of RGBD images, wherein the visual 3D reconstruction framework is a type of software or framework that can recover the geometric structure of a 3D scene and the camera posture corresponding to each image from a series of 2D images. Through the corresponding visual 3D reconstruction framework, multiple RGBD images can be processed to obtain the true posture of the current camera, and then the point cloud information of the scene where the first preset number of RGBD images are located can be obtained through the true posture and the depth information of the RGBD image. The corresponding model can be constructed using 3D Gaussian through the point cloud information of the current scene.

[0052] According to the point cloud information, 3D Gaussian can be used to obtain the mean μ and the corresponding color c; the mean μ represents the center position of the Gaussian in 3D space; the color c means that each Gaussian ball has a view-dependent color, and when viewed from different perspectives, the color of the Gaussian changes with the perspective. The color c is defined as a 48-dimensional vector that controls the color change seen from different angles; in the model built using 3D Gaussian, an appearance encoding module is introduced for each Gaussian ball, and an MLP (Multilayer Perceptron) is used to add image features to generate more robust color information, thereby improving performance under inconsistent lighting conditions.

[0053] In the present invention, the covariance matrix and opacity are pre-set at the same time, and the corresponding mean, color, covariance matrix and opacity are used as parameters to generate an initial 3D Gaussian model. Among them, the covariance matrix ∑ represents the range and shape covered by the Gaussian in space; the opacity o is used to measure the contribution of each Gaussian to the final rendered image, and the opacity determines its influence on the color of the pixel point in the alpha blending process, and is finally used to generate a renderable image.

[0054] In the present invention, the covariance matrix ∑ is represented in the form of a unit quaternion q and a 3D scaling vector s through a parameterization process. The specific parameterization process is to perform eigenvalue decomposition on the covariance matrix ∑ to obtain eigenvalues ​​and eigenvectors; use the eigenvalues ​​as the scaling vector s; and use the eigenvectors to generate a rotation matrix, which is then represented by the quaternion q. Therefore, for the initial 3D Gaussian model, the parameterization of the i-th Gaussian ball can be expressed as ,in represents the spatial position of the i-th Gaussian, that is, the mean, represents the rotation of the i-th Gaussian ball, represents the scaling of the i-th Gaussian ball, is the color of the i-th Gaussian ball, is the opacity of the i-th Gaussian sphere.

[0055] Further, the iterative optimization of the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model specifically includes:

[0056] Constructing a training opacity map, a training depth map, and a training color map according to the initial 3D Gaussian model, calculating a Gaussian loss according to the training opacity map, the training depth map, the training color map, and the true pose, and optimizing the initial 3D Gaussian model according to the Gaussian loss;

[0057] The initial 3D Gaussian model is optimized by multiple iterations. When the number of iterations reaches a second preset number, the iteration is terminated, and the initial 3D Gaussian model obtained at the last time is used as the target 3D Gaussian model.

[0058] Specifically, in the present invention, for a 3D Gaussian model, the alpha value is calculated by opacity, that is, the specific formula is as follows (1):

[0059] ; (1)

[0060] in, represents the alpha value of the i-th Gaussian.

[0061] After obtaining the alpha value, alpha rendering is performed, where alpha rendering follows the stacking order from far to near, gradually mixing the color and opacity of each Gaussian sphere to generate the final pixel value. The process of layer-by-layer mixing from back to front is as follows (2):

[0062] ; (2)

[0063] Where C represents the corresponding color map, represents the number of Gaussian balls in the scene, Represents the occlusion effect of the first i-1 Gaussian balls on the opacity of the i-th Gaussian ball, ensuring that the color superposition order conforms to the actual visibility, where represents cumulative multiplication, j represents the corresponding Gaussian, where the value of j ranges from 1 to i-1, represents the alpha value of the j-th Gaussian.

[0064] Correspondingly, the rendering formula for opacity is formula (3):

[0065] ; (3)

[0066] Among them, S represents the corresponding opacity map.

[0067] The depth rendering formula is as follows (4):

[0068] ; (4)

[0069] in is the distance from the i-th Gaussian to the camera.

[0070] In the present invention, for the mean, color, covariance matrix and opacity in the initial 3D Gaussian model, the above formulas (1), (2), (3) and (4) are used in sequence to calculate the training opacity map, the training depth map and the training color map.

[0071] The pixels in the training opacity map that exceed the first preset opacity threshold are used as target training pixels; in one embodiment of the present invention, the first preset opacity threshold is set to 0.9, that is, points with S>0.9 are selected for sampling, that is, this part is used as the target training pixel. The Gaussian loss between the depth map and the color map corresponding to the real pose in the training depth map and the training color map is calculated. The depth map and color map corresponding to the real pose are obtained by calculating the pixel coordinates of the real pose, where the pixel coordinates of the real pose q f1 = K f1 *P1*μ1,K f1 To obtain the camera intrinsic parameters of the camera of the first preset number of RGBD images, P1 is the camera pose of the corresponding camera, μ1 is the attribute of the corresponding Gaussian sphere, and the Gaussian sphere can be projected onto the view corresponding to the real pose through the pixel coordinates, that is, the number of Gaussians existing on each pixel can be obtained. Then, the color map and depth map corresponding to the real pose are calculated through formula (2) and formula (4), that is, the formulas (2) and (4) are replaced by By replacing the hit Gaussian number, we can get the color map and depth map corresponding to the real pose.

[0072] In the present invention, when calculating the loss between the corresponding training color map and the pose, the calculation is performed using the corresponding training color map and the color map of the real pose. The specific calculation formula is shown in the following formula (5):

[0073] ; (5)

[0074] in, represents the loss between the training color map and the color map of the true pose, The color map representing the corresponding true pose, represents the training color map, Represents the number of pixels in the training color map, k is used to represent the label of a pixel, Represents the color corresponding to the k-th pixel in the training color map.

[0075] When calculating the loss between the corresponding training depth map and the pose, the corresponding training depth map and the depth map of the real pose are used for calculation. The specific calculation formula is shown in the following formula (6):

[0076] ; (6)

[0077] in, represents the loss between the training depth map and the depth map of the true pose, represents the depth map corresponding to the true pose, represents the training depth map, Represents the depth corresponding to the kth pixel in the training depth map.

[0078] After obtaining the corresponding loss, Gaussian loss can be expressed as the following formula (7):

[0079] ; (7)

[0080] in, is the Gaussian loss, That is to select the training opacity map exceeds the first preset opacity threshold Pixels.

[0081] After obtaining the corresponding Gaussian loss, the initial 3D Gaussian model is optimized by gradient, that is, the mean, color, covariance matrix and opacity are optimized. The optimization method is to derive the corresponding parameters through Gaussian loss, that is, to solve the gradient , , and Implement optimizations for mean, color, covariance matrix, and opacity.

[0082] The initial 3D Gaussian model is optimized by multiple iterations. When the number of iterations reaches a second preset number, the iteration is terminated, and the initial 3D Gaussian model obtained last time is used as the target 3D Gaussian model; the second preset number is the number set by the user, preferably 20,000 times.

[0083] Step S20: Obtain a target RGB image, input the target RGB image into a deep learning network, and output an initial pose of the camera.

[0084] Specifically, in the present invention, the deep learning network used is pre-trained, and the deep learning network is PoseNet, which is a deep learning network used for the regression task of the absolute position of the camera. Its main function is to directly output the 6D camera position corresponding to the image by inputting a single RGB image, including 3D translation and 3D rotation; this method does not require a priori 3D models or complex geometric reconstruction, and can quickly estimate the position and orientation of the camera in space.

[0085] In the present invention, in the process of camera absolute pose regression, a deep learning network, namely PoseNet, is used to predict the pose of the input image and generate the 6D camera pose corresponding to the image, namely the initial pose, including 3D translation and 3D rotation. Through this process, the initial pose can be generated for the input image.

[0086] Step S30, generating an opacity map, a depth map and a color map according to the target 3D Gaussian model, and iteratively updating an initial pose according to the opacity map, the depth map and the color map to obtain a target camera pose estimate.

[0087] Specifically, in the present invention, for the initial pose, a target 3D Gaussian model is used for optimization, thereby obtaining a target camera pose estimate.

[0088] Further, generating an opacity map, a depth map and a color map according to the target 3D Gaussian model specifically includes:

[0089] Calculate an alpha value according to the opacity of the target 3D Gaussian model;

[0090] According to the alpha value and the target 3D Gaussian model, an alpha blending algorithm is used to respectively calculate the opacity map, the depth map and the color map.

[0091] Specifically, in the present invention, the alpha value of the target 3D Gaussian model is calculated according to the above formula (1), and then the opacity map, the depth map and the color map are respectively calculated by using the alpha value and the target 3D Gaussian model using an alpha blending algorithm, that is, the opacity map, the depth map and the color map are respectively calculated using the above formulas (2), (3) and (4).

[0092] Further, the iterative updating of the initial pose according to the opacity map, the depth map and the color map to obtain the target camera pose estimation specifically includes:

[0093] Calculate a current loss value according to the initial pose, the opacity map, the depth map, and the color map, and optimize the initial pose according to the current loss value;

[0094] The initial pose is iterated multiple times. When the number of iterations reaches a third preset number, the iteration is terminated, and the initial pose obtained in the last iteration is used as the target camera pose estimate.

[0095] Specifically, for the initial pose, the present invention calculates the corresponding current loss value through the opacity map, depth map and color map of the target 3D model, and optimizes the initial pose through the current loss value; after the initial pose has gone through a third preset number of iterations, the iteration ends and the corresponding target camera pose estimate is output. Among them, the third preset number is the number of iterations set by the user, preferably 30 times. Through multiple rounds of rendering and back propagation iterations, the 6D pose of the camera is gradually optimized, so that the target camera pose estimate is more accurate under the constraints of geometric information.

[0096] Further, the calculating the current loss value according to the initial pose, the opacity map, the depth map and the color map specifically includes:

[0097] Taking pixels in the opacity map that exceed a preset opacity threshold as target pixels;

[0098] Sampling the target pixel in the depth map and the color map to obtain a target depth map and a target color map, and generating a query color map and a query depth map according to the initial pose;

[0099] Calculating the consistency loss between the query depth map and the depth map according to the target depth map to obtain a depth consistency loss, and calculating the consistency loss between the query color map and the color map according to the target color map to obtain a color consistency loss;

[0100] A current loss value is generated according to the depth consistency loss and the color consistency loss.

[0101] Specifically, in the present invention, the pixels in the opacity map that exceed the preset opacity threshold are taken as target pixels, that is, for the pixels in the opacity map, the pixels that exceed the preset opacity threshold are selected as target pixels. At the same time, the present invention generates a query color map and a query depth map according to the initial pose. In the present invention, for the initial pose, the pixel coordinates thereof are calculated accordingly, where the pixel coordinate q f = K f *P*μ,K f is the camera internal parameter, P is the camera pose of the current camera, μ is the property of the corresponding Gaussian ball, that is, the Gaussian center of the target 3D Gaussian model. Through the pixel coordinates, the Gaussian ball can be projected onto the view corresponding to the initial pose, and the number of Gaussians on each pixel can be obtained. Then, the number of hit Gaussians is calculated through formulas (2) and (4), that is, By replacing the hit Gaussian number, the corresponding query color map and query depth map can be obtained. In the present invention, for the target pixel, the consistency loss between the query depth map and the depth map is calculated according to the target depth map to obtain the depth consistency loss, and the consistency loss between the query color map and the color map is calculated according to the target color map to obtain the color consistency loss.

[0102] Among them, the color consistency loss can be calculated by the following formula (8), which is expressed as:

[0103] ; (8)

[0104] in, represents the loss of color consistency, represents the query color map, represents the target color map, p represents the number of target pixels, n represents the label of a pixel, Indicates the color corresponding to the nth pixel in the target color map.

[0105] The depth consistency loss can be calculated by the following formula (9), which is expressed as:

[0106] ; (9)

[0107] in, represents the depth consistency loss, represents the query depth graph, represents the target depth map, p represents the number of target pixels, n represents the label of a pixel, Indicates the depth corresponding to the nth pixel in the target depth map.

[0108] After obtaining the color consistency loss and the depth consistency loss, the current loss value is generated according to formula (10), where the formula corresponding to the current loss value is:

[0109] ; (10)

[0110] Among them, L is the current loss value, That is, pixels in the training opacity map S that exceed the preset opacity threshold M are selected.

[0111] Furthermore, the optimizing the initial posture according to the current loss value specifically includes:

[0112] Solve the optimization gradient according to the current loss value and the current initial pose;

[0113] Optimize the current initial pose according to the optimization gradient.

[0114] Specifically, after obtaining the corresponding current loss value, solve the optimization gradient, that is, , to optimize the initial pose. In this process, the initial pose is iteratively adjusted to minimize the error between the synthetic image generated by rendering and the actual image, thereby improving the accuracy of the target camera pose estimation.

[0115] Furthermore, if Figure 2 As shown, in the present invention, the pose obtained by the neural network is optimized by generating an opacity map, a depth map and a color map, thereby obtaining the final target camera pose estimation.

[0116] The present invention obtains a first preset number of RGBD images, generates point cloud information of the real pose and the current scene based on the first preset number of RGBD images, generates an initial 3D Gaussian model based on the point cloud information, iteratively optimizes the initial 3D Gaussian model based on the real pose, and obtains a target 3D Gaussian model; obtains a target RGB image, inputs the target RGB image into a deep learning network, and outputs the initial pose of the camera; generates an opacity map, a depth map, and a color map based on the target 3D Gaussian model, iteratively updates the initial pose based on the opacity map, the depth map, and the color map, and obtains a target camera pose estimate. The present invention trains a Gaussian model, uses a 3D Gaussian model to explicitly model the geometric information of the scene, and optimizes the initial pose output by the deep learning network through reverse rendering gradient optimization based on 3DGS, ultimately achieving a higher-precision camera pose estimation.

[0117] Furthermore, if Figure 3 As shown, based on the above-mentioned camera pose solving method based on 3DGS, the present invention also provides a camera pose solving system based on 3DGS, wherein the camera pose solving system based on 3DGS includes:

[0118] A target 3D Gaussian model construction module 31 is used to obtain a first preset number of RGBD images, generate point cloud information of a real pose and a current scene according to the first preset number of RGBD images, generate an initial 3D Gaussian model according to the point cloud information, and iteratively optimize the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model;

[0119] An initial pose generation module 32 is used to obtain a target RGB image, input the target RGB image into a deep learning network, and output an initial pose of the camera;

[0120] The target camera pose estimation generation module 33 is used to generate an opacity map, a depth map and a color map according to the target 3D Gaussian model, and iteratively update the initial pose according to the opacity map, the depth map and the color map to obtain the target camera pose estimation.

[0121] Furthermore, if Figure 4 As shown, based on the above-mentioned 3DGS-based camera pose solving method and system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0122] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a camera pose solving program 40 based on 3DGS is stored on the memory 20, and the camera pose solving program 40 based on 3DGS can be executed by the processor 10, thereby realizing the camera pose solving method based on 3DGS in the present invention.

[0123] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the 3DGS-based camera pose solution method.

[0124] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0125] In one embodiment, when the processor 10 executes the 3DGS-based camera pose solving program 40 in the memory 20 , the steps of the above 3DGS-based camera pose solving method are implemented.

[0126] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a 3DGS-based camera pose solving program, and when the 3DGS-based camera pose solving program is executed by a processor, the steps of the 3DGS-based camera pose solving method as described above are implemented.

[0127] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0128] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0129] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A camera pose solution method based on 3DGS, characterized in that: The camera pose solving method based on 3DGS includes: Acquire a first preset number of RGBD images, generate point cloud information of a real pose and a current scene according to the first preset number of RGBD images, generate an initial 3D Gaussian model according to the point cloud information, and iteratively optimize the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model; Obtain a target RGB image, input the target RGB image into a deep learning network, and output the initial pose of the camera; Generate an opacity map, a depth map and a color map according to the target 3D Gaussian model, and iteratively update the initial pose according to the opacity map, the depth map and the color map to obtain a target camera pose estimate; The iterative updating of the initial pose according to the opacity map, the depth map and the color map to obtain the target camera pose estimation specifically includes: Calculate a current loss value according to the initial pose, the opacity map, the depth map, and the color map, and optimize the initial pose according to the current loss value; Iterate the initial pose multiple times, and when the number of iterations reaches a third preset number, end the iteration, and use the initial pose obtained from the last iteration as the target camera pose estimate; The calculating the current loss value according to the initial pose, the opacity map, the depth map and the color map specifically includes: Taking pixels in the opacity map that exceed a preset opacity threshold as target pixels; Sampling the target pixel in the depth map and the color map to obtain a target depth map and a target color map, and generating a query color map and a query depth map according to the initial pose; Calculating the consistency loss between the query depth map and the depth map according to the target depth map to obtain a depth consistency loss, and calculating the consistency loss between the query color map and the color map according to the target color map to obtain a color consistency loss; Generate a current loss value according to the depth consistency loss and the color consistency loss; Optimizing the initial pose according to the current loss value specifically includes: Solve the optimization gradient according to the current loss value and the current initial pose; Optimize the current initial pose according to the optimization gradient.

2. The camera pose solving method based on 3DGS according to claim 1, characterized in that: The acquiring of a first preset number of RGBD images, generating point cloud information of a real posture and a current scene according to the first preset number of RGBD images, and generating an initial 3D Gaussian model according to the point cloud information specifically includes: Acquire a first preset number of RGBD images, calculate the first preset number of RGBD images according to a visual 3D reconstruction framework to obtain a true pose, and generate point cloud information of the current scene according to the true pose and depth information of the RGBD images; The mean and color of the current scene are obtained according to the point cloud information, the preset covariance matrix and opacity are obtained, and an initial 3D Gaussian model is generated according to the mean, the color, the covariance matrix and the opacity.

3. The camera pose solving method based on 3DGS according to claim 2, characterized in that: The iterative optimization of the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model specifically includes: Constructing a training opacity map, a training depth map, and a training color map according to the initial 3D Gaussian model, calculating a Gaussian loss according to the training opacity map, the training depth map, the training color map, and the true pose, and optimizing the initial 3D Gaussian model according to the Gaussian loss; The initial 3D Gaussian model is optimized by multiple iterations. When the number of iterations reaches a second preset number, the iteration is terminated, and the initial 3D Gaussian model obtained at the last time is used as the target 3D Gaussian model.

4. The camera pose solving method based on 3DGS according to claim 1, characterized in that: Generating an opacity map, a depth map, and a color map according to the target 3D Gaussian model specifically includes: Calculate an alpha value according to the opacity of the target 3D Gaussian model; According to the alpha value and the target 3D Gaussian model, an alpha blending algorithm is used to respectively calculate the opacity map, the depth map and the color map.

5. A camera pose solving system based on 3DGS, characterized in that: The camera pose solving system based on 3DGS includes: a target 3D Gaussian model construction module, used to obtain a first preset number of RGBD images, generate point cloud information of the real pose and the current scene according to the first preset number of RGBD images, generate an initial 3D Gaussian model according to the point cloud information, and iteratively optimize the initial 3D Gaussian model according to the real pose to obtain a target 3D Gaussian model; An initial pose generation module is used to obtain a target RGB image, input the target RGB image into a deep learning network, and output the initial pose of the camera; A target camera pose estimation generation module is used to generate an opacity map, a depth map and a color map according to the target 3D Gaussian model, and iteratively update the initial pose according to the opacity map, the depth map and the color map to obtain a target camera pose estimate; The iterative updating of the initial pose according to the opacity map, the depth map and the color map to obtain the target camera pose estimation specifically includes: Calculate a current loss value according to the initial pose, the opacity map, the depth map, and the color map, and optimize the initial pose according to the current loss value; Iterate the initial pose multiple times, and when the number of iterations reaches a third preset number, end the iteration, and use the initial pose obtained from the last iteration as the target camera pose estimate; The calculating the current loss value according to the initial pose, the opacity map, the depth map and the color map specifically includes: Taking pixels in the opacity map that exceed a preset opacity threshold as target pixels; Sampling the target pixel in the depth map and the color map to obtain a target depth map and a target color map, and generating a query color map and a query depth map according to the initial pose; Calculating the consistency loss between the query depth map and the depth map according to the target depth map to obtain a depth consistency loss, and calculating the consistency loss between the query color map and the color map according to the target color map to obtain a color consistency loss; Generate a current loss value according to the depth consistency loss and the color consistency loss; Optimizing the initial pose according to the current loss value specifically includes: Solve the optimization gradient according to the current loss value and the current initial pose; Optimize the current initial pose according to the optimization gradient.

6. A terminal, characterized in that: The terminal includes: a memory, a processor, and a 3DGS-based camera pose solving program stored in the memory and executable on the processor. When the 3DGS-based camera pose solving program is executed by the processor, the steps of the 3DGS-based camera pose solving method as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a 3DGS-based camera pose solving program, and when the 3DGS-based camera pose solving program is executed by a processor, the steps of the 3DGS-based camera pose solving method as described in any one of claims 1 to 4 are implemented.

Images