Camera pose visual measurement method based on three-dimensional Gaussian splashing

Abstract

The invention discloses a camera pose vision measurement method based on three-dimensional Gaussian splash, and relates to the field of camera pose measurement, and the method comprises the steps: obtaining an RGB image of a target scene, a three-dimensional Gaussian splash model, and an initial pose of a camera; applying disturbance to the initial pose for a plurality of times to generate a plurality of disturbance poses; based on a three-dimensional Gaussian splash model, the initial pose and each disturbance pose are rendered to obtain L + 1 rendered images; carrying out 2D-2D feature matching on the RGB image of the target scene and each rendering image; based on the L + 1 groups of matching point pairs, determining a relative pose between each rendered image and the RGB image of the target scene; taking minimization of the target function as a target, determining a pose update value, updating the initial pose, and returning to the step to apply disturbance to the initial pose for a plurality of times; and repeating the steps until the convergence condition is reached. According to the invention, the accuracy and efficiency of pose measurement can be improved.

Description

Technical Field

[0001] This application relates to the field of camera pose measurement, and in particular to a camera pose visual measurement method based on three-dimensional Gaussian splashing. Background Technology

[0002] Camera pose measurement plays a crucial role in fields such as autonomous navigation of intelligent robots, autonomous driving, augmented reality, and drone localization. In recent years, optimization-based pose measurement methods have gained widespread attention. Compared to directly estimating the six-DOF initial pose from monocular RGB images, optimization-based pose measurement methods typically use the pose of the previous frame in the image sequence as input, and iteratively optimize to gradually improve the accuracy and robustness of the pose output.

[0003] Existing optimization-based camera pose measurement methods mainly fall into two categories: one based on 2D-3D correspondences, using motion reconstruction to model scene geometry and appearance features, and relying on specific feature descriptors to establish 2D-3D correspondences for pose optimization; the other relies on task-specific neural networks, typically requiring the design of scene-specific feature extraction modules and end-to-end training using a camera pose-based loss function. However, the former often requires rebuilding the scene model when replacing it with a more efficient feature descriptor, while the latter necessitates building a separate labeled dataset and retraining the neural network for each new application scenario, limiting the applicability and practical deployment efficiency of these methods across different scenarios. Recent methods evaluate the pose differences between query and rendered images based on image feature similarity and update pose using particle filters, but they fail to fully consider the geometric relationships between the new views during pose calculation, affecting the accuracy of pose measurement. Summary of the Invention

[0004] The purpose of this application is to provide a camera pose visual measurement method based on three-dimensional Gaussian splashing, which can improve the accuracy and efficiency of pose measurement.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a camera pose visual measurement method based on three-dimensional Gaussian splashing, including: Acquire the RGB image of the target scene, the 3D Gaussian splash model, and the initial pose of the camera; Apply several perturbations to the initial pose to generate several perturbed poses; wherein, each perturbation generates one perturbed pose. Based on the 3D Gaussian splash model, the initial pose and each perturbation pose are rendered to obtain L+1 rendered images; L represents the number of perturbation poses. The RGB image of the target scene is matched with each rendered image using 2D-2D features to obtain L+1 sets of 2D-2D matching point pairs. Based on L+1 sets of 2D-2D matching point pairs, the relative pose between each rendered image and the RGB image of the target scene is determined; Based on the relative pose between each rendered image and the RGB image of the target scene, and L+1 sets of 2D-2D matching point pairs, the pose update value is determined with the goal of minimizing the objective function; the objective function includes a geometric consistency objective function and an objective function based on epipolar geometric constraints. Determine whether the convergence condition has been met; if not, determine the pose update value as the initial pose of the camera and return to the step "apply several perturbations to the initial pose to generate several perturbed poses"; if yes, determine the pose update value as the final camera pose.

[0006] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a camera pose visual measurement method based on 3D Gaussian splashing. It acquires the RGB image of the target scene, a 3D Gaussian splashing model, and the initial pose of the camera. Several perturbation strategies are applied to the initial pose to generate a corresponding number of perturbed poses. Based on the 3D Gaussian splashing model, the initial pose and each perturbed pose are rendered to obtain L+1 rendered images (L being the number of perturbed poses). The RGB image of the target scene is then matched with each rendered image using 2D-2D feature matching to obtain L+1 sets of 2D-2D matching point pairs. Based on the L+1 sets of 2D-2D matching point pairs... The algorithm uses D matching point pairs to determine the relative pose between each rendered image and the RGB image of the target scene. Based on the relative pose between each rendered image and the RGB image of the target scene, and L+1 sets of 2D-2D matching point pairs, it determines the pose update value by minimizing the objective function (geometric consistency objective function and epipolar geometric constraint objective function). It then determines whether the convergence condition has been met. If not, the pose update value is determined as the initial pose of the camera, and the algorithm returns to the step "apply several perturbations to the initial pose to generate several perturbed poses". If yes, the pose update value is determined as the final camera pose. The above-mentioned solution directly utilizes the existing 3D Gaussian splash model to render images from new perspectives, eliminating the need for additional training or fine-tuning for different scenes, thus providing a lightweight solution and improving efficiency. Furthermore, based on the geometric constraint objective function and iterative optimization, it minimizes the error, thereby achieving accurate measurement of the camera pose and significantly improving the adaptability and robustness of the technical solution in practical applications. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is an application environment diagram of the camera pose visual measurement method based on three-dimensional Gaussian splashing in one embodiment of this application; Figure 2 A schematic flowchart of a camera pose visual measurement method based on three-dimensional Gaussian splashing provided in an embodiment of this application; Figure 3 A flowchart illustrating another camera pose visual measurement method based on three-dimensional Gaussian splashing provided in an embodiment of this application; Figure 4 A schematic diagram of pose estimation based on geometric consistency constraints provided in an embodiment of this application; Figure 5 A schematic diagram of pose optimization based on epipolar geometry provided in an embodiment of this application; Figure 6 This is a schematic diagram of a parallel optimization strategy provided in an embodiment of this application; Figure 7(a) is a schematic diagram of the effect of the head pose measurement method provided in an embodiment of this application; Figure 7(b) is a schematic diagram of the effect of the stair pose measurement method provided in an embodiment of this application; Figure 8 A schematic diagram of the functional modules of a camera pose vision measurement device based on three-dimensional Gaussian splashing provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0011] The camera pose visual measurement method based on three-dimensional Gaussian splashing provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can send the RGB image of the target scene, the 3D Gaussian splash model, and the initial pose of the camera to server 104. After receiving the data, server 104 applies several perturbation strategies to the initial pose to generate several perturbed poses; based on the 3D Gaussian splash model, it renders the initial pose and each perturbed pose to obtain L+1 rendered images; it performs 2D-2D feature matching between the RGB image of the target scene and each rendered image to obtain L+1 sets of 2D-2D matching point pairs; based on L+... One set of 2D-2D matching point pairs is used to determine the relative pose between each rendered image and the RGB image of the target scene. Based on the relative pose between each rendered image and the RGB image of the target scene and L+1 sets of 2D-2D matching point pairs, the pose update value is determined with the objective function as the minimization objective. It is then determined whether the convergence condition has been met. If not, the pose update value is determined as the initial pose of the camera, and the process returns to the step "apply several perturbations to the initial pose to generate several perturbed poses". If yes, the pose update value is determined as the final camera pose. The server 104 can feed back the obtained final camera pose to the terminal 102. In addition, in some embodiments, the camera pose visual measurement method based on three-dimensional Gaussian splash can also be implemented by the server 104 or the terminal 102 separately. For example, the terminal 102 can directly process the RGB image of the target scene, the three-dimensional Gaussian splash model, and the initial pose of the camera, or the server 104 can obtain the RGB image of the target scene, the three-dimensional Gaussian splash model, and the initial pose of the camera from the data storage system and process them.

[0012] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0013] In one exemplary embodiment, such as Figure 2 and Figure 3 As shown, a camera pose visual measurement method based on three-dimensional Gaussian splashing is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 207. Wherein: Step 201: Obtain the RGB image of the target scene (also called the scene), the 3D Gaussian splash model, and the initial pose of the camera. The initial pose can be the calculation result of the previous frame in the video sequence, or provided by an external sensor or other pose estimation method.

[0014] Step 202: Apply several perturbations to the initial pose to generate several perturbed poses; wherein, each perturbation generates one perturbed pose.

[0015] Step 203: Based on the three-dimensional Gaussian splash model, render the initial pose and each perturbation pose to obtain L+1 rendered images; L represents the number of perturbation poses.

[0016] Step 204: Perform 2D-2D feature matching between the RGB image of the target scene and each rendered image to obtain L+1 sets of 2D-2D matching point pairs.

[0017] Step 205: Based on the L+1 group of 2D-2D matching point pairs, determine the relative pose between each rendered image and the RGB image of the target scene.

[0018] Step 206: Based on the relative poses between each rendered image and the RGB image of the target scene and L+1 sets of 2D-2D matching point pairs, determine the pose update value with the objective function as the goal; the objective function includes a geometric consistency objective function and an objective function based on epipolar geometric constraints.

[0019] Step 207: Determine whether the convergence condition has been met; if not, determine the pose update value as the initial pose of the camera and return to the step "Apply several perturbations to the initial pose to generate several perturbed poses"; if yes, determine the pose update value as the final camera pose.

[0020] As an optional implementation, step 206 specifically includes: Step 206.1: Based on the relative pose between each rendered image and the RGB image of the target scene, determine the first initial pose with the objective of minimizing the geometric consistency objective function.

[0021] Step 206.2: Based on the first initial pose and L+1 sets of 2D-2D matching point pairs, determine the pose update value with the objective of minimizing the objective function based on epipolar geometry constraints.

[0022] As an optional implementation, the mathematical expression of the geometric consistency objective function is: ; in, This represents the rotation matrix in the first initial pose of the camera to be estimated; the superscript -1 indicates the inverse. This represents the translation vector in the first initial pose of the camera to be estimated; Represents the RGB image of the target scene and the initial pose rendering image. The relative rotation matrix; Indicates the initial pose rendering image The rotation matrix; Represents the RGB image and the perturbed pose rendering image of the target scene. The relative rotation matrix; Represents perturbation pose rendering image The rotation matrix; Indicates the initial pose rendering image Translation vector; Represents perturbation pose rendering image Translation vector; Represents the RGB image of the target scene and the initial pose rendering image. The normalized translation vector; Represents the RGB image and the perturbed pose rendering image of the target scene. The normalized translation vector; Represents the normalized translation vector Scale factor; Represents the normalized translation vector The scale factor.

[0023] As an optional implementation, the mathematical expression of the objective function based on epipolar geometry constraints is: ; ; ; in, This represents the pose update value to be estimated; The initial value is the first initial pose; This indicates the number of feature points in the RGB image of the target scene; Indicates the initial pose rendering image The number of feature points in the middle; Represents perturbation pose rendering image The number of feature points in the middle; Represents feature points in the RGB image of the target scene With corresponding mapping point The distance between the polar lines and the mapping point The polar line is based on the mapping point. The perturbation pose of the rendered image and the pose update value to be estimated. Certain; for or ;when for At that time, the mapping point Rendering images for initial pose Zhongyu Matching feature points, and ;when for At that time, the mapping point Rendering images for perturbed poses Zhongyu Matching feature points, and ; express or .

[0024] Mapping point Determining the polar line: First, based on the mapping point The perturbation pose of the rendered image and the pose update value to be estimated. Determine the mapping point The relative pose of the rendered image and the RGB image of the target scene; then, based on the mapping points... The relative pose of the rendered image and the RGB image of the target scene determines the mapping point. The polar line in question. See the polar equation below.

[0025] As an optional implementation, the objective function based on epipolar geometry constraints is solved using the Levenberg-Marquardt algorithm.

[0026] As an alternative implementation, before using the Levenberg-Marquardt algorithm to solve the objective function based on epipolar geometry constraints, the pose update value to be estimated by the camera is expressed in Lie algebra form, and then the pose update value to be estimated is solved based on the Lie algebra form.

[0027] As an optional implementation, based on a 3D Gaussian splash model, the initial pose and each perturbation pose are rendered separately to obtain L+1 rendered images, specifically including: Based on the three-dimensional Gaussian splash model, parallel processing is used to render the initial pose and each perturbation pose simultaneously to obtain L+1 rendered images.

[0028] As an optional implementation, 2D-2D feature matching is performed between the RGB image of the target scene and each rendered image to obtain L+1 sets of 2D-2D matching point pairs, specifically including: By using parallel processing, the RGB image of the target scene is simultaneously matched with each rendered image using 2D-2D features to obtain L+1 sets of 2D-2D matching point pairs.

[0029] As an optional implementation, the convergence condition is that the number of iterations reaches a preset number. That is, in specific applications, a fixed number of iterations can be preset, such as 5 or 6 times.

[0030] To help those skilled in the art to further understand, further explanation is provided below.

[0031] This application proposes a camera pose visual measurement method based on 3D Gaussian splashing. This method optimizes the camera's position and attitude parameters using RGB query images (RGB images) captured by a monocular camera, along with calculations from the previous frame or initial poses provided by external sensors / other pose estimation methods, provided the camera's intrinsic parameters are known and a 3D Gaussian splashing model of the scene is pre-constructed. The method includes the following steps: Step 101: Obtain the scene RGB image captured by the monocular camera as the query image, and load the pre-built scene 3D Gaussian splash model and initial pose, wherein the initial pose can be calculated from the previous frame in the video sequence or provided by an external sensor / other pose estimation method.

[0032] Step 102: Apply a perturbation strategy to the initial pose to generate multiple perturbation poses, and render multiple rendering images corresponding to the initial pose and each perturbation pose based on the three-dimensional Gaussian splash model, forming a set of rendering images consisting of a reference image and multiple candidate images.

[0033] Step 103: Perform 2D-2D feature matching between each image in the rendered image set and the query image to obtain multiple sets of feature matching point pairs, and calculate the relative pose between each rendered image and the query image based on the matching point pairs.

[0034] Step 104: Construct an objective function based on the relative pose relationship between the query image and the rendered image and the feature matching point pairs, and update the camera pose with the goal of minimizing the objective function.

[0035] Step 105: Use the optimized pose as the initial value for the new iteration and return to step 102 to continue optimization until the convergence condition is met, and output the final camera pose.

[0036] The advantages of this method are: it does not rely on specific feature descriptors and does not require training of task-related neural networks. It achieves camera pose optimization through multiple sets of 2D-2D matching relationships, and can flexibly accommodate any feature extraction and matching methods, thereby improving the accuracy of camera pose estimation based on monocular RGB images.

[0037] Step 1: Pose perturbation strategy and new perspective image rendering.

[0038] First, obtain the currently queried image. Initial pose And the current scene's 3D Gaussian splash model, where, Represents the target pose. and These represent the rotation matrix and translation vector of the scene coordinate system relative to the camera coordinate system, respectively.

[0039] SO(3) is an abbreviation for Special Orthogonal Group, representing the rotation group in three-dimensional Euclidean space. Specifically, it can be represented as... .

[0040] Therefore, the initial pose can be determined using a three-dimensional Gaussian splash model. Render the image to generate a corresponding reference image. This refers to the initial pose rendering image. Subsequently, the initial pose... Apply Gaussian noise perturbation to generate The assumed pose, where, for the th Each of the assumed poses is subjected to rotational and translational perturbations, where the rotational perturbation has a mean of 0 and a standard deviation of . The mean of the translational disturbance is 0, and the standard deviation is . The assumed pose after perturbation is obtained. (i.e., perturbed pose). For each perturbed assumed pose, repeat the above rendering process to obtain its corresponding synthetic viewpoint rendered image. This involves perturbing the pose to render the image. Through the above process, a set of rendered images is obtained. .

[0041] Step 2: 2D-2D feature matching.

[0042] Compare the query image with the generated This method performs 2D-2D feature matching on the rendered image. It can use any 2D-2D feature extraction and matching algorithm to obtain the query image. With rendering images Point correspondence: ; in, and Each contains One and One feature point, express The Middle Each feature point and The Middle Each feature point forms a one-to-one correspondence. In this paper, we use... Unified representation .

[0043] Step 3: Pose estimation based on geometric consistency constraints.

[0044] (1) Based on what was obtained Group 2D-2D matching point pairs to query image With rendering images The relative poses between them can be calculated using the following formula: ; in, Represents the camera intrinsic parameter matrix. Indicates query image With rendering images The relative rotation matrix. Since the above equation equals zero, it introduces uncertainty in the scale of the position vector, making it impossible to directly calculate the matrix containing the scale factor. absolute translation vector It can only solve for the normalized translation vector. , express An antisymmetric matrix.

[0045] (2) Regarding Zhang Rendered Image The above formula can be used to obtain and Relative pose between , and Relative pose between To solve for the scaling factor and Based on the camera's motion relationships between views, the following constraints are established: ; ; in, To retrieve the pose estimate corresponding to the image, we can further obtain: ; ; Therefore, the scaling factor can be obtained. and Substitute the obtained scaling factor into and , can obtain There are several pose estimation solutions. To estimate the camera pose from these multiple pose solutions, consistency optimization is needed.

[0046] Among them, the above and This is a theoretical model describing the constraints that should be satisfied between the camera pose to be solved and the poses of different rendered views. Ideally, the same solution should be obtained; however, in the actual solution process, due to the influence of noise and other factors, the results may differ. Each independent equation established from the new perspective yields a solution for the camera pose, resulting in a total of L+1 solutions. This presents an overdetermined problem. Therefore, to address this issue, a pose estimation method based on geometric consistency constraints is proposed. The aim is to find an optimal camera pose that simultaneously satisfies... and Given the described geometric relationships, find a solution that minimizes the residuals of all geometric constraints globally.

[0047] (3) The geometric consistency objective function is defined as the sum of squared transformation errors between the camera pose to be solved and the relative poses of multiple different viewpoints, so that the estimated camera pose results under different viewpoints are as consistent as possible, such as Figure 4 As shown. In Figure 4 middle, , , , , Both represent points in three-dimensional space.

[0048] Step 4: Pose optimization based on epipolar geometry.

[0049] Since the relative pose is calculated based on 2D-2D matching point pairs, a pose optimization method based on epipolar geometry is further proposed to reduce the impact of mismatches on pose measurement accuracy. (See [reference needed]). Figure 5 .

[0050] (1) In epipolar geometry, for the image Feature points in There exists a unique polar line. Passing through its corresponding mapping point (Located in the rendered image) The polar equation can be expressed as: ; in, This represents a relative translation vector that includes a scale factor; the superscript -T indicates the inverse and transpose; the superscript T indicates the transpose. They are respectively Projections onto the x-axis, y-axis, and z-axis.

[0051] After constructing the polar equation to its corresponding polar line The distance can be expressed as: ; Therefore, by minimizing the square of the sum of the distances from all matching feature points to the epipolar line, such as Figure 3 As shown, an objective function based on epipolar geometric constraints is established.

[0052] Among these, the Huber function is used to improve the robustness of the objective function to outliers during the update process, making We can obtain: ; .

[0053] In the above formula, the following is adopted: Unified representation and .

[0054] Since step three calculates the relative pose based on the positions of matched keypoints in the image and then solves the pose, mismatched point pairs will affect the calculation results. Therefore, this step uses the pose obtained in step three as the starting point for iterative optimization, and establishes an objective function based on minimizing the square of the sum of the distances from all matched feature points to the epipolar line. By minimizing this objective function, the pose solution obtained in the previous step is further optimized.

[0055] exist Figure 5 middle, It is a point in three-dimensional space; , , Points In the image , , Imaging points in; express Corresponding mapping point The polar line where it is located; express Corresponding mapping point The polar line where it is located; express to the poles The distance; express to the poles The distance.

[0056] (2) Solving for pose update.

[0057] The Levenberg-Marquardt (LM) algorithm is used to solve the objective function based on epipolar geometry constraints. To ensure that the algorithm minimizes the objective function using gradient descent optimization, the derivative of the objective function with respect to the camera pose must be calculated to determine the update direction. Due to the camera pose... When, rotation matrix The camera pose is inherently constrained (i.e., orthogonal and with a determinant of 1). Using it as an optimization variable would introduce additional constraints. Therefore, the camera pose needs to be parameterized into its corresponding Lie algebra form in order to calculate the gradient of the error function with respect to the camera pose and update the camera pose solution. SE(3) denotes a special Euclidean group.

[0058] Specifically, this method uses Lie algebras. Constrained camera pose matrix Mapped to an unconstrained six-dimensional vector This achieves parameterization without redundancy. The rotation and translation vectors of this vector can be represented as: ; in, Represents the pose in the form of a six-dimensional vector; Represents the translation vector; This represents the rotation vector.

[0059] and These are the Lie algebra forms of SE(3) and SO(3), respectively.

[0060] make , ,in The following formula is used to calculate: ; in, Represents the coefficient matrix; Represents the identity matrix; express The length of the mold, Its corresponding unit direction vector; express transpose; express An antisymmetric matrix.

[0061] The above method can also be used to solve the geometric consistency objective function.

[0062] Finally, by iteratively executing steps 1 to 4 above multiple times, the camera pose is estimated and updated iteratively.

[0063] Parallel optimization strategy: To improve the execution efficiency of the algorithm, a parallel optimization strategy was designed, such as... Figure 6 As shown, this method is divided into a parallel processing stage and a pose optimization and update stage. The parallel processing stage includes two parts: pose perturbation strategy and new viewpoint image rendering, and 2D-2D feature matching. This stage outputs the query image and... 2D-2D feature matching point pairs between rendered images. The pose optimization and update part includes two parts: pose estimation based on geometric consistency constraints and pose optimization based on epipolar geometry. The camera pose is optimized through the constructed objective function to achieve pose update. By iterating the above process, the optimal camera pose is obtained.

[0064] The technical effects of this application are further explained below.

[0065] The 7-Scenes dataset, proposed by Microsoft Research at the 2013 CVPR international conference, is an indoor scene dataset used to evaluate the performance of camera six-DOF pose measurement algorithms. This dataset includes seven indoor scenes, each containing 2000-12000 image sequences, and includes challenging scenarios such as weak textures and blurred regions. Therefore, this dataset is widely used for testing camera localization methods. The dataset uses the mean translation error metric and the mean rotation error metric as evaluation metrics to assess the camera pose parameters obtained by the algorithms.

[0066] Translation error measures the estimated value of the translation vector. and the true value Distance between: ; Rotation error measures the estimated value of the target's rotation. and the true value Differences between them: ; in, This indicates the computation of the trace of a matrix.

[0067] This method uses the pose calculation results from DFNet as initial values ​​and performs further optimization. The performance test results of this method are shown in Table 1. By comparing it with current mainstream methods such as DFNet, Marepo, FQN, CrossFire, NeFeS, HR-APR, and MCLoc, it can be seen that NeFeS and HR-APR methods both use the calculation results from DFNet as the initial pose. It is evident that this method performs better in seven scenarios. The average value of the indicator is 1.3 cm. The average value of the index is 0.4°, which is higher than other algorithms. The pose measurement results of this method are shown in Figure 7(a) and Figure 7(b). Each image is divided into two parts by a diagonal line. The lower left part represents the real image, and the upper right part represents the image rendered by the predicted pose.

[0068] Table 1. Test results of mainstream monocular localization methods and our proposed method on the 7-Scenes dataset. / (cm / °)

[0069] Among them, Chess represents a chess scene; Fire represents a fire hydrant scene; Heads represents an avatar scene; Office represents an office scene; Pumpkin represents a pumpkin scene; Redkitchen represents a red kitchen scene; and Stairs represents a staircase scene.

[0070] The effectiveness of the key modules proposed in this method is analyzed by removing different components. As shown in Table 2, the algorithm achieves optimal performance when all modules are used.

[0071] Table 2 compares the performance of different algorithms.

[0072] Based on the same inventive concept, this application also provides a camera pose visual measurement device for implementing the above-mentioned camera pose visual measurement method based on three-dimensional Gaussian splashing. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more camera pose visual measurement device embodiments provided below can be found in the limitations of the camera pose visual measurement method based on three-dimensional Gaussian splashing described above, and will not be repeated here.

[0073] In one exemplary embodiment, such as Figure 8 As shown, a camera pose visual measurement device based on three-dimensional Gaussian splashing is provided. The device consists of five parts: an image acquisition module, a storage module, a pose perturbation and rendering module, a matching module, and a pose optimization module.

[0074] The image acquisition module is used to drive the camera to acquire real-time scene images and output the image to be estimated as the query image.

[0075] The storage module is used to store the query image, a pre-built 3D Gaussian splash model of the current scene, and the current initial pose as the initial value for pose optimization; the current initial pose is provided by the calculation result of the previous frame in the video sequence, or by an external sensor / other pose estimation method.

[0076] The pose perturbation and rendering module is used to read the current initial pose and the 3D Gaussian splash model, and generate multiple perturbed camera poses by applying a preset perturbation strategy to the initial pose. Then, based on the 3D Gaussian splash model of the current scene, the initial pose and multiple perturbed poses are rendered respectively to generate multiple RGB rendered images of corresponding viewpoints, which constitute a set of rendered images for subsequent matching.

[0077] The matching module is used to perform 2D-2D image feature matching between each rendered image in the rendered image set and the query image to establish a pixel-level correspondence between the query image and each rendered image.

[0078] The pose optimization module is used to calculate the relative pose of the query image and the rendered image based on the pixel-level correspondence output by the matching module, construct an objective function, and iteratively optimize the initial pose with the goal of minimizing the objective function to obtain the final camera pose.

[0079] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores camera pose data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a camera pose visual measurement method based on three-dimensional Gaussian splashing.

[0080] Those skilled in the art will understand that Figure 9The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0081] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0082] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0083] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0084] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0085] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0086] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A camera pose visual measurement method based on three-dimensional Gaussian splashing, characterized in that, include: Acquire the RGB image of the target scene, the 3D Gaussian splash model, and the initial pose of the camera; Apply several perturbations to the initial pose to generate several perturbed poses; wherein, each perturbation generates one perturbed pose. Based on the 3D Gaussian splash model, the initial pose and each perturbation pose are rendered to obtain L+1 rendered images; L represents the number of perturbation poses. The RGB image of the target scene is matched with each rendered image using 2D-2D features to obtain L+1 sets of 2D-2D matching point pairs. Based on L+1 sets of 2D-2D matching point pairs, the relative pose between each rendered image and the RGB image of the target scene is determined; Based on the relative pose between each rendered image and the RGB image of the target scene, and L+1 sets of 2D-2D matching point pairs, the pose update value is determined with the goal of minimizing the objective function; the objective function includes a geometric consistency objective function and an objective function based on epipolar geometric constraints. Determine whether the convergence condition has been met; if not, determine the pose update value as the initial pose of the camera and return to the step "apply several perturbations to the initial pose to generate several perturbed poses"; if yes, determine the pose update value as the final camera pose.

2. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, Based on the relative pose between each rendered image and the RGB image of the target scene, the pose update value is determined with the objective function as the goal, specifically including: Based on the relative pose between each rendered image and the RGB image of the target scene, the first initial pose is determined with the goal of minimizing the geometric consistency objective function; Based on the first initial pose and L+1 sets of 2D-2D matching point pairs, the pose update value is determined with the objective of minimizing the objective function based on epipolar geometry constraints.

3. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 2, characterized in that, The mathematical expression for the geometric consistency objective function is: ； in, This represents the rotation matrix in the first initial pose of the camera to be estimated; the superscript -1 indicates the inverse. This represents the translation vector in the first initial pose of the camera to be estimated; Represents the RGB image of the target scene and the initial pose rendering image. The relative rotation matrix; Indicates the initial pose rendering image The rotation matrix; Represents the RGB image and the perturbed pose rendering image of the target scene. The relative rotation matrix; Represents perturbation pose rendering image The rotation matrix; Indicates the initial pose rendering image Translation vector; Represents perturbation pose rendering image Translation vector; Represents the RGB image of the target scene and the initial pose rendering image. The normalized translation vector; Represents the RGB image and the perturbed pose rendering image of the target scene. The normalized translation vector; Represents the normalized translation vector Scale factor; Represents the normalized translation vector The scale factor.

4. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 2, characterized in that, The mathematical expression for the objective function based on epipolar geometry constraints is: ； ； ； in, This represents the pose update value to be estimated; The initial value is the first initial pose; This indicates the number of feature points in the RGB image of the target scene; Indicates the initial pose rendering image The number of feature points in the middle; Represents perturbation pose rendering image The number of feature points in the middle; Represents feature points in the RGB image of the target scene With corresponding mapping point The distance between the polar lines and the mapping point The polar line is based on the mapping point. The perturbation pose of the rendered image and the pose update value to be estimated. Certain; for or ;when for At that time, the mapping point Rendering images for initial pose Zhongyu Matching feature points, and ;when for At that time, the mapping point Rendering images for perturbed poses Zhongyu Matching feature points, and ; express or .

5. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The objective function based on epipolar geometry constraints is solved using the Levenberg-Marquardt algorithm.

6. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, Before using the Levenberg-Marquardt algorithm to solve the objective function based on epipolar geometry constraints, the pose update value to be estimated by the camera is expressed in Lie algebra form, and then the pose update value to be estimated is solved based on the Lie algebra form.

7. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, Apply several perturbations to the initial pose to generate several perturbed poses, specifically including: Apply several Gaussian noise perturbations to the initial pose to generate several hypothetical poses; Apply rotational and translational perturbations to each assumed pose to obtain the perturbation pose corresponding to each assumed pose.

8. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, Based on a 3D Gaussian splash model, the initial pose and each perturbation pose are rendered separately to obtain L+1 rendered images, specifically including: Based on the three-dimensional Gaussian splash model, parallel processing is used to render the initial pose and each perturbation pose simultaneously to obtain L+1 rendered images.

9. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The RGB image of the target scene is matched with each rendered image using 2D-2D features to obtain L+1 sets of 2D-2D matching point pairs, specifically including: By using parallel processing, the RGB image of the target scene is simultaneously matched with each rendered image using 2D-2D features to obtain L+1 sets of 2D-2D matching point pairs.

10. The camera pose visual measurement method based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The convergence condition is that the number of iterations reaches a preset number.

Images