Three-dimensional Gaussian sputtering training method and system based on hierarchical common-view relationship
By introducing hierarchical common-visual relationships and weight matrix in 3D Gaussian sputtering training, the three-dimensional skeleton reconstruction results are optimized, and the problem of floating noise in 3D point clouds is solved, and the accuracy and consistency of 3D Gaussian reconstruction is improved.
Patent Information
- Application Number
- CN202411012629.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-10
AI Technical Summary
Due to the lack of consistency verification during the optimization process, the existing three-dimensional Gaussian sputtering training methods have inaccurate positions between multiple viewing angles, which easily generates floating noise, affecting the training quality of the three-dimensional point cloud.
Using a method based on hierarchical common viewing relationship, by acquiring multiple common viewing camera groups and calculating the corresponding weight matrix, multi-level three-dimensional Gaussian sputtering optimization is performed on the three-dimensional framework reconstruction results based on the common viewing image and weight matrix to eliminate floating noise.
The common-view constraints make the position and attribute estimation of three-dimensional points more accurate, reducing the generation of floating noise, and improving the quality and consistency of three-dimensional Gaussian reconstruction.
Smart Images
Figure CN119107428A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of augmented reality and mixed reality, and in particular to a three-dimensional Gaussian sputtering training method, system, computer device and computer-readable storage medium based on hierarchical common view relationship. Background Art
[0002] 3D gaussian splatting (3DGS) uses the explicit expression of 3D Gaussian. With its many advantages such as fast training speed and strong editability, it has quickly become the preferred technology for high-fidelity 3D reconstruction.
[0003] In the related art, the conventional 3DGS training method in this field uses the stochastic gradient descent method to optimize the training by inputting the known camera intrinsic parameters and posture, as well as the corresponding three-dimensional Gaussian point cloud. However, due to the lack of consistency verification of some three-dimensional points between multiple perspectives during the optimization process of this type of method, the position or matching relationship of these points in the current perspective is inaccurate, which leads to ambiguity in the optimized target during the optimization process, and it is easy to generate floating noise points in the air, affecting the training quality of the three-dimensional point cloud.
[0004] At present, no effective solution has been proposed for the problem of floating noise easily generated in pure visual 3DGS training in related technologies. Summary of the invention
[0005] The embodiments of the present application provide a three-dimensional Gaussian sputtering training method, system, computer device and computer-readable storage medium based on hierarchical common view relationship, so as to at least solve the problem of floating noise points that are easily generated in 3DGS training in related technologies.
[0006] In a first aspect, an embodiment of the present application provides a three-dimensional Gaussian sputtering training method based on a hierarchical common view relationship, which is used to perform three-dimensional Gaussian sputtering optimization training based on the three-dimensional skeleton reconstruction result after obtaining the three-dimensional skeleton reconstruction result, and the method includes:
[0007] Based on the 3D skeleton reconstruction result, multiple common-view camera groups are obtained, wherein any common-view camera group includes multiple camera models having common-view 3D points;
[0008] Obtaining a weight matrix corresponding to each common viewing camera group according to the viewing cone triangle projected onto the two-dimensional plane by the camera model;
[0009] Based on the common view image corresponding to the common view camera group and the weight matrix, a three-dimensional Gaussian sputtering optimization of multiple levels is performed on the three-dimensional skeleton reconstruction result to obtain a three-dimensional Gaussian model.
[0010] In some embodiments, acquiring multiple common-view camera groups based on the three-dimensional skeleton reconstruction result includes:
[0011] Based on the three-dimensional skeleton reconstruction result, a matching relationship between the 2D points in each feature image and the 3D points in the real space is obtained;
[0012] According to the matching relationship, all camera models that have common observation with any 3D point are acquired to form an initial common viewing camera group, wherein any initial common viewing camera group includes at least two camera models that have common viewing 3D points;
[0013] The initial common view camera group is screened based on a preset screening rule to obtain the common view camera group.
[0014] In some embodiments, the initial common view camera group is filtered based on a preset screening rule, and the common view camera group includes:
[0015] Performing a preliminary screening of the initial common-view camera group according to the number of common-view 3D images of camera pairs consisting of any two camera models in the initial common-view camera group;
[0016] In the common view camera group after the initial screening, all target common view models are obtained according to the 3D point intersection-union ratio between each camera pair to form the common view camera group.
[0017] In some embodiments, according to the projection of the camera model onto the viewing cone triangle of the two-dimensional plane, the weight matrix corresponding to each common viewing camera group is obtained, including:
[0018] The cone triangle is obtained by projecting the three-dimensional cone of each camera model in the common viewing camera group onto a two-dimensional plane and intercepting the projected two-dimensional plane according to the median of the depth value of the 3D point in the camera model;
[0019] In the camera common view group, the weight of each camera pair is obtained according to the intersection-and-union ratio of the areas of the viewing cone triangles between any two camera models;
[0020] The weights of all camera pairs in any camera common view group are combined to obtain a weight coefficient of the camera common view group.
[0021] In some embodiments, based on the common view camera group and the weight matrix, performing multiple levels of three-dimensional Gaussian sputtering optimization on the three-dimensional skeleton reconstruction result includes:
[0022] Based on all the image matching pairs consisting of all two co-viewing images in the co-viewing camera group, a first round of training is performed on the three-dimensional skeleton reconstruction result,
[0023] After the first round of training, the skeleton model is iteratively trained by gradually increasing the number of co-visual images in the image matching pairs during the training process until the number of iterations reaches a preset threshold, thereby obtaining the three-dimensional Gaussian model;
[0024] The loss function of the training process is used to minimize the error of the three-dimensional point in the co-viewing image, and the loss function is obtained by weighted calculation based on the weight matrix.
[0025] In some of the embodiments, when the total number of feature images in the three-dimensional skeleton reconstruction result is less than a preset threshold and the image resolution is less than a preset resolution, the method further includes:
[0026] After obtaining the three-dimensional Gaussian model;
[0027] Obtain the feature images corresponding to all camera models in all 3D skeleton reconstruction results;
[0028] Taking all the co-viewing images in the co-viewing camera group as constraint conditions and based on the feature images, the three-dimensional Gaussian model is subjected to multiple full-image iterative optimizations to obtain an optimized three-dimensional Gaussian model.
[0029] In a second aspect, the present embodiment further provides a three-dimensional Gaussian sputtering training system based on a hierarchical common view relationship, which is used to perform three-dimensional Gaussian sputtering optimization training based on the three-dimensional skeleton reconstruction result after obtaining the three-dimensional skeleton reconstruction result. The system includes: a preprocessing module and an optimization training module, wherein:
[0030] The preprocessing module is used to obtain multiple common-view camera groups based on the three-dimensional skeleton reconstruction result, wherein any common-view camera group includes multiple camera models with common-view 3D points;
[0031] And, according to the viewing cone triangle projected onto the two-dimensional plane by the camera model, a weight matrix corresponding to each common viewing camera group is obtained;
[0032] The optimization training module is used to perform multiple levels of three-dimensional Gaussian sputtering optimization on the three-dimensional skeleton reconstruction result based on the common view image corresponding to the common view camera group and the weight matrix to obtain a three-dimensional Gaussian model.
[0033] In some embodiments, the system further includes a full-image optimization module, and the full-image optimization module is used to:
[0034] When the total number of feature images in the three-dimensional skeleton reconstruction result is less than a preset threshold and the image resolution is less than a preset resolution, after obtaining the three-dimensional Gaussian model;
[0035] Obtain the feature images corresponding to all camera models in all 3D skeleton reconstruction results;
[0036] Taking all the co-viewing images in the co-viewing camera group as constraint conditions and based on the feature images, the three-dimensional Gaussian model is subjected to multiple full-image iterative optimizations to obtain an optimized three-dimensional Gaussian model.
[0037] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.
[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.
[0039] Compared with the related art, the embodiment of the present application provides a 3D Gaussian sputtering training method based on hierarchical common view relationship, which obtains multiple common view camera groups based on the reconstruction results of the 3D skeleton, wherein any common view camera group includes multiple camera models with common view 3D points; according to the cone triangle projected onto the 2D plane by the camera model, the weight matrix corresponding to each common view camera group is obtained; based on the common view image and weight matrix corresponding to the common view camera group, the 3D skeleton reconstruction results are optimized by multiple levels of 3D Gaussian sputtering to obtain a 3D Gaussian model, and the position and attribute estimation of the 3D point is more accurate due to the use of common view information from multiple perspectives. The common view constraint makes the reconstruction results of the same 3D point more consistent under different perspectives, reducing the generation of floating noise points. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0041] Figure 1 is a schematic top view of a three-dimensional Gaussian point cloud model according to an embodiment of the present application;
[0042] Figure 2 It is a schematic diagram of a three-dimensional Gaussian point cloud training result in the related art;
[0043] Figure 3 is a flow chart of a three-dimensional Gaussian sputtering training method based on hierarchical common view relationship according to an embodiment of the present application;
[0044] Figure 4 is a schematic diagram of the pose corresponding to the captured image;
[0045] Figure 5 Schematic diagram of the matching relationship between 2D points in the image and 3D points in the real scene;
[0046] Figure 6 is a schematic diagram of a camera co-viewing relationship based on 3D points according to an embodiment of the present application;
[0047] Figure 7 It is a schematic diagram of the common viewing relationship of cameras based on the viewing cone;
[0048] Figure 8 is a schematic diagram of a viewing cone triangle obtained after interception according to the median depth according to an embodiment of the present application;
[0049] Fig. 9 is a schematic diagram of a viewing cone triangle of a camera common viewing group according to an embodiment of the present application;
[0050] Fig.10 It is a schematic diagram of a surrounding collection path and an arc collection path;
[0051] Fig.11 It is a structural block diagram of a three-dimensional Gaussian sputtering training system based on hierarchical common view relationship according to an embodiment of the present application;
[0052] Fig.12 is a structural block diagram of another three-dimensional Gaussian sputtering training system based on hierarchical common view relationship according to an embodiment of the present application;
[0053] Fig.13 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0055] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0056] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0057] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0058] 3D gaussian splatting (3DGS) uses the explicit expression of 3D Gaussian and has quickly become the preferred technology for high-fidelity 3D reconstruction due to its many advantages such as fast training speed and strong editability. Figure 1 It is a top view schematic diagram of a three-dimensional Gaussian point cloud model according to an embodiment of the present application. The three-dimensional Gaussian point cloud model includes: three-dimensional position, covariance matrix (expressed by rotation matrix (quaternion) and numerical expression of three axes), opacity, color and spherical harmonics and other attributes, among which spherical harmonics are optional attributes. The three-dimensional Gaussian point cloud model includes multiple 3D points, and each 3D point carries multiple attributes. Multiple such 3D points constitute a three-dimensional Gaussian point cloud model.
[0059] The existing 3DGS training inputs known camera intrinsic parameters and poses, as well as the corresponding 3D point cloud, and uses the stochastic gradient descent method for optimization to improve the accuracy and quality of the 3D reconstruction model; however, this type of method is prone to generating floating noise in the air, and this problem has not yet been solved in pure vision 3DGS training.
[0060] The inventor of this case learned through analyzing the prior art that the mainstream 3D Gaussian sputtering usually uses only one picture for each forward reasoning and gradient back propagation during training, which ignores the important common view relationship in the field of 3D reconstruction (the common view relationship means that a 3D point in the physical space is photographed by multiple cameras, and the series of pictures taken are called a group of common view pictures), which easily generates floating noise points. Figure 2 is a schematic diagram of a three-dimensional Gaussian point cloud training result in the related technology, such as Figure 2 As shown, points 10, 11, 12, and 13 are noise points floating in the air that often appear in pure visual 3DGS training. This patent can remove these noise points through multi-level common vision training.
[0061] In view of this, the embodiment of the present application provides a three-dimensional Gaussian sputtering training method based on hierarchical common view relationship, by introducing common view relationship in the three-dimensional Gaussian sputtering training, generating more constraints of different levels for the three-dimensional Gaussian sputtering points obtained by training, thereby improving the quality of three-dimensional Gaussian reconstruction. Figure 3 is a flowchart of a three-dimensional Gaussian sputtering training method based on hierarchical common view relationship according to an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps:
[0062] S301, based on the three-dimensional skeleton reconstruction result, obtaining multiple common-view camera groups, wherein any common-view camera group includes multiple camera models having common-view 3D points;
[0063] It should be noted that this step and subsequent steps are implemented after obtaining the 3D skeleton reconstruction result through the motion recovery structure (SFM (Structure from Motion, SFM), and the 3D skeleton reconstruction result includes: the captured image, the pose corresponding to the captured image, the matching relationship between the 2D points in the captured image and the 3D points in the real space, and the camera parameters after dedistortion; Figure 4 Yes and Figure 5 They are respectively a schematic diagram of the pose corresponding to the captured image, and a schematic diagram of the matching relationship between the 2D points in the image and the 3D points in the real scene;
[0064] Furthermore, the 3D skeleton reconstruction result includes the matching relationship between the 2D points in each image and the 3D points in the real space; Figure 6 is a schematic diagram of a camera co-viewing relationship based on 3D points according to an embodiment of the present application, such as Figure 6As shown, according to this matching relationship, all camera models that have a common view relationship with any 3D point can be obtained by traversing all 3D points in the three-dimensional skeleton model.
[0065] In this embodiment, the set of camera models that have a common view relationship is defined as a common view camera group. Optionally, the common view camera group can be recorded as: F_j = {c_j^1, c_j^2, ...., c_j^n}, one F_j is called a camera common view group, and a total of N_t common view camera groups are obtained.
[0066] Furthermore, considering that the above common view camera group may contain noise, errors or inconsistent data, in this step, based on preset screening rules, a common view camera group that meets the requirements is obtained from the initial common view camera group obtained in the above step, specifically:
[0067] The screening can be performed based on the number of common view 3D: for any camera common view group F_j among the N_t common view camera groups, take any two cameras in the camera group and calculate the number of common view 3D points. If the common view camera group satisfies: the number of common view 3D points of any two cameras is greater than the preset point threshold w, then the camera group is retained, otherwise the camera group is discarded. Optionally, in this embodiment, the preset point threshold w can be set to 15;
[0068] After the initial screening, by setting the point threshold, we can measure the degree of overlap of the viewing angles between cameras within the camera group and their ability to jointly observe the same part of the scene, filter out those camera groups generated due to accidental factors or wrong matching, and obtain N_w camera co-viewing groups.
[0069] Furthermore, further screening can be performed based on the intersection-and-union ratio of 3D points in any two camera pairs with a co-viewing relationship. Specifically, the intersection-and-union ratio of each pair of N_w co-viewing camera groups is calculated, that is, (the number of cameras in the intersection of the two camera groups) / (the number of cameras in the union of the two camera groups). If the intersection-and-union ratio of the two camera groups is greater than a preset threshold e, the one with the smaller number of cameras in the two camera groups is discarded; optionally, in this embodiment, e is set to 0.6.
[0070] After the above steps, the intersection over union ratio is set to measure the similarity and overlap between camera groups. By discarding camera groups that have a high overlap with other camera groups (i.e., IoU is greater than a certain threshold) and have a small number of cameras, redundant information in the data can be reduced, which helps avoid repeated calculations or conflicting results in subsequent processing.
[0071] Through the above step S301, the camera common view group that meets specific conditions can be effectively screened out from the 3D points of the skeleton model and the camera observations, providing an accurate and effective data basis for the subsequent training of the three-dimensional Gaussian sputtering model.
[0072] S302, obtaining a weight matrix corresponding to each common viewing camera group according to the viewing cone triangle of the camera model viewing cone;
[0073] Figure 7 It is a schematic diagram of the camera co-viewing relationship based on the viewing cone, such as Figure 7 As shown, by calculating the view cone triangles of each camera model in the common view camera group after screening, and according to the ratio of the intersection area and the union area of the view cone triangles of any two cameras, the weight of the camera in the optimization process is obtained. Specifically, the following steps are included:
[0074] Step 1, obtain the viewing cone triangle of each camera;
[0075] In an exemplary embodiment, there are N1 3D points in the three-dimensional model, and N1 camera common view groups can be obtained. Figure 7 As shown, all three-dimensional camera cones are projected onto the xy (z=0) plane to obtain the projection result, and further the median depth of the z coordinates (depth) of all 3D points in the camera is taken, and the cone triangle is intercepted at the depth; Figure 8 It is a schematic diagram of a viewing cone triangle obtained after being intercepted according to the median depth according to an embodiment of the present application.
[0076] Step 2, in the camera common view group, the weight of each camera pair is obtained according to the intersection-and-union ratio of the viewing cone triangles between any two camera models;
[0077] Fig. 9 is a schematic diagram of a viewing cone triangle of a camera common viewing group according to an embodiment of the present application, such as Fig. 9 As shown, for each camera common view group F_j, where the depth of the view cone triangle corresponding to each camera is D_j = {d_1, d_2, ..., d_n}, select any two cameras from n cameras, totaling C_n^2. Further, calculate the intersection area and union area of the two view cone triangles, and calculate the intersection-union ratio IOU = intersection area / union area as the weight of the two cameras;
[0078] Finally, for any common view camera group, the weight matrix m_j shown in Table 1 below is obtained:
[0079] C_1 C_2 …… C_3 C_1 W_11 W_12 …… W_1n C_2 W_12 W_22 …… W_2n …… …… …… …… …… C_3 W_1n W_2n …… W_nn
[0080] Table 1: Weight matrix of camera co-viewing group
[0081] The weight matrix is calculated for the N1 camera common view groups to obtain the weight matrix M = {m_1, m_2, ..., m_{N1}} corresponding to each common view camera group.
[0082] S303, based on the common view camera group and the weight matrix, perform multiple levels of three-dimensional Gaussian sputtering optimization on the three-dimensional skeleton reconstruction result to obtain a three-dimensional Gaussian model.
[0083] In this field, 3D Gaussian Splatting training is mainly a process of 3D reconstruction using a series of images. Training specifically refers to the process of optimizing the 3D point cloud so that the reconstructed 3D scene is more consistent and accurate with the real world. The result of training is a higher quality and more accurate 3D Gaussian point cloud model.
[0084] During the training of mainstream 3D Gaussian sputtering, only one image is usually used for each forward inference and gradient back propagation, which ignores the important common view relationship in the field of 3D reconstruction and leads to the problem of floating noise.
[0085] In the related art, the optimization training process of 3D Gaussian sputtering point cloud includes the following steps:
[0086] a. The input data includes:
[0087] 1) Camera intrinsics and extrinsics: Camera intrinsics (such as focal length, principal point coordinates, etc.) and extrinsics (camera position and posture in the world coordinate system);
[0088] 2) Co-view images corresponding to the co-view camera group: images taken from multiple perspectives, which contain different perspective information of the same scene.
[0089] 3) Initial 3D point cloud: a preliminarily generated 3D point cloud, which represents some preliminary 3D positions in the scene. In this embodiment, the initial 3D point cloud can be obtained from the 3D skeleton reconstruction result.
[0090] b. Further, using the input camera parameters and the co-viewed image, the three-dimensional points are projected onto the image plane through the projection model, the matching errors between these projected points and the pixels in the image are calculated, and the loss function is further determined;
[0091] c. Finally, the gradient of the loss function relative to the 3D point coordinates, camera parameters, etc. is calculated through the back-propagation algorithm, and the position, color and other parameters of the 3D point are updated using the stochastic gradient descent method, so that the loss function gradually decreases and the accuracy of 3D reconstruction gradually improves.
[0092] In this embodiment, during the 3DGS training process, multiple images with common view are added to perform the stochastic gradient descent method together, and with the assistance of the common view relationship, multiple levels of optimization training are performed to remove the above-mentioned floating noise.
[0093] In this embodiment, the introduction of the common-view relationship can significantly improve the quality of the above three-dimensional reconstruction. For each common-view group, the three-dimensional points are projected onto all common-view images, and the reprojection error at each viewing angle is calculated.
[0094] When calculating the loss function, multiple levels of co-viewing images are introduced, requiring that the matching error loss function of a certain 3D point in multiple co-viewing images be continuously iterated to be as small as possible, and the loss function is additionally multiplied by the weight parameters in the above-mentioned weight matrix to constrain the optimization of the 3D point from multiple perspectives, and finally obtain a highly reliable result.
[0095] Furthermore, by combining the gradient information of multiple perspectives for back propagation, the update direction of the 3D point not only considers the error of a single perspective, but also takes into account the errors of all common perspectives, thereby adjusting the position of the 3D point more accurately.
[0096] Specifically, in an exemplary embodiment, the exemplary steps of the optimization training process using the solution of the present application are as follows:
[0097] Step 1: Based on the image matching pairs consisting of all two co-viewing images in the co-viewing camera group, the first round of training is performed on the 3D skeleton reconstruction results;
[0098] In the first iteration, a camera common view group F_j is used. Assuming that x images are used for training at a certain moment, x images are randomly selected from the common view camera group for training, with a total of C_n^x permutations and combinations. It should be noted that when calculating the loss function, an additional weight is multiplied, and the weight is obtained from the m_{j} matrix;
[0099] For example, if {c_1,c_2,...,c_x} images are selected, then the weight of the c_1 image is the mean of the m_{j} matrix (m_{j}[1][2],m_{j}[1][3],...,m_{j}[1][x]), where m_{j}[1][2] means the number in the first row and second column of the matrix, m_{j}[1][2] means the number in the first row and second column of the matrix, and so on.
[0100] Step 2, after the first round of training, the skeleton model is iteratively trained by gradually increasing the number of co-viewed images in the image matching pair during the training process until the number of iterations reaches a preset threshold, at which time a three-dimensional Gaussian model is obtained; wherein the loss function of the training process is used to minimize the error of the three-dimensional points in the co-viewed images, and the loss function is determined based on the preset loss and weight matrix.
[0101] As the number of training iterations increases, gradually increase the number of images trained at one time in each iteration. Gradually increase from 2 to N. Specifically, there are T iterations in total, set the normalization factor q = (2*3*4*....*N / (2+3+4+...+N)), the number of iterations using 2 images is T*(q / 2), the number of iterations using 3 images is T*(q / 3)... to T*(q / N).
[0102] In the method of hierarchical training based on co-viewing images provided in this embodiment, the co-viewing images gradually increase from 2 to N, which can ensure the robustness of model training and steadily improve the effect of model training. Optionally, matching points are added through multi-view stereo matching, because the increase in matching points means more "point" co-viewing relationships. In addition, multiple similar or identical perspectives can be selected for repeated training to further improve the optimization effect.
[0103] In addition, it should be noted that the above steps have achieved the purpose of using the common view relationship as auxiliary information to improve the training effect of 3D Gaussian point cloud; however, considering that the total number of feature images in the 3D skeleton reconstruction result is less than the preset threshold and the image resolution is less than the preset resolution, after the optimization is completed through the above steps, the optimization effect may still be limited, and there are still many noise points in the reconstruction result.
[0104] For the above problem, the present embodiment also provides a full-image optimization step. After obtaining the three-dimensional Gaussian model, the feature images corresponding to all camera models in all three-dimensional skeleton reconstruction results are obtained; with all co-viewing images in the co-viewing camera group as constraints, multiple full-image iterative optimizations are performed on the three-dimensional Gaussian model based on all feature images to obtain the optimized three-dimensional Gaussian model.
[0105] Optionally, the preset threshold of the total number of images can be 100, and the preset image resolution can be 640*480; in addition, there are also restrictions on the acquisition method for the initial mapping image. Fig.10 is a schematic diagram of a surrounding collection path and an arc collection path. Specifically, it can be limited to adopting Fig.10 The path shown collects the original mapping image.
[0106] Through the above steps S101 to S103, in the process of optimizing and training the 3D Gaussian point cloud model, the position and attribute estimation of the model's three-dimensional points are more accurate due to the use of multi-view common-view information. The common-view constraint makes the reconstruction results of the same three-dimensional point under different viewpoints more consistent, reducing the generation of floating noise. In addition, in order to solve the problem of a small number of images and poor resolution, a full-image optimization process is also provided to further optimize the consistency and accuracy of the three-dimensional Gaussian source.
[0107] This embodiment also provides a three-dimensional Gaussian sputtering training system based on hierarchical common view relationship, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.
[0108] The system is used to perform 3D Gaussian sputtering optimization training based on the 3D skeleton reconstruction results after obtaining the 3D skeleton reconstruction results. Fig.11 is a structural block diagram of a three-dimensional Gaussian sputtering training system based on hierarchical common view relationship according to an embodiment of the present application, such as Fig.11 As shown, the system includes a preprocessing module 10 and an optimization training module 20, wherein:
[0109] The preprocessing module 10 is used to obtain multiple common view camera groups based on the three-dimensional skeleton reconstruction result, wherein any common view camera group includes multiple camera models with common view 3D points; and obtain a weight matrix corresponding to each common view camera group according to the view cone triangle projected onto the two-dimensional plane by the camera model;
[0110] The optimization training module 20 is used to perform multiple levels of three-dimensional Gaussian sputtering optimization on the three-dimensional skeleton reconstruction result based on the common view image and weight matrix corresponding to the common view camera group to obtain a three-dimensional Gaussian model.
[0111] Through the above system, in the process of optimizing and training the 3D Gaussian point cloud model, the position and attribute estimation of the model's three-dimensional points are more accurate due to the use of multi-view common view information. The common view constraint makes the reconstruction results of the same three-dimensional point under different view angles more consistent, reducing the generation of floating noise. In addition, in order to solve the problem of small number of images and poor resolution, a full-image optimization process is also provided to further optimize the consistency and accuracy of the three-dimensional Gaussian power.
[0112] also, Fig.12 This is a structural block diagram of another three-dimensional Gaussian sputtering training system based on hierarchical common view relationship according to an embodiment of the present application. Through this system, 3DGS optimization training is performed by adopting the hierarchical common view relationship, and the noise points floating in the air can be eliminated during the training process. The geometric degradation and geometric inconsistency of the final result of the model can be removed during the training process.
[0113] In one embodiment, Fig.13 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Fig.13 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Fig.13As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected through an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system, the computer program is executed by the processor to implement a three-dimensional Gaussian sputtering training method based on hierarchical common view relationship in augmented reality, and the database is used to store data.
[0114] Those skilled in the art will understand that Fig.13 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0116] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A three-dimensional Gaussian sputtering training method based on hierarchical common view relationship, characterized in that: After obtaining a three-dimensional skeleton reconstruction result, performing three-dimensional Gaussian sputtering optimization training based on the three-dimensional skeleton reconstruction result, the method comprises: Based on the 3D skeleton reconstruction result, multiple common-view camera groups are obtained, wherein any common-view camera group includes multiple camera models having common-view 3D points; Obtaining a weight matrix corresponding to each common viewing camera group according to the viewing cone triangle projected onto the two-dimensional plane by the camera model; Based on the common view image corresponding to the common view camera group and the weight matrix, a three-dimensional Gaussian sputtering optimization of multiple levels is performed on the three-dimensional skeleton reconstruction result to obtain a three-dimensional Gaussian model.
2. The method according to claim 1, characterized in that: The obtaining of multiple common-view camera groups based on the three-dimensional skeleton reconstruction result includes: Based on the three-dimensional skeleton reconstruction result, a matching relationship between the 2D points in each feature image and the 3D points in the real space is obtained; According to the matching relationship, all camera models that have common observation with any 3D point are acquired to form an initial common viewing camera group, wherein any initial common viewing camera group includes at least two camera models that have common viewing 3D points; The initial common view camera group is screened based on a preset screening rule to obtain the common view camera group.
3. The method according to claim 2, characterized in that Based on the preset screening rules, the initial common view camera group is filtered to obtain the common view camera group including: Performing a preliminary screening of the initial common-view camera group according to the number of common-view 3D images of camera pairs consisting of any two camera models in the initial common-view camera group; In the common view camera group after the initial screening, all target common view models are obtained according to the 3D point intersection-union ratio between each camera pair to form the common view camera group.
4. The method according to any one of claims 1 to 3, characterized in that According to the projection of the camera model onto the cone triangle of the two-dimensional plane, the weight matrix corresponding to each common viewing camera group is obtained, including: The cone triangle is obtained by projecting the three-dimensional cone of each camera model in the common viewing camera group onto a two-dimensional plane and intercepting the projected two-dimensional plane according to the median of the depth value of the 3D point in the camera model; In the camera common view group, the weight of each camera pair is obtained according to the intersection-and-union ratio of the areas of the viewing cone triangles between any two camera models; The weights of all camera pairs in any camera common view group are combined to obtain a weight coefficient of the camera common view group.
5. The method according to claim 1, characterized in that Based on the common view camera group and the weight matrix, performing multiple levels of three-dimensional Gaussian sputtering optimization on the three-dimensional skeleton reconstruction result includes: Based on all the image matching pairs consisting of all two co-viewing images in the co-viewing camera group, a first round of training is performed on the three-dimensional skeleton reconstruction result, After the first round of training, the skeleton model is iteratively trained by gradually increasing the number of co-visual images in the image matching pairs during the training process until the number of iterations reaches a preset threshold, thereby obtaining the three-dimensional Gaussian model; The loss function of the training process is used to minimize the error of the three-dimensional point in the co-viewing image, and the loss function is obtained by weighted calculation based on the weight matrix.
6. The method according to claim 1, characterized in that When the total number of feature images in the three-dimensional skeleton reconstruction result is less than a preset threshold and the image resolution is less than a preset resolution, the method further includes: After obtaining the three-dimensional Gaussian model; Obtain the feature images corresponding to all camera models in all 3D skeleton reconstruction results; Taking all the co-viewing images in the co-viewing camera group as constraint conditions and based on the feature images, the three-dimensional Gaussian model is subjected to multiple full-image iterative optimizations to obtain an optimized three-dimensional Gaussian model.
7. A three-dimensional Gaussian sputtering training system based on hierarchical common view relationship, characterized in that: After obtaining the three-dimensional skeleton reconstruction result, the system performs three-dimensional Gaussian sputtering optimization training based on the three-dimensional skeleton reconstruction result, and the system includes: a preprocessing module and an optimization training module, wherein: The preprocessing module is used to obtain multiple common-view camera groups based on the three-dimensional skeleton reconstruction result, wherein any common-view camera group includes multiple camera models with common-view 3D points; And, according to the viewing cone triangle projected onto the two-dimensional plane by the camera model, a weight matrix corresponding to each common viewing camera group is obtained; The optimization training module is used to perform multiple levels of three-dimensional Gaussian sputtering optimization on the three-dimensional skeleton reconstruction result based on the common view image corresponding to the common view camera group and the weight matrix to obtain a three-dimensional Gaussian model.
8. The system according to claim 7, characterized in that The system further includes a full-image optimization module, which is used to: When the total number of feature images in the three-dimensional skeleton reconstruction result is less than a preset threshold and the image resolution is less than a preset resolution, after obtaining the three-dimensional Gaussian model; Obtain the feature images corresponding to all camera models in all 3D skeleton reconstruction results; Taking all the co-viewing images in the co-viewing camera group as constraint conditions and based on the feature images, the three-dimensional Gaussian model is subjected to multiple full-image iterative optimizations to obtain an optimized three-dimensional Gaussian model.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.