Multi-view image reconstruction method and device based on three-dimensional gaussian sputtering characterization

Abstract

The application provides a multi-view image reconstruction method and device based on a three-dimensional Gaussian sputtering representation, which comprises the following steps: collecting multi-view images of a target object to be reconstructed, the multi-view images being shooting images of the target object under multiple views, and each image having corresponding pose information; training a three-dimensional Gaussian sputtering model by single-view rendering, Loss calculation, gradient back propagation and model densification according to the multi-view images and the pose information, so as to obtain a final Gaussian graph element; labeling each part of the final Gaussian graph element, and reconstructing the final Gaussian graph element into a grid body model by using a rendering mode corresponding to the labeling, and generating a rendered depth image and an RGB image. By collecting multi-view depth rendering results, a better grid model can be obtained, and the model can be further adjusted by using a three-dimensional modeling software, so that the model can be conveniently integrated into an industrial three-dimensional reconstruction process, the reconstruction operation cycle is shortened, and the reconstruction quality is improved.

Description

Technical Field

[0001] The present invention relates to three-dimensional reconstruction technology, and in particular to a multi-view image reconstruction method, device, electronic device, computer-readable storage medium and computer program product based on three-dimensional Gaussian sputtering characterization. Background Art

[0002] 3D Gaussian splatting (3DGS) is a 3D reconstruction method based on 3D Gaussian probability distributions. Its main components include a differentiable renderer, a visualization tool, and a reconstruction module. The differentiable renderer performs rasterization on 3D Gaussian data, providing mapping indices between pixels and Gaussian primitives and implementing pixel gradient backpropagation. The visualization module provides a functional interface for interactively switching perspectives and rendering modes. The reconstruction module evaluates the differences between 3D Gaussian data and a reference image and, in conjunction with the differentiable renderer, gradually optimizes the 3D Gaussian model.

[0003] The Gaussian primitives used in the 3D Gaussian reconstruction method are defined by 3D pose (position, rotation, and scale) and color attributes. Each primitive has 58 optimizable attributes (one for transparency, three each for position, rotation, and scale, and RGB color attributes defined by third-order spherical harmonics, for a total of (3+1)2*3=48 attributes). The differentiable renderer calculates the screen coordinates of each Gaussian primitive's center, the projected depth in the line of sight, the sampled color in the direction of realization, and a linear approximation of the nonlinear perspective transformation at the primitive center, given viewport information (including camera pose, viewport size, and clipping distance). This linear approximation is applied to the Gaussian primitive to obtain its projected shape on the imaging plane (the resulting projection is a 2D Gaussian distribution). The corresponding index of the screen pixel and the Gaussian primitive is then recorded based on the 2D projected shape. All primitives at each screen pixel are sorted from nearest to farthest according to their projected depth. Finally, at each pixel, the RGB colors of the primitive are blended using the probability of each primitive's corresponding 2D Gaussian as transparency to produce the final rendered result. RGB color is the direct optimization target, and parameter gradients are propagated backwards along two paths: RGB → spherical harmonic color, and RGB → transparency → 2D Gaussian distribution covariance → projective approximation transformation → 3D pose, enabling supervised training of the Gaussian model. Furthermore, the reconstruction module includes adaptive splitting / cloning capabilities. For primitives with large transparency gradients, multiple small primitives are sampled from primitives with scaling factors greater than a threshold based on their 3D probability distribution, or multiple small primitives are cloned from a primitive with scaling factors less than a threshold.

[0004] The 3D Gaussian reconstruction method has achieved excellent visual effects in both indoor and outdoor scene reconstruction tasks. Image quality indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) all exceed those of previous multi-view reconstruction methods. However, the results of 3D Gaussian reconstruction still have deficiencies in multi-view consistency, the uniformity of Gaussian primitive distribution, and the accuracy of surface fitting. It cannot output a mesh model that is topologically consistent with the reconstructed target and cannot meet the needs of actual scene reconstruction work. At the same time, its differentiable rendering pipeline lacks gradient transfer paths for depth and normals, making it difficult to provide effective supervision information for the reconstruction optimization process. In addition, existing 3D Gaussian visualization tools only contain 3D Gaussian sputtering results and 3D Gaussian primitive visualization results, which is not convenient for intuitively evaluating the surface reconstruction quality of the reconstructed model. In addition, the visualization tool cannot directly interact with the Gaussian primitives, making it impossible to directly edit and adjust the reconstruction results.

[0005] Specifically, the reconstruction module of the 3DGS method only uses RGB images as supervision information. At the same time, the optimization freedom of the three-dimensional Gaussian primitives is large, the model is in an under-constrained state, and the optimization process inevitably falls into an overfitting state. Its specific manifestations are: the distribution of three-dimensional Gaussian primitives in the model is disordered, there are a large number of disordered interlacings between primitives, the morphological distribution cannot reflect the surface characteristics of the model, the visual effect of the local surface may be pieced together by multiple parts that are far away, and it only has a good visual effect under a specific viewing angle. The overall appearance does not match the actual situation of the model. Summary of the Invention

[0006] While analyzing the projection scheme for 3D Gaussian primitives, the inventors discovered that their optimization freedom was too high, resulting in an overall underconstrained training process, which could easily lead to non-compliance with actual model requirements or even overfitting. At the same time, they realized that surface normal continuity constraints could effectively guide Gaussian primitives to fit the actual surface of the model. Furthermore, the original projection scheme for 3D Gaussian primitives was not based on strict projective geometry derivation, resulting in inconsistent projection results from multiple perspectives, which interfered with the parallax information required for model training. Therefore, the inventors redesigned the projection scheme for 3D Gaussian primitives based on the fundamental principles of projective geometry and quadratic surfaces, and on this basis, achieved differentiable rendering of normals and depth.

[0007] In view of the shortcomings of existing technologies, such as Figure 3 As shown, the present invention proposes a multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization, which includes:

[0008] A data acquisition step is to acquire multi-view images of the target object to be reconstructed, wherein the multi-view images are images of the target object taken at multiple viewpoints, and each image has corresponding pose information;

[0009] A model training step, based on the multi-view image and the pose information, trains a three-dimensional Gaussian sputtering model through single-view rendering, loss calculation, gradient backpropagation, and model densification to obtain a final Gaussian primitive;

[0010] In the image reconstruction step, each part of the final Gaussian primitive is labeled, and a rendering method corresponding to the label is used to reconstruct the final Gaussian primitive into a mesh model, and a rendered depth image and RGB image are generated.

[0011] The multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization, wherein the model training step includes:

[0012] In the preprocessing step, the Gaussian primitives behind the camera are removed in the camera coordinate system; the position of the camera in the local coordinates of the Gaussian primitive is calculated based on the camera position, the position, rotation and scaling of the Gaussian primitive, that is, , where C is the camera position, P, R, and S are the position vector, rotation matrix, and scaling matrix of the Gaussian primitive respectively; remove Gaussian primitives whose camera position vector length in the local coordinate system is less than the preset value; calculate the tangent value t of the cone angle and the direction vector d of the vector based on the length r of the camera position vector in the local coordinate system of the Gaussian primitive; calculate the cone covariance of the Gaussian primitive in the local coordinate system , where I is the unit matrix; calculate the cone covariance in the camera coordinate system, that is , where V is the camera rotation matrix; calculate the intersection equation on the viewing plane according to the cone equation , where zfar is the viewing plane distance, W and H are the width and height of the picture;

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] correspond:

[0019]

[0020]

[0021]

[0022] Corresponding to the two-dimensional Gaussian distribution in screen space:

[0023]

[0024]

[0025] According to the center position and covariance of the screen, Gaussian primitives outside the screen and Gaussian primitives projected as hyperboloids are filtered out; the depth reference plane equation in the camera coordinate system is calculated based on r and d, where x and y are the coordinates of the pixel in the screen space. The depth reference plane equation in the local coordinate system is:

[0026]

[0027] In the camera coordinate system, the depth reference plane equation is:

[0028]

[0029] In the camera coordinate system, the ray parameter equation of the line of sight corresponding to each pixel is:

[0030]

[0031] Solving the equations yields the ray parameter u and depth ;

[0032] Calculate the primitive normal based on R and V, and adjust the normal to an angle less than 90 degrees with the camera's Z direction; calculate the primitive coverage;

[0033] Calculate the color of the primitive: calculate the RGB components separately from the spherical harmonics; when the type attribute is greater than 0, adjust the RGB components according to the configuration;

[0034] Calculate the depth of the primitive at the center of each block based on the reference plane; sort by location ID and center depth; collect RGB, 2D covariance, depth plane, and normal information of the relevant primitives for each block; calculate the probability density function pixel by pixel, and filter out primitives with opacity below a preset value; calculate the depth of the primitive at each pixel based on the reference plane; blend RGB, depth, and normal based on transparency, and normalize the normal blending result;

[0035] In the post-processing step, the screen space normal is calculated based on the depth image and the depth image is normalized.

[0036] The multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization, wherein the model training step includes:

[0037] Loss calculation step, calculate RGB loss based on rendering results and reference image , add the self-supervised Laplace loss function of cumulative normal and the 3D rotation loss R based on the screen space normal SN , and the shape constraint regularization term REC ;

[0038] The difference in local normals of the 3D Gaussian sputtering model is calculated using the Laplace operator. This difference should be minimized during the optimization process of the 3D Gaussian sputtering model. The edge of the 3D Gaussian sputtering model is approximately detected using the depth image gradient information as the shielding area.

[0039] Based on the viewing angle and depth information, the camera space coordinates corresponding to the pixel are obtained. The radial vector cross product between the three-dimensional points corresponding to adjacent pixels in the x and y directions of the pixel screen is performed to obtain the normal direction of the pixel. The normal direction is converted to the world space coordinate system according to the camera transformation matrix, and the supervised difference is calculated with the normal image based on the screen space. The supervised difference is calculated using a three-dimensional rotation loss function to adapt to the problem that the rasterization result of the three-dimensional Gaussian primitive is non-orientable.

[0040]

[0041] To constrain the shape of the primitive, the reconstruction module adds a shape constraint regularization term R EC , which is used to guide the primitives to change to a flat and uniform shape:

[0042]

[0043] The optimization objective loss function of the reconstruction module is: .in is the local gradient mask of the depth image, is the self-supervised Laplace loss function The weight factor, is the three-dimensional rotation loss R SN The weight factor of .

[0044] The multi-view image reconstruction method based on three-dimensional Gaussian sputtering representation, wherein the gradient return includes: calculating the transparency gradient, depth reference plane gradient, normal gradient, and RGB gradient of each primitive according to the depth, normal, RGB gradient of each pixel and the transparency of each primitive; calculating the corresponding two-dimensional covariance gradient according to the transparency gradient of each primitive; calculating the rotation, displacement, and scaling gradient of the primitive according to the two-dimensional covariance gradient, depth reference plane gradient, and normal gradient of each primitive; and calculating the spherical harmonic function gradient according to the RGB gradient of each primitive;

[0045] The model densification includes: classifying primitives into large primitives and small primitives according to the y-axis length of the primitive and the scene size, cloning the gradient of the small primitive and splitting the large primitive; eliminating primitives whose primitive size exceeds the preset value; eliminating primitives whose z-axis length is greater than the x-axis length by a preset multiple and eliminating primitives whose eccentricity is greater than the preset value.

[0046] The multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization, wherein the three-dimensional Gaussian sputtering model is annotated and adjusted through a visualization window:

[0047] View the effects of the 3D Gaussian sputtering model in real time during training using the normal, screen space normal, and depth visualization options. Label the model with different types so that different types of primitives in the model are rendered differently.

[0048] like Figure 4 As shown, the present invention also proposes a multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization, which includes:

[0049] The data acquisition module collects and obtains multi-view images of the target object to be reconstructed. The multi-view images are images of the target object taken from multiple perspectives, and each image has corresponding posture information;

[0050] The model training module trains a three-dimensional Gaussian sputtering model based on the multi-view image and the pose information through single-view rendering, loss calculation, gradient backpropagation, and model densification to obtain the final Gaussian primitive;

[0051] The image reconstruction module labels each part of the final Gaussian primitive, and uses a rendering method corresponding to the labeling to reconstruct the final Gaussian primitive into a mesh model, and generates a rendered depth image and RGB image.

[0052] The multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization, wherein the model training module includes:

[0053] The preprocessing module removes the Gaussian primitives behind the camera in the camera coordinate system; calculates the camera position in the local coordinates of the Gaussian primitive based on the camera position, the position, rotation and scaling of the Gaussian primitive, that is, , where C is the camera position, P, R, and S are the position vector, rotation matrix, and scaling matrix of the Gaussian primitive respectively; remove Gaussian primitives whose camera position vector length in the local coordinate system is less than the preset value; calculate the tangent value t of the cone angle and the direction vector d of the vector based on the length r of the camera position vector in the local coordinate system of the Gaussian primitive; calculate the cone covariance of the Gaussian primitive in the local coordinate system , where I is the unit matrix; calculate the cone covariance in the camera coordinate system, that is , where V is the camera rotation matrix; calculate the intersection equation on the viewing plane according to the cone equation , where zfar is the viewing plane distance, W and H are the width and height of the picture;

[0054]

[0055]

[0056]

[0057]

[0058]

[0059] correspond:

[0060]

[0061]

[0062]

[0063] Corresponding to the two-dimensional Gaussian distribution in screen space:

[0064]

[0065]

[0066] According to the center position and covariance of the screen, Gaussian primitives outside the screen and Gaussian primitives projected as hyperboloids are filtered out; the depth reference plane equation in the camera coordinate system is calculated based on r and d, where x and y are the coordinates of the pixel in the screen space. The depth reference plane equation in the local coordinate system is:

[0067]

[0068] In the camera coordinate system, the depth reference plane equation is:

[0069]

[0070] In the camera coordinate system, the ray parameter equation of the line of sight corresponding to each pixel is:

[0071]

[0072] Solving the equations yields the ray parameter u and depth ;

[0073] Calculate the primitive normal based on R and V, and adjust the normal to an angle less than 90 degrees with the camera's Z direction; calculate the primitive coverage;

[0074] Calculate the color of the primitive: calculate the RGB components separately from the spherical harmonics; when the type attribute is greater than 0, adjust the RGB components according to the configuration;

[0075] Calculate the depth of the primitive at the center of each block based on the reference plane; sort by location ID and center depth; collect RGB, 2D covariance, depth plane, and normal information of the relevant primitives for each block; calculate the probability density function pixel by pixel, and filter out primitives with opacity below a preset value; calculate the depth of the primitive at each pixel based on the reference plane; blend RGB, depth, and normal based on transparency, and normalize the normal blending result;

[0076] The post-processing module calculates the screen space normal based on the depth image and normalizes the depth image;

[0077] The model training module includes:

[0078] Loss calculation module, calculates RGB loss based on rendering results and reference images , add the self-supervised Laplace loss function of cumulative normal and the 3D rotation loss R based on the screen space normal SN , and the shape constraint regularization term R EC ;

[0079] The difference in local normals of the 3D Gaussian sputtering model is calculated using the Laplace operator. This difference should be minimized during the optimization process of the 3D Gaussian sputtering model. The edge of the 3D Gaussian sputtering model is approximately detected using the depth image gradient information as the shielding area.

[0080] Based on the viewing angle and depth information, the camera space coordinates corresponding to the pixel are obtained. The radial vector cross product between the three-dimensional points corresponding to adjacent pixels in the x and y directions of the pixel screen is performed to obtain the normal direction of the pixel. The normal direction is converted to the world space coordinate system according to the camera transformation matrix, and the supervised difference is calculated with the normal image based on the screen space. The supervised difference is calculated using a three-dimensional rotation loss function to adapt to the problem that the rasterization result of the three-dimensional Gaussian primitive is non-orientable.

[0081]

[0082] To constrain the shape of the primitive, the reconstruction module adds a shape constraint regularization term R EC , which is used to guide the primitives to change to a flat and uniform shape:

[0083]

[0084] The optimization objective loss function of the reconstruction module is: ; is the local gradient mask of the depth image, is the self-supervised Laplace loss function The weight factor, is the three-dimensional rotation loss R SNThe weight factor of .

[0085] The gradient return includes: calculating the transparency gradient, depth reference plane gradient, normal gradient, and RGB gradient of each primitive based on the depth, normal, RGB gradient of each pixel and the transparency of each primitive; calculating the corresponding two-dimensional covariance gradient based on the transparency gradient of each primitive; calculating the rotation, displacement, and scaling gradient of each primitive based on the two-dimensional covariance gradient, depth reference plane gradient, and normal gradient of each primitive; and calculating the spherical harmonic function gradient based on the RGB gradient of each primitive.

[0086] The model densification includes: classifying primitives into large primitives and small primitives according to the y-axis length of the primitive and the scene size, cloning the gradient of the small primitive and splitting the large primitive; eliminating primitives whose primitive size exceeds the preset value; eliminating primitives whose z-axis length is greater than the x-axis length by a preset multiple and eliminating primitives whose eccentricity is greater than the preset value.

[0087] Use the visualization window to mark and adjust the 3D Gaussian sputtering model:

[0088] View the effects of the 3D Gaussian sputtering model in real time during training using the normal, screen space normal, and depth visualization options. Label the model with different types so that different types of primitives in the model are rendered differently.

[0089] The present invention also proposes an electronic device, including the multi-perspective image reconstruction device based on three-dimensional Gaussian sputtering representation. The electronic device may be connected to an information display device, and the information display device is used to display the grid model, the rendered depth image and the RGB image according to the display parameters and attributes set by the user or through an artificial intelligence model.

[0090] The present invention also proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization are implemented.

[0091] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization are implemented.

[0092] From the above scheme, it can be seen that the advantages of the present invention are:

[0093] The reconstruction result model has high quality, dense and reasonable distribution of graphics elements. The present invention can be applied to the reconstruction of plant leaves, and can better perceive the changes in leaf curvature, and output a three-dimensional Gaussian model with reasonable distribution of graphics elements, flat shape, and fit to the page. It is also used in the reconstruction of outdoor open scenes and indoor scenes. The reconstruction results can provide better depth information than the original three-dimensional Gaussian reconstruction method. The reconstruction results of the present invention in a variety of scenes are significantly improved compared to the original version. By collecting multi-view depth rendering results and processing them with TSDF, a better mesh model can be obtained. The model can be further adjusted subsequently through three-dimensional modeling software, and can be more conveniently integrated into the industrial three-dimensional reconstruction process. Its convenience and high efficiency greatly shorten the reconstruction operation cycle, reduce reconstruction costs, and improve reconstruction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 Schematic diagram comparing the rasterization scheme of the present invention with the original rasterization scheme;

[0095] Figure 2 Schematic diagram of the depth plane D, projected depth, and normal calculation scheme;

[0096] Figure 3 Flow chart of the method of the present invention;

[0097] Figure 4 This is a module diagram of the device of the present invention;

[0098] Figure 5 This is a schematic structural diagram of a first electronic device of the present invention;

[0099] Figure 6 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0100] Figure 7 This is a schematic structural diagram of a second electronic device according to the present invention.

[0101] Reference numerals:

[0102] A-First electronic device;

[0103] B-Multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization;

[0104] C-data acquisition equipment;

[0105] D-information display device;

[0106] 1000- second electronic device;

[0107] Ⅰ-computing unit;

[0108] II-ROM;

[0109] III-RAM;

[0110] IV-bus;

[0111] V-interface;

[0112] VI-input unit;

[0113] VII-output unit;

[0114] VIII-Storage medium;

[0115] IX-Communication unit. DETAILED DESCRIPTION

[0116] It should be noted that, in this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0117] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0118] The processor described in the present invention is the control center of an electronic device and can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0119] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0120] In a specific implementation, as an example, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptop computers, smartphones, tablet computers, embedded computers, etc., where the embedded computers include vehicles and robots, etc.

[0121] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0122] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0123] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0124] It should also be understood that the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0125] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0126] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0127] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0131] In response to the problems of the 3DGS method, such as lack of effective supervision information, poor surface fitting quality, inconsistent multi-view rendering, and imperfect visualization tools, the present invention proposes a dense three-dimensional Gaussian reconstruction method based on envelope surfaces and surface constraints, optimizes the three-dimensional Gaussian primitive rasterization method, provides a consistent and accurate two-dimensional projection range under multiple perspectives, supplements the normal and depth gradient transmission paths, optimizes the primitive morphological representation scheme, constrains the primitive shape, improves the reconstruction training efficiency, adds a self-supervised Laplace loss function of cumulative normals and a three-dimensional rotation loss based on screen space normals, optimizes the arrangement characteristics and spatial posture of the three-dimensional Gaussian primitives, obtains a result that is more consistent with the surface characteristics of the reconstructed object, and supplements the normal and depth visualization functions, so that the model details can be viewed intuitively, conveniently, and at any time to evaluate the model quality. At the same time, the present invention designs a three-dimensional Gaussian annotation tool according to actual use needs to achieve rapid annotation and classification display of three-dimensional Gaussian primitives.

[0132] To illustrate the above-mentioned features and effects of the present invention more clearly and easily, the following embodiments are specifically described below with reference to the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are for illustrative purposes only. The scope of protection of the present invention is not limited to the disclosed embodiments; the present invention is defined by the appended claims.

[0133] The specific use of the present invention is mainly divided into three stages, including data acquisition, three-dimensional reconstruction, annotation and gridding.

[0134] The first stage of data collection requires obtaining multi-view imagery and pose information required for reconstruction. Multi-view images can be captured using either stills or videos. Camera focus, exposure, and color temperature should be adjusted according to the actual shooting environment. Camera parameters and ambient lighting should be standardized or fixed throughout the recording process. The subject should be kept as still as possible, and the chosen angles should ensure full coverage of the subject. Depending on the reconstruction object and actual needs, various methods can be used, including handheld, gantry, or rail-mounted, or drone-mounted. For video recording, clear, jitter-free frames should be extracted from the video as reconstruction material. For each reconstructed object, a minimum of 40 images / 40 frames of reconstruction material is recommended. The camera pose information for each image / frame can be calculated using Colmap software. For footage captured by a drone, robotic arm, or rail-mounted system equipped with high-precision positioning and a gyroscope, the positioning data of the capture device itself can also be used. The size of the reconstructed image can be adjusted based on the required reconstruction accuracy and device performance. A width and height of approximately 1000 pixels yields relatively good reconstruction time and quality. Image proportions should be consistent, with minimal discrepancies.

[0135] Finally, the acquired images and pose data files are classified and stored in the same folder. The specific format is the same as the output format of Colmap.

[0136] In the second stage, the training code is called to generate a Gaussian model (3D Gaussian sputtering model). The command line input is:

[0137] python train.py -s data folder path [-m output model path] [--scale<ab|ef|raw> ] [--local_normal_loss] [--local_normal_loss_after normal-depth constraint startup round] [--vf_loss] [--vf_loss_after normal self-supervision startup round]

[0138] Configure the startup parameters as needed. Additionally, add "--GUI" to the startup parameters to start the visualization tool at the same time as training. Use the visualization tool's UI slider to adjust the display mode.

[0139] In the training code, in the order of data transmission, there are model loading here and scene initialization, single-view rendering, loss calculation, gradient backpropagation and model densification. The main code flow of the present invention is as follows:

[0140] Model loading (Python implementation)

[0141] 1. Load the SfM point cloud model and randomly assign size attributes to each point cloud, including X-axis length, eccentricity, and flatness

[0142] 2. The size attribute is a three-dimensional vector, with the x, y, and z components corresponding to the X-axis length, flattening, and eccentricity, respectively.

[0143] 3. X-axis length is

[0144] 4. Flatness is the ratio of the Y-axis length to the X-axis length. If the flatness is greater than 1, it is expressed as , the Y-axis length of this point is

[0145] 5. Eccentricity is the ratio of the Z-axis length to the Y-axis length, limited to 1-5, and is used Indicates that the Z-axis length of this point is

[0146] 6. Each point cloud is assigned a type attribute, initialized to 0

[0147] Single-view rendering is divided into four parts: pre-processing, depth sorting, rasterization and post-processing

[0148] Preprocessing (Cuda)

[0149] 13. Eliminate Gaussian primitives behind the camera in the camera coordinate system

[0150] 14. Calculate the position of the camera in the local coordinates of the Gaussian element based on the camera position, the position, rotation and scaling of the Gaussian element, that is, , where C is the camera position, P, R, and S are the position vector, rotation matrix, and scaling matrix of the Gaussian primitive, respectively.

[0151] 15. Eliminate primitives whose camera position vector length in the local coordinate system is less than 1 (projected as a virtual sphere)

[0152] 16. Calculate the tangent value t of the cone angle and the direction vector d of the vector based on the length r of the camera position vector in the local coordinate system

[0153] 17. Calculate the local coordinate system cone covariance , where I is the identity matrix

[0154] 18. Calculate the cone covariance in the camera coordinate system, that is , where V is the camera rotation matrix

[0155] 19. Calculate the equation of the intercept on the viewing plane according to the cone equation, as follows: Where zfar is the viewing plane distance, W and H are the width and height of the screen

[0156]

[0157] Cone equation:

[0158] Joint , we can get the intercept equation:

[0159]

[0160]

[0161] correspond:

[0162]

[0163]

[0164]

[0165] Corresponding to the two-dimensional Gaussian distribution in screen space:

[0166]

[0167]

[0168] 20. Filter out Gaussian primitives outside the screen and Gaussian primitives projected as hyperbolic surfaces based on the screen center position and covariance

[0169] 21. Calculate the depth reference plane equation in the camera coordinate system based on r and d, where x and y are the coordinates of the pixel in screen space

[0170] The depth reference plane equation in the local coordinate system is:

[0171]

[0172] In the camera coordinate system, the depth reference plane equation is

[0173]

[0174] In the camera coordinate system, the ray parameter equation of each pixel corresponding to the line of sight is

[0175]

[0176] Solving the equations yields the ray parameter u and depth

[0177] 22. Calculate the primitive normal based on R and V, and adjust the normal to an angle less than 90 degrees with the camera's Z direction

[0178] 23. Calculate primitive coverage

[0179] 24. Calculate the color of the primitive

[0180] a) Calculate RGB components separately from spherical harmonics

[0181] b) When the type attribute is greater than 0, the RGB components are adjusted according to the configuration. At this time, the color does not participate in the gradient transfer.

[0182] Depth sorting (Cuda)

[0183] 1. Calculate the depth of the primitive at the center of each patch based on the reference plane. A patch is a square pixel block area that is automatically allocated by Cuda during rasterization. It is the representation scheme used by the original Gaussian sputtering scheme to accelerate rendering. The general block size is 4x4.

[0184] 2. Sort by location ID and center depth

[0185] Rasterization (Cuda)

[0186] 1. Each block collects RGB, 2D covariance, depth plane, and normal information of relevant primitives

[0187] 2. Calculate the probability density function pixel by pixel to filter out the elements with too low opacity

[0188] 3. Calculate the depth of the primitive at each pixel based on the reference plane

[0189] 4. Mixing RGB, depth and normal according to transparency, and normalize the normal mixing result

[0190] 5. Post-processing (Pytorch implementation)

[0191] 6. Calculate screen space normals based on depth image

[0192] 7. Normalize the depth image

[0193] Loss calculation (Pytorch implementation)

[0194] 1. Calculate L1 Loss and SSIM Loss based on the rendering results and reference images

[0195] 2. Calculate the local gradient mask of the depth image through the Sob operator. The pixels with a gradient less than \delta are the valid area.

[0196] 3. Calculate the local depth difference regularization term through the Laplace operator

[0197] 4. Calculate the screen space normal regularization term through 3D rotation loss

[0198] 5. Calculate the shape constraint regularization term

[0199] 6. Merge Loss

[0200]

[0201] Here Indicates loss Loss, is the corresponding weight factor, subscript D represents the depth loss, Represents the screen space normal Loss, EC represents the eccentricity Loss, LossRGB represents the image L1Loss and SSIM Loss, and mask represents the edge MASK mentioned above.

[0202] Gradient backpropagation (Cuda implementation)

[0203] 5. Calculate the transparency gradient, depth reference plane gradient, normal gradient, and RGB gradient of each primitive based on the depth, normal, RGB gradient of each pixel and the transparency of each primitive;

[0204] 6. Calculate the corresponding two-dimensional covariance gradient based on the transparency gradient of each primitive

[0205] 7. Calculate the primitive rotation, displacement, and scaling gradient based on the two-dimensional covariance gradient, depth reference plane gradient, and normal gradient of each primitive

[0206] 8. Calculate the spherical harmonic function gradient based on the RGB gradient of each primitive

[0207] Model densification (Pytorch implementation)

[0208] 1. Distinguish small and large primitives based on the primitive's y-axis length and scene size, clone small primitives with larger gradients, and split large primitives with larger gradients;

[0209] 2. Eliminate oversized primitives;

[0210] 3. Eliminate primitives whose z-axis length is much longer than the x-axis length and whose eccentricity is greater than 4.8.

[0211] In the third stage, before data annotation, a configuration file named "modifier" can be added to the model output folder. This file sequentially configures the visualization colors of different types of components. Each line of the file contains six floating-point numbers, corresponding to nine types of component color offsets, numbered 1-9. The six floating-point numbers represent the relative and absolute offset values ​​for the three RGB channel color values. The final color of each primitive is the original RGB * relative offset value + absolute offset value. Open the model using the Sibr_GaussianViewer_app.exe. In the options box, you can select visualization options such as normal, screen-space normal, depth, and RGB. You can also switch between the original 3D Gaussian rasterization module and the new rasterization module used in this invention. Use the number keys 1-9 to switch between different annotation types, and the number key 0 to clear the annotation type. Left-clicking the target area with the mouse will annotate the primitives in that area with the set type. Holding the Alt key and the left mouse button while moving the mouse controls the camera's viewing angle. After annotation is complete, select "Save Gaussian Model" from the menu bar to save the annotation results as a PLY file. To generate a mesh model, run the render.py script, specify the model folder, generate rendered depth images and RGB images, and run the TSDF reconstruction script to generate a mesh model. The mesh model is a 3D mesh model that can be rendered from any perspective using a renderer.

[0212] In order to achieve the above technical effects, the present invention further proposes the following key technical points:

[0213] Key point 1. Realize three-dimensional Gaussian primitive projection based on envelope surface projection.

[0214] This paper re-derives the projection equation of three-dimensional primitives based on the underlying principles of projective geometry to calculate the pixel range covered by the primitive and the corresponding projection depth of each pixel. The specific derivation ideas are as follows:

[0215] The isosurface of the probability density function of the three-dimensional Gaussian distribution is a quadratic surface, and the covariance matrix of the three-dimensional Gaussian distribution is a semi-positive matrix. Combined with the quadratic surface discrimination conditions, it can be seen that the quadratic surface is an ellipsoid, and its shape (three-axis length and orientation) is directly corresponding to the covariance matrix of the Gaussian distribution. We assume that the position E, scale S, and rotation R of the three-dimensional Gaussian primitive correspond to the expectation and covariance of the three-dimensional Gaussian distribution. Then, we can select one of its isosurfaces so that its shape is the ellipsoid defined by the posture attributes of the Gaussian primitive. The ellipsoid is selected as the projection object and examined in the camera coordinate system. The envelope of the projection of the ellipsoid on the imaging plane at a distance Z directly in front of the camera is the curve where all the tangents of the ellipsoid passing through the camera position and the imaging plane intersect. That is, the envelope of the ellipsoid projection is the intersection line I of the envelope E formed by the tangents of the ellipsoid passing through the camera position (0,0,0) and the imaging plane. The ellipsoid and the sphere can be converted into each other through linear transformations. Assume that the linear transformation T transforms the ellipsoid in the camera space into a unit sphere.

[0216] In the changed coordinate system (called the local coordinate system), the surface formed by all tangents of the unit sphere passing through the camera T(0,0,0) is a conical surface C (or an imaginary sphere, but this case needs to be eliminated)

[0217]

[0218] in

[0219]

[0220]

[0221] Then in the camera coordinate system, the envelope surface is

[0222]

[0223] The conic surface is also a conical surface, and the intersection equation of the envelope surface and the imaging plane can be calculated. For envelope I, we choose to eliminate the two non-closed shapes, the parabola and the hyperbola, based on the conic discriminant, retaining the elliptical curve. We assume that the ellipse corresponds to an isosurface of a 2D Gaussian distribution. The probability density function of this 2D Gaussian distribution determines the transparency attenuation of the projection of the 3D Gaussian primitive at each position on the imaging plane.

[0224] Key point 2: Determine the depth reference plane based on the plane Jacobi transform and calculate the depth of the primitive.

[0225] Depth plane D, projection depth, and normal calculation scheme are as follows Figure 2 At the same time, in the local coordinate system, the intersection of the cone and the unit sphere is located on the same plane, and we choose this plane as the reference plane D for the projection depth of the Gaussian element.

[0226]

[0227] The depth of the Gaussian element observed by different sight lines is the distance from the sight line to the reference plane. The plane has the d direction as the normal direction. The reference plane D' in the camera coordinate is obtained by changing D through J(T). The corresponding equation is

[0228]

[0229]

[0230] Pixels The corresponding depth is:

[0231]

[0232] In addition, we select the positive direction of the shortest axis of the ellipse as the normal direction of the 3D Gaussian primitive. Since each plane has positive and negative normals, a unified direction must be selected when accumulating normals. The normal direction of each primitive also needs to be calculated with the positive line of sight. Normals with dot products less than 0 are negated. The color sampling of the Gaussian primitive is the same as the original version, and is sampled according to the camera direction.

[0233] One-dimensional Gaussian distribution in [-3 , 3 ] is greater than 99%. Similarly, we calculate the maximum eigenvalue based on the covariance matrix and obtain the maximum variance of the two-dimensional Gaussian distribution corresponding to I. , centered on I, 3 Select a pixel block for the half-side length to determine the rasterization range of the 3D Gaussian primitive. The distance from the ray from the camera position to the center of each pixel block to the primitive depth reference plane D is used as the projection depth of the primitive in the pixel block.

[0234] Key Point 3: Calculate screen space normals based on rendered depth image

[0235] Each pixel then overlays the color of any 2D Gaussian primitives it may overlay, based on the projected depth of the pixel block, from near to far, according to the attenuated transparency. Simultaneously, the depth is overlaid directly based on the transparency of the 3D Gaussian primitive itself. After outputting the depth image, the screen-space normal and world-space normal can be calculated based on the depth gradient and camera coordinate transformation. Furthermore, the normals are overlaid in exactly the same way as RGB, resulting in a cumulative normal image. The gradient transfer from depth and normal pixels to the Gaussian primitive is calculated in the opposite manner of the overlay process, iteratively calculating the weight of each primitive from far to near.

[0236] At this point, the rasterization rendering of the three-dimensional Gaussian primitive is completed.

[0237] Key point 4: Modify the scaling representation scheme of the 3D Gaussian primitive and select the shortest axis of the primitive as the normal direction.

[0238] The flatter graphics have greater parallax under different viewing angles, which makes it easier to determine the normal direction and calculate the normal gradient. However, the scaling and rotation of the three-dimensional Gaussian can affect the shape of the Gaussian graphics. It is difficult for the normal gradient to reasonably guide the changes in the rotation and scaling factors, resulting in difficulty in converging the training effect. The present invention modifies the representation of the three-dimensional graphics, and the y- and z-axis scaling factors are based on the x-axis scaling factor and are greater than 1. This makes it so that in the local coordinates of the graphics, the x-axis direction is always the shortest axis, constraining the solution space of the three-dimensional Gaussian model and making the normal guidance effect better. At the same time, the flatness and eccentricity are defined by the ratio of the x-, y-, and z-axis scaling factors, and during the training process, a portion of graphics that are too flat, sharp, and may cause gradient explosions are eliminated based on the flatness and eccentricity.

[0239] Key point 5: Add Laplace self-supervised loss, 3D rotation loss, and shape constraint regularization terms to the cumulative normal, screen space normal, and primitive shape respectively.

[0240] In the reconstruction module, in addition to the RGB supervision information, a self-supervised Laplace loss function of cumulative normals is added and the 3D rotation loss R based on the screen space normal SN , and the shape constraint regularization term R EC .

[0241] Based on the assumption of smooth surface, we can approximate that the surface normals vary uniformly. Using the Laplace operator, we calculate the difference in local normals, minimizing this difference during model optimization. Furthermore, because the assumption of smooth normals is limited to the same surface, normals on different surfaces do not affect each other. We can use depth image gradient information to approximate model edges and use them as shielding areas.

[0242] Used to represent the xy index coordinates of the image, which traverses all pixels of the image. represents the Laplace operator;

[0243] Based on the viewing angle and depth information, the camera space coordinates corresponding to the pixel can be estimated. The radial vector cross product between the three-dimensional points corresponding to adjacent pixels in the x and y directions of the pixel screen can approximate the normal direction at that pixel. This can then be converted to the world space coordinate system based on the camera transformation matrix, thereby calculating the supervised difference with the screen-space normal image. This difference is calculated using a three-dimensional rotation loss function to adapt to the non-orientability problem of the rasterized results of three-dimensional Gaussian primitives. Similarly, the self-supervised loss based on the screen-space normal also requires masking edge regions.

[0244]

[0245] In addition, to constrain the shape of the primitive, the reconstruction module can also optionally add a shape constraint regularization term R EC , which is used to guide the primitives to change into a flat and uniform shape.

[0246]

[0247] r represents the radius, rx, ry, and rz represent the radii of the Gaussian element in the xyz direction, respectively, θ is the weight parameter, the subscript f represents the flatness coefficient, and the subscript e represents the ellipse coefficient;

[0248] Finally, the optimization goal of the reconstruction module is:

[0249]

[0250] Key Point 6: Optimize the 3D Gaussian visualization tool to provide rendering mode switching, model grouping and annotation, and rendering functions

[0251] During training, you can view the model performance in real time through sibr_remoteGaussian_app_d.exe. To better monitor the training process, we have added visualization options for normals, screen space normals, and depth, with sliders to control different display modes.

[0252] In our visualization tool, you can switch between different annotation types by pressing keys 0-9, switch the visualization perspective in FPS, interactively mark Gaussian primitives in the specified area with the mouse button, and render different types of primitives in different colors. The annotated model can be saved as a new model with classification information. In addition to being able to switch between multiple rendering results, the visualization tool can also seamlessly switch between the original rasterization rendering and the new rasterization method of this method in real time. The comparison between the rasterization scheme of the present invention and the original rasterization scheme is as follows: Figure 1 shown.

[0253] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0254] like Figure 4 As shown, the present invention also proposes a multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization, which includes:

[0255] The data acquisition module collects and obtains multi-view images of the target object to be reconstructed. The multi-view images are images of the target object taken from multiple perspectives, and each image has corresponding posture information;

[0256] The model training module trains a three-dimensional Gaussian sputtering model based on the multi-view image and the pose information through single-view rendering, loss calculation, gradient backpropagation, and model densification to obtain the final Gaussian primitive;

[0257] The image reconstruction module labels each part of the final Gaussian primitive, and uses a rendering method corresponding to the labeling to reconstruct the final Gaussian primitive into a mesh model, and generates a rendered depth image and RGB image.

[0258] The multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization, wherein the model training module includes:

[0259] The preprocessing module removes the Gaussian primitives behind the camera in the camera coordinate system; calculates the camera position in the local coordinates of the Gaussian primitive based on the camera position, the position, rotation and scaling of the Gaussian primitive, that is, , where C is the camera position, P, R, and S are the position vector, rotation matrix, and scaling matrix of the Gaussian primitive respectively; remove Gaussian primitives whose camera position vector length in the local coordinate system is less than the preset value; calculate the tangent value t of the cone angle and the direction vector d of the vector based on the length r of the camera position vector in the local coordinate system of the Gaussian primitive; calculate the cone covariance of the Gaussian primitive in the local coordinate system , where I is the unit matrix; calculate the cone covariance in the camera coordinate system, that is , where V is the camera rotation matrix; calculate the intersection equation on the viewing plane according to the cone equation , where zfar is the viewing plane distance, W and H are the width and height of the picture;

[0260]

[0261]

[0262]

[0263]

[0264]

[0265] correspond:

[0266]

[0267]

[0268]

[0269] Corresponding to the two-dimensional Gaussian distribution in screen space:

[0270]

[0271]

[0272] According to the center position and covariance of the screen, Gaussian primitives outside the screen and Gaussian primitives projected as hyperboloids are filtered out; the depth reference plane equation in the camera coordinate system is calculated based on r and d, where x and y are the coordinates of the pixel in the screen space. The depth reference plane equation in the local coordinate system is:

[0273]

[0274] In the camera coordinate system, the depth reference plane equation is:

[0275]

[0276] In the camera coordinate system, the ray parameter equation of the line of sight corresponding to each pixel is:

[0277]

[0278] Solving the equations yields the ray parameter u and depth ;

[0279] Calculate the primitive normal based on R and V, and adjust the normal to an angle less than 90 degrees with the camera's Z direction; calculate the primitive coverage;

[0280] Calculate the color of the primitive: calculate the RGB components separately from the spherical harmonics; when the type attribute is greater than 0, adjust the RGB components according to the configuration;

[0281] Calculate the depth of the primitive at the center of each block based on the reference plane; sort by location ID and center depth; collect RGB, 2D covariance, depth plane, and normal information of the relevant primitives for each block; calculate the probability density function pixel by pixel, and filter out primitives with opacity below a preset value; calculate the depth of the primitive at each pixel based on the reference plane; blend RGB, depth, and normal based on transparency, and normalize the normal blending result;

[0282] The post-processing module calculates the screen space normal based on the depth image and normalizes the depth image;

[0283] The model training module includes:

[0284] Loss calculation module, calculates RGB loss based on rendering results and reference images , add the self-supervised Laplace loss function of cumulative normal and the 3D rotation loss R based on the screen space normal SN , and the shape constraint regularization term R EC ;

[0285] The difference in local normals of the 3D Gaussian sputtering model is calculated using the Laplace operator. This difference should be minimized during the optimization process of the 3D Gaussian sputtering model. The edge of the 3D Gaussian sputtering model is approximately detected using the depth image gradient information as the shielding area.

[0286] Based on the viewing angle and depth information, the camera space coordinates corresponding to the pixel are obtained. The radial vector cross product between the three-dimensional points corresponding to adjacent pixels in the x and y directions of the pixel screen is performed to obtain the normal direction of the pixel. The normal direction is converted to the world space coordinate system according to the camera transformation matrix, and the supervised difference is calculated with the normal image based on the screen space. The supervised difference is calculated using a three-dimensional rotation loss function to adapt to the problem that the rasterization result of the three-dimensional Gaussian primitive is non-orientable.

[0287]

[0288] To constrain the shape of the primitive, the reconstruction module adds a shape constraint regularization term R EC , which is used to guide the primitives to change to a flat and uniform shape:

[0289]

[0290] The optimization objective loss function of the reconstruction module is: ; is the local gradient mask of the depth image, is the self-supervised Laplace loss function The weight factor, is the three-dimensional rotation loss R SN The weight factor of .

[0291] The gradient return includes: calculating the transparency gradient, depth reference plane gradient, normal gradient, and RGB gradient of each primitive based on the depth, normal, RGB gradient of each pixel and the transparency of each primitive; calculating the corresponding two-dimensional covariance gradient based on the transparency gradient of each primitive; calculating the rotation, displacement, and scaling gradient of each primitive based on the two-dimensional covariance gradient, depth reference plane gradient, and normal gradient of each primitive; and calculating the spherical harmonic function gradient based on the RGB gradient of each primitive.

[0292] The model densification includes: classifying primitives into large primitives and small primitives according to the y-axis length of the primitive and the scene size, cloning the gradient of the small primitive and splitting the large primitive; eliminating primitives whose primitive size exceeds the preset value; eliminating primitives whose z-axis length is greater than the x-axis length by a preset multiple and eliminating primitives whose eccentricity is greater than the preset value.

[0293] Use the visualization window to mark and adjust the 3D Gaussian sputtering model:

[0294] View the effects of the 3D Gaussian sputtering model in real time during training using the normal, screen space normal, and depth visualization options. Label the model with different types so that different types of primitives in the model are rendered differently.

[0295] like Figure 5 As shown, the present invention further proposes a first electronic device A in another embodiment, including the multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization.

[0296] like Figure 6 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to acquire multi-perspective images, such as the multi-perspective videos or pictures described in the embodiment of the present invention, and the information display device D is used to display the grid model, rendered depth image and RGB image analyzed by the present invention.

[0297] The information display device D can organize and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll, etc. The user is presented with the key information specified by the user, such as the rendered depth image and RGB image of a specific perspective, so that the user can understand this information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present the user with rich and necessary key information.

[0298] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-perspective image reconstruction method based on three-dimensional Gaussian sputtering characterization provided by the above methods.

[0299] In another embodiment, the present invention further proposes a storage medium VIII for storing a computer program for executing the multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM).

[0300] Figure 7 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0301] Second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) II or loaded from a storage medium VIII into a random access memory (RAM) III. RAM III may also store various programs and data required for the operation of device 1000. Computing unit I, ROM II, and RAM III are interconnected via a bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0302] Multiple components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard and mouse; an output unit VII, such as various types of displays and speakers; a storage medium VIII, such as a magnetic disk and optical disk; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0303] Computing unit I can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit I performs the various methods and processes described above, such as method steps S1-S3. For example, in some embodiments, the method can be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the method described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform the method by any other suitable means (e.g., via firmware).

[0304] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization, characterized in that: include: A data acquisition step is to acquire multi-view images of the target object to be reconstructed, wherein the multi-view images are images of the target object taken at multiple viewpoints, and each image has corresponding pose information; A model training step, based on the multi-view image and the pose information, trains a three-dimensional Gaussian sputtering model through single-view rendering, loss calculation, gradient backpropagation, and model densification to obtain a final Gaussian primitive; The image reconstruction step labels each part of the final Gaussian primitive, and uses a rendering method corresponding to the labeling to reconstruct the final Gaussian primitive into a mesh model, and generates a rendered depth image and RGB image; The model training steps include: In the preprocessing step, the Gaussian primitives behind the camera are removed in the camera coordinate system; the position of the camera in the local coordinates of the Gaussian primitive is calculated based on the camera position, the position, rotation and scaling of the Gaussian primitive, that is, , where C is the camera position, P, R, and S are the position vector, rotation matrix, and scaling matrix of the Gaussian primitive respectively; remove Gaussian primitives whose camera position vector length in the local coordinate system is less than the preset value; calculate the tangent value t of the cone angle and the direction vector d of the vector based on the length r of the camera position vector in the local coordinate system of the Gaussian primitive; calculate the cone covariance of the Gaussian primitive in the local coordinate system , where I is the unit matrix; calculate the cone covariance in the camera coordinate system, that is , where V is the camera rotation matrix; calculate the intersection equation on the viewing plane according to the cone equation , where zfar is the viewing plane distance, W and H are the width and height of the picture; correspond: Corresponding to the two-dimensional Gaussian distribution in screen space: According to the center position and covariance of the screen, Gaussian primitives outside the screen and Gaussian primitives projected as hyperboloids are filtered out; the depth reference plane equation in the camera coordinate system is calculated based on r and d, where x and y are the coordinates of the pixel in the screen space. The depth reference plane equation in the local coordinate system is: In the camera coordinate system, the depth reference plane equation is: In the camera coordinate system, the ray parameter equation of the line of sight corresponding to each pixel is: Solving the equations yields the ray parameter u and depth ; Calculate the primitive normal based on R and V, and adjust the normal to an angle less than 90 degrees with the camera's Z direction; calculate the primitive coverage; Calculate the color of the primitive: calculate the RGB components separately from the spherical harmonics; when the type attribute is greater than 0, adjust the RGB components according to the configuration; Calculate the depth of the primitive at the center of each block based on the reference plane; sort by location ID and center depth; collect RGB, 2D covariance, depth plane, and normal information of the relevant primitives for each block; calculate the probability density function pixel by pixel, and filter out primitives with opacity below a preset value; calculate the depth of the primitive at each pixel based on the reference plane; blend RGB, depth, and normal based on transparency, and normalize the normal blending result; In the post-processing step, the screen space normal is calculated based on the depth image and the depth image is normalized.

2. The multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization according to claim 1, characterized in that: The model training steps include: Loss calculation step, calculate RGB loss based on rendering results and reference image , add the self-supervised Laplace loss function of cumulative normal and the 3D rotation loss R based on the screen space normal SN , and the shape constraint regularization term R EC ; The difference in local normals of the 3D Gaussian sputtering model is calculated using the Laplace operator. This difference should be minimized during the optimization process of the 3D Gaussian sputtering model. The edge of the 3D Gaussian sputtering model is approximately detected using the depth image gradient information as the shielding area. Based on the viewing angle and depth information, the camera space coordinates corresponding to the pixel are obtained. The radial vector cross product between the three-dimensional points corresponding to adjacent pixels in the x and y directions of the pixel screen is performed to obtain the normal direction of the pixel. The normal direction is converted to the world space coordinate system according to the camera transformation matrix, and the supervised difference is calculated with the normal image based on the screen space. The supervised difference is calculated using a three-dimensional rotation loss function to adapt to the problem that the rasterization result of the three-dimensional Gaussian primitive is non-orientable. To constrain the shape of the primitive, the reconstruction module adds a shape constraint regularization term R EC , which is used to guide the primitives to change to a flat and uniform shape: The optimization objective loss function of the reconstruction module is: is the local gradient mask of the depth image, is the self-supervised Laplace loss function The weight factor, is the three-dimensional rotation loss R SN The weight factor of .

3. The multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization according to claim 1, characterized in that: The gradient return includes: calculating the transparency gradient, depth reference plane gradient, normal gradient, and RGB gradient of each primitive based on the depth, normal, RGB gradient of each pixel and the transparency of each primitive; calculating the corresponding two-dimensional covariance gradient based on the transparency gradient of each primitive; calculating the rotation, displacement, and scaling gradient of each primitive based on the two-dimensional covariance gradient, depth reference plane gradient, and normal gradient of each primitive; and calculating the spherical harmonic function gradient based on the RGB gradient of each primitive. The model densification includes: classifying primitives into large primitives and small primitives according to the y-axis length of the primitive and the scene size, cloning the gradient of the small primitive and splitting the large primitive; eliminating primitives whose primitive size exceeds the preset value; eliminating primitives whose z-axis length is greater than the x-axis length by a preset multiple and eliminating primitives whose eccentricity is greater than the preset value.

4. The multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization according to claim 1, characterized in that: Use the visualization window to mark and adjust the 3D Gaussian sputtering model: View the effects of the 3D Gaussian sputtering model in real time during training using the normal, screen space normal, and depth visualization options. Label the model with different types so that different types of primitives in the model are rendered differently.

5. A multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization, characterized in that: include: The data acquisition module collects and obtains multi-view images of the target object to be reconstructed. The multi-view images are images of the target object taken from multiple perspectives, and each image has corresponding posture information; The model training module trains a three-dimensional Gaussian sputtering model based on the multi-view image and the pose information through single-view rendering, loss calculation, gradient backpropagation, and model densification to obtain the final Gaussian primitive; An image reconstruction module labels each part of the final Gaussian primitive, and uses a rendering method corresponding to the labeling to reconstruct the final Gaussian primitive into a mesh model, and generates a rendered depth image and RGB image; Among them, the model training module includes: The preprocessing module removes the Gaussian primitives behind the camera in the camera coordinate system; calculates the camera position in the local coordinates of the Gaussian primitive based on the camera position, the position, rotation and scaling of the Gaussian primitive, that is, , where C is the camera position, P, R, and S are the position vector, rotation matrix, and scaling matrix of the Gaussian primitive respectively; remove Gaussian primitives whose camera position vector length in the local coordinate system is less than the preset value; calculate the tangent value t of the cone angle and the direction vector d of the vector based on the length r of the camera position vector in the local coordinate system of the Gaussian primitive; calculate the cone covariance of the Gaussian primitive in the local coordinate system , where I is the unit matrix; calculate the cone covariance in the camera coordinate system, that is , where V is the camera rotation matrix; calculate the intersection equation on the viewing plane according to the cone equation , where zfar is the viewing plane distance, W and H are the width and height of the picture; correspond: Corresponding to the two-dimensional Gaussian distribution in screen space: According to the center position and covariance of the screen, Gaussian primitives outside the screen and Gaussian primitives projected as hyperboloids are filtered out; the depth reference plane equation in the camera coordinate system is calculated based on r and d, where x and y are the coordinates of the pixel in the screen space. The depth reference plane equation in the local coordinate system is: In the camera coordinate system, the depth reference plane equation is: In the camera coordinate system, the ray parameter equation of the line of sight corresponding to each pixel is: Solving the equations yields the ray parameter u and depth ; Calculate the primitive normal based on R and V, and adjust the normal to an angle less than 90 degrees with the camera's Z direction; calculate the primitive coverage; Calculate the color of the primitive: calculate the RGB components separately from the spherical harmonics; when the type attribute is greater than 0, adjust the RGB components according to the configuration; Calculate the depth of the primitive at the center of each block based on the reference plane; sort by location ID and center depth; collect RGB, 2D covariance, depth plane, and normal information of the relevant primitives for each block; calculate the probability density function pixel by pixel, and filter out primitives with opacity below a preset value; calculate the depth of the primitive at each pixel based on the reference plane; blend RGB, depth, and normal based on transparency, and normalize the normal blending result; The post-processing module calculates the screen space normal based on the depth image and normalizes the depth image.

6. The multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization according to claim 5, characterized in that: The model training module includes: Loss calculation module, calculates RGB loss based on rendering results and reference images , add the self-supervised Laplace loss function of cumulative normal and the 3D rotation loss R based on the screen space normal SN , and the shape constraint regularization term R EC ; The difference in local normals of the 3D Gaussian sputtering model is calculated using the Laplace operator. This difference should be minimized during the optimization process of the 3D Gaussian sputtering model. The edge of the 3D Gaussian sputtering model is approximately detected using the depth image gradient information as the shielding area. Based on the viewing angle and depth information, the camera space coordinates corresponding to the pixel are obtained. The radial vector cross product between the three-dimensional points corresponding to adjacent pixels in the x and y directions of the pixel screen is performed to obtain the normal direction of the pixel. The normal direction is converted to the world space coordinate system according to the camera transformation matrix, and the supervised difference is calculated with the normal image based on the screen space. The supervised difference is calculated using a three-dimensional rotation loss function to adapt to the problem that the rasterization result of the three-dimensional Gaussian primitive is non-orientable. To constrain the shape of the primitive, the reconstruction module adds a shape constraint regularization term R EC , which is used to guide the primitives to change to a flat and uniform shape: The optimization objective loss function of the reconstruction module is: ; is the local gradient mask of the depth image, is the self-supervised Laplace loss function The weight factor, is the three-dimensional rotation loss R SN The weight factor of The gradient return includes: calculating the transparency gradient, depth reference plane gradient, normal gradient, and RGB gradient of each primitive based on the depth, normal, RGB gradient of each pixel and the transparency of each primitive; calculating the corresponding two-dimensional covariance gradient based on the transparency gradient of each primitive; calculating the rotation, displacement, and scaling gradient of each primitive based on the two-dimensional covariance gradient, depth reference plane gradient, and normal gradient of each primitive; and calculating the spherical harmonic function gradient based on the RGB gradient of each primitive. The model densification includes: classifying primitives into large primitives and small primitives according to the primitive y-axis length and scene size, cloning the gradient of the small primitive and splitting the large primitive; eliminating primitives whose primitive size exceeds a preset value; eliminating primitives whose z-axis length is greater than the x-axis length by a preset multiple and eliminating primitives whose eccentricity is greater than a preset value; Use the visualization window to mark and adjust the 3D Gaussian sputtering model: View the effects of the 3D Gaussian sputtering model in real time during training using the normal, screen space normal, and depth visualization options. Label the model with different types so that different types of primitives in the model are rendered differently.

7. An electronic device, characterized in that: Including the multi-view image reconstruction device based on three-dimensional Gaussian sputtering characterization as described in claim 5 or 6, the electronic device is connected to an information display device, and the information display device is used to display the grid model, the rendered depth image and the RGB image according to the display parameters and attributes set by the user or through an artificial intelligence model.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the steps of the multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization according to any one of claims 1 to 4.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multi-view image reconstruction method based on three-dimensional Gaussian sputtering characterization described in any one of claims 1 to 4 are implemented.

Images