A three-dimensional reconstruction method based on learning perspective-dependent two-dimensional splash kernel

Abstract

The application discloses a three-dimensional reconstruction method based on a learning perspective-dependent two-dimensional splat kernel. The application takes a boundary ellipsoid as an initialization basis, uses a multi-layer perception machine perceived by the boundary ellipsoid to perform a neural three-dimensional-to-two-dimensional projection, maps a potential vector representing a three-dimensional kernel into a two-dimensional kernel potential vector, directly models an integral result, and avoids expensive explicit line integral calculation in a traditional method; a decoder is used to convert the obtained two-dimensional potential vector into a radial symmetric two-dimensional kernel based on Mahalanobis distance, as a final splat shape, and learned perspective dependence is introduced to optimize reconstruction details; finally, the generated shape is transmitted into a standard splat pipeline for shading and mixing. The application solves the problems of limited expression capability of a fixed Gaussian kernel and complex general primitive numerical integral calculation in the prior art, realizes automatic discovery of an optimal splat kernel, and improves reconstruction quality and representation efficiency of a three-dimensional scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method based on a learned viewpoint-related two-dimensional splash kernel. Background Technology

[0002] With the rapid development of computer graphics and computer vision technologies, image-based scene representation and rendering have achieved great success in the field of new perspective synthesis. Among them, 3D Gaussian Splatting (3DGS) technology, as an emerging representation method, utilizes anisotropic 3D Gaussian kernels as basic primitives and models the scene through scaling, rotation, and translation. In the rendering stage, the density of these 3D primitives is integrated along the projection direction using closed-form equations and splashed onto the screen using an efficient tile-based differentiable renderer. Finally, relevant parameters are optimized based on multi-view input images. The shape of the splatting kernel is a key factor determining reconstruction quality and representation efficiency. For example, in the 2D case, using a set of squares to represent rectangles is often more efficient than using a Gaussian kernel. Therefore, how to select or generate the optimal splatting kernel has become a research hotspot.

[0003] In recent years, research on improving splatter primitive shapes has mainly fallen into two categories: analytical kernel methods and learning-based representation methods. Analytical kernel methods typically replace the standard Gaussian kernel in 3DGS with other manually defined explicit voxel primitives, such as the generalized exponential distribution, Student's t-distribution, or deformable Beta kernel, or use view-independent 2D kernels such as linear kernels or B-splines. These kernels usually have analytical formulas to calculate line integrals, thus ensuring rendering performance. Learning-based methods attempt to automatically learn primitives directly from data, for example, by approximating primitives by optimizing the vertices of convex shapes, or by using specific types of shallow neural networks (such as SplatNet) to represent primitives and derive their integrals. In addition, research on planar primitives has also introduced Hermite polynomials, Gabor filters, etc., as analytical 2D kernels, or added details through learnable textures.

[0004] However, existing splash kernel generation and rendering techniques still face many unresolved problems. On the one hand, the expressive power of analytical kernel methods is severely limited; their shape variations depend entirely on manually designed mathematical equations, making it impossible to fully adapt to complex and varied input scenarios in a data-driven manner, resulting in poor generalization when handling non-standard shapes. On the other hand, for methods attempting to automatically learn general 3D primitives, the main obstacle lies in the calculation of line integrals. Except for a few special cases such as Gaussian or Student's t-distribution, there is no closed-form line integral equation for general 3D density fields. To avoid expensive numerical integration approximations, existing techniques often have to restrict primitives to planar shapes or limit them to specific types of neural network structures, which greatly sacrifices the flexibility and accuracy of representation. Furthermore, while some existing techniques implicitly model voxel primitives as 2D kernels, they often ignore the 3D consistency that 2D kernels should possess across different viewpoints. This viewpoint-independent processing leads to suboptimal reconstruction results. Currently, there is a lack of a technical solution that can systematically and automatically learn general splash kernels, avoiding complex line integral calculations while introducing viewpoint dependence to improve reconstruction quality. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel.

[0006] The objective of this invention is achieved through the following technical solution: a three-dimensional reconstruction method based on a learning-view-related two-dimensional splash kernel, the method comprising the following steps: First, a set of 3D volumetric primitives for the scene is constructed. Each primitive is defined as an attribute tuple, which contains explicit geometric attributes for defining the spatial extent, and a learnable 3D kernel latent vector for encoding density distribution features. The explicit geometric attributes of primitives are projected onto a two-dimensional image plane. The resulting two-dimensional boundary ellipse is used as a geometric proxy for primitive depth sorting and tile partitioning. Invalid pixels are removed to obtain the projection center of the primitive on the image plane. A neural projection model is constructed. The three-dimensional kernel latent vector and the geometric state features of the primitives under the current view are input into the neural projection model, and the two-dimensional kernel latent vector with view dependence is output. A kernel decoder model is constructed to generate the shape of the 2D splash kernel. The model receives the 2D kernel latent vector and the normalized distance of the pixel relative to the primitive projection center, outputs the opacity value of the pixel through nonlinear mapping, and accelerates rendering by using presampling and numerical interpolation. A photometric loss function is constructed between the rendered image and the real image. The backpropagation algorithm is used to jointly optimize the explicit geometric properties of primitives, the 3D kernel latent vector, and the network parameters of the neural projection model and the kernel decoder model using the loss function. 3D reconstruction is achieved using the trained primitives and the network.

[0007] Furthermore, the graphic elements specifically include: in, Indicates the center coordinates of the element. Indicates the scaling factor. Represents a rotation quaternion; Indicates color coefficient; Indicates opacity; Representing a dimension as Learnable three-dimensional kernel latent vectors.

[0008] Furthermore, the projection of the explicit geometric attributes of primitives onto the two-dimensional image plane includes: First, a three-dimensional covariance matrix is ​​constructed based on the scaling factor and rotation quaternion of the primitives. Then, using the observation transformation matrix of the current viewpoint and the affine approximation Jacobian matrix of the projective transformation, the three-dimensional covariance matrix is ​​transformed into a two-dimensional covariance matrix on the image plane. This two-dimensional covariance matrix defines the shape and orientation of the two-dimensional ellipse after projection. At the same time, the center coordinates of the primitives are also mapped to two-dimensional center coordinates on the image plane through the same viewpoint transformation and projective transformation.

[0009] Furthermore, the neural projection model is a multilayer perceptron structure, and the specific inputs are the three-dimensional kernel latent vector of the primitive, the coordinates of the primitive center in the camera coordinate system for providing relative spatial position perception, the scaling factor for providing scale perception, and the rotation matrix in the camera coordinate system for providing primitive orientation and viewpoint perception.

[0010] Furthermore, the normalized distance of the pixel relative to the primitive projection center includes: any coordinate on the pixel plane. Relative to the projection center The squared Mahalanobis distance is calculated using the following formula: in, Represents the square of the Mahalanobis distance; A two-dimensional coordinate vector representing the current pixel; Indicates the coordinates of the projection center of the graphic element onto the image plane; This represents the inverse of the two-dimensional covariance matrix.

[0011] Furthermore, the kernel decoder model is a multilayer perceptron structure. Before optimizing for a specific scene, the kernel decoder model is pre-trained, and the network weights are trained using a function of the given opacity value and Mahalanobis distance as the objective.

[0012] Furthermore, the method of accelerating rendering using presampling and numerical interpolation includes: Before rendering each primitive, the normalized squared Mahalanobis distance is used. Select within the range A discrete pre-sampling sequence is constructed from 10 sample points; Regarding the first One sample point, The opacity value is calculated using a kernel decoder, and the calculation formula is as follows: in, Indicates the first Pre-calculated opacity values ​​for each sample point; Represents the kernel decoder model. Indicates the first The squared Mahalanobis distances corresponding to each sample point; This represents the number of sampling points; During the rasterization stage, for any pixel within the boundary ellipse, its actual Mahalanobis distance is first calculated. Subsequently, based on the position of this distance in the presampled sequence, the final opacity is calculated using a numerical interpolation algorithm; first, determine... Interval index falling in the presampled sequence , making Then, the calculation is performed using two adjacent pre-sampling points, as shown in the following formula: in: This indicates the opacity of the pixel that will ultimately be used for rendering; and Indicates the index of two adjacent sample points in the presampled sequence; The weighting coefficients for linear interpolation are calculated as follows: ; Calculated opacity Combining the color attributes of the primitives The image is then colored, sorted by depth, and accumulated using a tile-based blending formula to generate the final rendered image.

[0013] Furthermore, the photometric loss function is: The first two items are standard image reconstruction losses. For photometric error, For structural similarity error, The weights for structural similarity error, For regularization terms related to primitive scaling and opacity, The weight of the regularization term.

[0014] Furthermore, a two-stage stability training strategy is implemented during the joint optimization process. The first phase is before training begins. In the next iteration, the neural projection model is frozen. and kernel decoder model The parameters are optimized only for the explicit geometric and appearance attributes of the primitives; The second phase is After the next iteration, the neural network parameters are unfrozen, and the attributes of all primitives in the scene and the neural network are analyzed. and The weights are jointly optimized end-to-end to minimize the loss function. .

[0015] According to another aspect of the specification, a three-dimensional reconstruction device based on a learning view-related two-dimensional splash kernel is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned three-dimensional reconstruction method based on a learning view-related two-dimensional splash kernel.

[0016] The beneficial effects of this invention are: 1. By directly learning the projected two-dimensional kernel representation, the complex mathematical operations of explicit line integrals on general primitives are avoided, significantly reducing computational complexity and avoiding dependence on specific neural network structures; 2. A view-dependent neural projection mechanism was introduced, which enabled the shape of the splash kernel to adaptively adjust with the viewpoint. Compared with the fixed-shape analytical kernel, it has a stronger scene expression ability and can reconstruct more fine details. 3. By using two-dimensional boundary ellipses as geometric proxies for sorting and culling, and combining one-dimensional contour presampling and numerical interpolation techniques, a rendering frame rate that meets the requirements of real-time interaction is achieved while maintaining high-quality reconstruction. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a three-dimensional reconstruction method based on a learning-view-related two-dimensional splash kernel provided in an embodiment of the present invention; Figure 2 A flowchart of the three-dimensional to two-dimensional projection step of the nerve in the method of the present invention is provided for embodiments of the present invention; Figure 3 A flowchart of the neural nucleus decoding step in the method of this invention is provided for embodiments of the present invention; Figure 4 These are comparative experimental diagrams of the Room scene in the embodiments of the present invention; Figure 5 These are comparative experimental images of the Bonsai scene in the embodiments of the present invention; Figure 6 These are comparative experimental diagrams of the Train scenario in the embodiments of the present invention; Figure 7 These are comparative experimental images of the Playroom scene in the embodiments of the present invention; Figure 8 This is a schematic diagram of a three-dimensional reconstruction device based on a learning perspective-related two-dimensional splash kernel in an embodiment of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] like Figure 1 As shown in the figure, the three-dimensional reconstruction method based on a learned viewpoint-related two-dimensional splash kernel provided by this invention specifically includes the following steps: Step 1: Construct and initialize a set of 3D primitives containing implicit kernel features. In this embodiment, the scene is modeled as a set of discrete 3D volumetric primitives. Each primitive is defined as an attribute tuple, denoted as... } .in , and These three parameters, representing the 3D center position, scaling factor, and rotation quaternion respectively, together define a 3D boundary ellipsoid used to define the effective geometric range of the primitive in space. The color parameters are represented using spherical harmonics in this embodiment. Indicates opacity; Defined as a learnable 3D kernel latent vector, preferably with a dimension of 5, it is used to implicitly encode the 3D density distribution features of primitives. In the initialization phase, this embodiment uses a sparse point cloud generated by the Structure for Motion Recovery (SfM) algorithm to initialize the primitive set.

[0021] Step Two: Neural 3D-to-2D Projection Based on Boundary Ellipsoid Perception. For a given camera viewpoint, this step converts 3D primitives into a splash description on a 2D image plane, specifically including two sub-steps: geometric projection and neural feature projection, as follows... Figure 2 As shown.

[0022] 2.1 Geometric Projection. In this embodiment, the ellipsoidal weighted average splatter approximation method (or referring to the standard projection process of 3DGaussian Splatting) is used to project the three-dimensional boundary ellipsoid onto the two-dimensional image plane to obtain a two-dimensional geometric proxy. This two-dimensional geometric proxy is used for depth sorting and tile binning in the rasterization stage, and invalid pixels are removed using its boundary range to reduce the amount of computation.

[0023] Specifically, firstly, based on the scaling factor of the primitives... and rotation quaternions A three-dimensional covariance matrix is ​​constructed; subsequently, using the observation transformation matrix of the current viewpoint and the affine approximation Jacobian matrix of the projective transformation, this three-dimensional covariance matrix is ​​transformed into a two-dimensional covariance matrix on the image plane. The two-dimensional covariance matrix defines the shape and orientation of the projected two-dimensional ellipse. Simultaneously, the center coordinates of the primitives are mapped to two-dimensional center coordinates on the image plane through the same viewpoint transformation and projective transformation. .

[0024] After obtaining the aforementioned two-dimensional geometric proxy (i.e., two-dimensional center coordinates and two-dimensional covariance matrix), this scheme follows the rasterization architecture of the original 3D Gaussian Splatting (3DGS) standard algorithm, sequentially performing tile partitioning and depth sorting steps. The specific process is as follows: (1) Tile Binning Process: First, the two-dimensional image plane is divided into non-overlapping pixel tiles of a fixed size (in this embodiment) (A pixel is a tile). For each 2D geometric proxy generated by projection, the boundary box of the 2D ellipse on the image plane is calculated based on its 2D center coordinates and 2D covariance matrix. Then, the boundary box is intersected with all the predefined pixel tiles. If the boundary box of the 2D ellipse covers a tile, the primitive is instantiated and assigned to the corresponding tile.

[0025] (2) Depth Sorting Process: After tile partitioning, the primitives within each tile need to be depth-sorted to ensure that the subsequent rasterization stage can perform forward volume rendering according to the correct occlusion relationships. Based on the original 3DGS approach, this scheme assigns a 64-bit composite key value to each intersecting "primitive-tile" instance. The high 32 bits of this composite key value are the index number of the tile, and the low 32 bits are the depth value of the primitive center in the current camera view space. Subsequently, the Radix Sort algorithm is used to perform a global fast sort of all primitive instances based on the above composite key value.

[0026] The two processes described above efficiently establish the valid primitives contained in each pixel patch and obtain a list of primitives arranged from near to far in depth within each patch, providing the necessary position and occlusion order prerequisites for subsequent removal of invalid pixels and forward volume rendering of neural kernels.

[0027] 2.2 Neural Feature Projection. To achieve the viewpoint relevance of the splash kernel, this invention constructs a neural projection model for boundary ellipsoid perception, denoted as... The model takes a 3D kernel latent vector and the geometric state of the boundary ellipsoid in the current camera coordinate system as input, and the calculation formula is as follows: in, The output represents a two-dimensional kernel latent vector related to the viewpoint, and its dimension is preferably set to 5; Represents the input three-dimensional kernel latent vector; It represents the coordinates of the primitive center in the camera coordinate system, used to provide relative spatial position awareness; This represents the scaling factor, used to provide scale awareness; Represents the rotation matrix in the camera coordinate system, used to provide primitive orientation and view angle perception; The neural projection model is preferably a 3-layer multilayer perceptron with a hidden layer dimension of 64 and the activation function of the hidden layer is the LeakyReLU function.

[0028] Step 3: Decoding neural nuclei based on Mahalanobis distance. This step utilizes the decoder model to decode the neural nuclei generated in Step 2. Decoded into the actual 2D splatter opacity distribution, and rendered faster using presampling and numerical interpolation, such as... Figure 3 As shown.

[0029] 3.1 Neural Kernel Decoding. To ensure rotation invariance and simplify calculations, this invention constrains the generated two-dimensional kernel to a radially symmetric shape about the Mahalanobis distance. First, arbitrary coordinates on the pixel plane are calculated. Relative to the projection center The squared Mahalanobis distance is calculated using the following formula: in, Represents the square of the Mahalanobis distance; A two-dimensional coordinate vector representing the current pixel; Indicates the coordinates of the projection center of the graphic element onto the image plane; This represents the inverse of the two-dimensional covariance matrix. Next, the kernel decoder model is run. The opacity is calculated based on the Mahalanobis distance and the two-dimensional kernel latent vector, using the following formula: in, This indicates the opacity value at that pixel location; The kernel decoder model is preferably a 4-layer multilayer perceptron with 4 hidden layers, the activation function of the hidden layers being the LeakyReLU function, and the activation function of the output layer being the Sigmoid function.

[0030] To ensure the stability and convergence speed of the subsequent optimization process, as a preferred implementation, this invention optimizes the kernel decoder model before optimizing for a specific scenario. Pre-training is performed. Specifically, the network weights are trained using a function of the given opacity value and the Mahalanobis distance as the target. In this embodiment, a one-dimensional cosine function is used as the target contour for pre-training, and its mathematical expression is: This initialization strategy enables the network to possess a reasonable physical shape representation capability from the very beginning of optimization.

[0031] 3.2 Discrete Presampling and Interpolation Accelerate Rendering. To meet the requirements of real-time rendering, this embodiment avoids performing neural network inference for each pixel covered by the boundary ellipse. Instead, it proposes an accelerated rendering strategy based on one-dimensional contour presampling. This strategy consists of two sub-steps.

[0032] a) Discrete presampling. Before rendering each primitive, the normalized squared Mahalanobis distance is used for presampling. Select within the range A discrete presampling sequence is constructed from n sample points. In this preferred embodiment, a uniform sampling strategy is adopted. For the nth sample point... sample points ( The opacity value is calculated using a kernel decoder, and the calculation formula is as follows: in, Indicates the first Pre-calculated opacity values ​​for each sample point; Indicates the first The squared Mahalanobis distances corresponding to each sample point; This represents the number of sampling points. It should be noted that... The value of can be set according to the balance requirements between accuracy and speed. In this embodiment, The preferred value is 2.

[0033] b) Interpolation accelerates rendering. During the rasterization stage, for any pixel within the boundary ellipse, its actual Mahalanobis distance is first calculated. Subsequently, based on the position of this distance in the presampled sequence, the final opacity is calculated using a numerical interpolation algorithm. As a computationally efficient preferred implementation, this embodiment employs linear interpolation. Specifically, firstly, the opacity is determined... Interval index falling in the presampled sequence , making Then, the calculation is performed using two adjacent pre-sampling points, as shown in the following formula: in: This indicates the opacity of the pixel that will ultimately be used for rendering; and Indicates the index of two adjacent sample points in the presampled sequence; The weighting coefficients for linear interpolation are calculated as follows: .

[0034] In this way, the originally complex neural network inference is transformed into extremely low-overhead memory lookups and simple linear calculations, significantly improving the rendering frame rate. Of course, those skilled in the art will understand that in other embodiments, other numerical methods such as cubic spline interpolation or nearest neighbor interpolation may also be used, all of which fall within the protection scope of this invention.

[0035] Finally, the calculated opacity Combining the color attributes of the primitives The image is then colored, sorted by depth, and accumulated using a tile-based blending formula to generate the final rendered image.

[0036] Step 4: Optimize Training. Multi-view images of the real scene are acquired as supervision signals, and model parameters are jointly optimized by comparing rendered images with real images. This invention employs a comprehensive loss function including a regularization term and a kernel-agnostic Markov chain Monte Carlo (MCMC) density control strategy. This invention also implements a two-stage stability training strategy and an exponential decay strategy for the neural network weight learning rate.

[0037] 4.1 A comprehensive loss function including a regularization term is adopted. In the optimization process, this embodiment adopts the image reconstruction loss construction method commonly used in the field of 3D Gaussian splashing (3DGS) and introduces a regularization term to constrain the geometric properties of the neural nucleus.

[0038] Specifically, the total loss function The construction references the loss design ideas in 3D Gaussian Splatting proposed by Kerbl et al. and 3D Gaussian Splatting as Markov Chain Monte Carlo proposed by Kheradmand et al. The total loss function is defined as: The first two items are standard image reconstruction losses. For photometric error, This represents the structural similarity error. In this embodiment, the weights... Set to 0.2. This is a regularization term for primitive scaling and opacity, used to prevent the implicit kernel from producing excessively large or physically unrealistic semi-transparent artifacts during optimization. The specific calculation form and weight allocation of this regularization term are detailed below. All configurations are based on open-source implementations of existing work.

[0039] 4.2 Kernel-agnostic MCMC density control strategy is adopted. Regarding the dynamic adjustment of primitive quantity and distribution, this embodiment directly adopts the "Kernel-agnostic MCMC" density control strategy (refer to the paper: Deformable Beta Splatting). Unlike the simple splitting based on gradient thresholds in standard 3DGS, this embodiment utilizes this MCMC strategy, which does not rely on a specific kernel analytical equation. Instead, it periodically performs primitive sampling operations, including cloning, splitting, and pruning, based on the primitive's gradient history and opacity state. This strategy can effectively guide primitives to high-frequency detail regions while removing redundant primitives. The specific implementation logic and pseudocode of this algorithm will not be elaborated in this embodiment.

[0040] 4.3 Implementing a two-stage stability training strategy. Given that the primitives in this invention incorporate neural projection and neural decoding models, their degrees of freedom are significantly higher than standard 3DGS, and direct joint optimization can easily lead to training instability. Therefore, this embodiment employs a two-stage training strategy: a) First Phase (Stabilization Phase): Before training begins Second iteration (preferred) Frozen neural projection model and kernel decoder model The parameters are optimized only for the explicit geometric properties (position, rotation, scaling) and appearance properties (color, opacity) of the primitives. This stage aims to provide a reasonable initial geometric layout for the neural kernels.

[0041] b) Second Phase (Joint Optimization Phase): In After the iteration, the neural network parameters are unfrozen, and the attributes of all primitives in the scene (including the implicit kernel latent vector) are analyzed. ) and neural networks and The weights are jointly optimized end-to-end to minimize .

[0042] 4.4 Implement an exponential decay strategy for the learning rate of neural network weights. For neural projection models... and kernel decoder model The preferred learning rate for its weights is an exponential decay strategy, starting from the initial value. Gradually decrease to For other explicit properties of primitives (such as position, rotation, scaling, etc.), the learning rate settings are configured with reference to the open-source implementations of existing work.

[0043] The present invention will now demonstrate the application effect of the three-dimensional reconstruction method based on the learning perspective-related two-dimensional splash kernel described in steps one to five of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0044] This invention tested its method on four standard datasets: Mip-NeRF 360, Tanks & Temples, Deep Blending, and NeRF Synthetic. The results demonstrated the novel view synthesis effects of this method compared to the 3DGS-MCMC method (see paper: 3D Gaussian Splatting as Markov Chain Monte Carlo), the DBS method (see paper: Deformable Beta Splatting), and the SSS method (see paper: 3D Student Splatting and Scooping). Examples of rendering effects in different scenarios are shown below. Figure 4 ~ Figure 7 As shown, the indicators are listed in Tables 1 to 4. Figure 4 Corresponding to the Room scenario, Figure 5 Corresponding to the Bonsai scenario, Figure 6 Corresponding to the Train scenario, Figure 7This corresponds to the Playroom scene. In Tables 1 to 4, PSNR is Peak Signal-to-Noise Ratio, SSIM is Structural Similarity, and the higher the value of either indicator, the higher the rendering quality; LPIPS is Perceptual Similarity, and the lower the value, the higher the rendering quality.

[0045] Table 1. Data results on the Mip-NeRF 360 dataset Table 2. Dataset Results on the Tanks & Temples Dataset Table 3. Data results on the Deep Blending dataset Table 4. Data results on the NeRF Synthetic dataset Corresponding to the aforementioned embodiment of a three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel, the present invention also provides an embodiment of a three-dimensional reconstruction device based on a learning perspective-related two-dimensional splash kernel.

[0046] See Figure 8 The present invention provides a three-dimensional reconstruction device based on a learning view-related two-dimensional splash kernel, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a three-dimensional reconstruction method based on a learning view-related two-dimensional splash kernel as described in the above embodiment.

[0047] The embodiment of the 3D reconstruction device based on a learning-view-related 2D splash kernel provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is an 3D reconstruction device based on a learning perspective-related 2D splash kernel provided by the present invention. (Except for...) Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0048] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0049] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0050] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel as described in the above embodiments.

[0051] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0052] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel.

[0053] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method based on a learning-perspective-related two-dimensional splash kernel, characterized in that, The method includes the following steps: First, a set of 3D volumetric primitives for the scene is constructed. Each primitive is defined as an attribute tuple, which contains explicit geometric attributes for defining the spatial extent, and a learnable 3D kernel latent vector for encoding density distribution features. The explicit geometric attributes of primitives are projected onto a two-dimensional image plane. The resulting two-dimensional boundary ellipse is used as a geometric proxy for primitive depth sorting and tile partitioning. Invalid pixels are removed to obtain the projection center of the primitive on the image plane. A neural projection model is constructed. The three-dimensional kernel latent vector and the geometric state features of the primitives under the current view are input into the neural projection model, and the two-dimensional kernel latent vector with view dependence is output. A kernel decoder model is constructed to generate the shape of the 2D splash kernel. The model receives the 2D kernel latent vector and the normalized distance of the pixel relative to the primitive projection center, outputs the opacity value of the pixel through nonlinear mapping, and accelerates rendering by using presampling and numerical interpolation. A photometric loss function is constructed between the rendered image and the real image. The backpropagation algorithm is used to jointly optimize the explicit geometric properties of primitives, the 3D kernel latent vector, and the network parameters of the neural projection model and the kernel decoder model using the loss function. 3D reconstruction is achieved using the trained primitives and the network.

2. The three-dimensional reconstruction method based on a learning-view-related two-dimensional splash kernel according to claim 1, characterized in that, The graphic elements specifically include: in, Indicates the center coordinates of the element. Indicates the scaling factor. Represents a rotation quaternion; Indicates color coefficient; Indicates opacity; Representing a dimension as Learnable three-dimensional kernel latent vectors.

3. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel as described in claim 1, characterized in that, The process of projecting explicit geometric attributes of primitives onto a two-dimensional image plane includes: First, a three-dimensional covariance matrix is ​​constructed based on the scaling factor and rotation quaternion of the primitives. Then, using the observation transformation matrix of the current viewpoint and the affine approximation Jacobian matrix of the projective transformation, the three-dimensional covariance matrix is ​​transformed into a two-dimensional covariance matrix on the image plane. This two-dimensional covariance matrix defines the shape and orientation of the two-dimensional ellipse after projection. At the same time, the center coordinates of the primitives are also mapped to two-dimensional center coordinates on the image plane through the same viewpoint transformation and projective transformation.

4. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel according to claim 1, characterized in that, The neural projection model is a multilayer perceptron structure. The specific inputs are the three-dimensional kernel latent vector of the primitive, the coordinates of the primitive center in the camera coordinate system for providing relative spatial position perception, the scaling factor for providing scale perception, and the rotation matrix in the camera coordinate system for providing primitive orientation and viewpoint perception.

5. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel according to claim 1, characterized in that, The normalized distance of the pixel relative to the primitive projection center includes: any coordinate on the pixel plane. Relative to the projection center The squared Mahalanobis distance is calculated using the following formula: in, Represents the square of the Mahalanobis distance; A two-dimensional coordinate vector representing the current pixel; Indicates the coordinates of the projection center of the graphic element onto the image plane; This represents the inverse of the two-dimensional covariance matrix.

6. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel according to claim 1, characterized in that, The kernel decoder model is a multilayer perceptron structure. Before optimizing for specific scenarios, the kernel decoder model is pre-trained, and the network weights are trained using a function of the given opacity value and Mahalanobis distance as the objective.

7. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel according to claim 1, characterized in that, The method of accelerating rendering using presampling and numerical interpolation includes: Before rendering each primitive, the normalized squared Mahalanobis distance is used. Select within the range A discrete pre-sampling sequence is constructed from 10 sample points; Regarding the first One sample point, The opacity value is calculated using a kernel decoder, and the calculation formula is as follows: in, Indicates the first Pre-calculated opacity values ​​for each sample point; Represents the kernel decoder model. Indicates the first The squared Mahalanobis distances corresponding to each sample point; This represents the number of sampling points; During the rasterization stage, for any pixel within the boundary ellipse, its actual Mahalanobis distance is first calculated. Subsequently, based on the position of this distance in the presampled sequence, the final opacity is calculated using a numerical interpolation algorithm; first, determine... Interval index falling in the presampled sequence , making Then, the calculation is performed using two adjacent pre-sampling points, as shown in the following formula: in: This indicates the opacity of the pixel that will ultimately be used for rendering; and Indicates the index of two adjacent sample points in the presampled sequence; The weighting coefficients for linear interpolation are calculated as follows: ; Calculated opacity Combining the color attributes of the primitives The image is then colored, sorted by depth, and accumulated using a tile-based blending formula to generate the final rendered image.

8. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel according to claim 1, characterized in that, The photometric loss function is: The first two items are standard image reconstruction losses. For photometric error, For structural similarity error, The weights for structural similarity error, For regularization terms related to primitive scaling and opacity, The weight of the regularization term.

9. The three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel according to claim 1, characterized in that, The joint optimization process implements a two-stage stability training strategy. The first phase is before training begins. In the next iteration, the neural projection model is frozen. and kernel decoder model The parameters are optimized only for the explicit geometric and appearance attributes of the primitives; The second phase is After the next iteration, the neural network parameters are unfrozen, and the attributes of all primitives in the scene and the neural network are analyzed. and The weights are jointly optimized end-to-end to minimize the loss function. .

10. A three-dimensional reconstruction device based on a learning perspective-related two-dimensional splash kernel, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a three-dimensional reconstruction method based on a learning perspective-related two-dimensional splash kernel as described in any one of claims 1-9.

Images