Training methods, 3D reconstruction methods, devices, storage media, and program products for 3D Gaussian point rendering models.

Abstract

This disclosure relates to a training method, a 3D reconstruction method, an apparatus, a storage medium, and a program product for a 3D Gaussian point rendering model. The method includes: dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of Gaussian points in the Gaussian point cloud corresponding to the target scene in 3D space; inputting camera parameters of the viewpoint to be predicted into the 3D Gaussian point rendering model corresponding to the target scene, and outputting a rendered image corresponding to the viewpoint to be predicted through the 3D Gaussian point rendering model; calculating the value of a loss function corresponding to the 3D Gaussian point rendering model based on the rendered image and a reference image corresponding to the viewpoint to be predicted; and updating the parameters of the Gaussian points in the Gaussian point cloud based on the value of the loss function, wherein the parameters of Gaussian points in the same Gaussian point cluster are stored in contiguous video memory. This disclosure can significantly optimize video memory access efficiency, reduce fragmented access to video memory, and thus reduce video memory pressure.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method for a three-dimensional Gaussian point drawing model, a three-dimensional reconstruction method, a training device for a three-dimensional Gaussian point drawing model, a three-dimensional reconstruction device, a non-volatile computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of computer graphics and 3D reconstruction technology, efficiently converting 2D images into 3D models has become a research hotspot. In recent years, 3D reconstruction technology has made significant progress through methods such as structured light scanning, laser scanning, multi-view stereo vision (MVS), and depth cameras. However, these traditional methods still have limitations in terms of efficiency and flexibility. In recent years, neural network-based reconstruction methods such as NeRF (Neural Radiance Fields) and 3DGS (3D Gaussian Splatting) have gradually emerged, providing new ideas for efficient 3D reconstruction. Both methods can be trained using a single video image to obtain a parametric representation of the scene in the video. After training, the scene can be observed from any angle, thus achieving an effective conversion from 2D video to a 3D scene.

[0003] NeRF is a neural network-based 3D scene representation method that predicts the color and density of any point in the scene by encoding the scene into a continuous neural network. Although NeRF can achieve high-quality 3D reconstruction, its rendering process requires multiple forward propagations of the neural network for each pixel, resulting in extremely high computational complexity, especially in complex scenes. The training and rendering times are too long, placing extremely high demands on computing resources.

[0004] 3DGS is a more efficient 3D reconstruction method that approximates the scene's color and density fields by representing the scene as a series of semi-transparent Gaussian point clouds. Because Gaussian point clouds can be quickly projected into 2D space through simple mathematical transformations, 3DGS has a significant advantage in rendering speed. However, despite 3DGS's excellent rendering efficiency, its training process still has room for improvement, especially in large-scale scenes, where training speed and GPU (Graphics Processing Unit) memory usage still need optimization. Summary of the Invention

[0005] In view of this, this disclosure provides a training technique for a three-dimensional Gaussian point rendering model.

[0006] According to one aspect of this disclosure, a method for training a three-dimensional Gaussian point rendering model is provided, comprising:

[0007] Based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, the Gaussian point cloud is divided into multiple Gaussian point clusters;

[0008] The camera parameters of the viewpoint to be predicted are input into the three-dimensional Gaussian point rendering model corresponding to the target scene, and the rendered image corresponding to the viewpoint to be predicted is output through the three-dimensional Gaussian point rendering model.

[0009] Based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted, calculate the value of the loss function corresponding to the 3D Gaussian point rendering model;

[0010] Based on the value of the loss function, the parameters of the Gaussian points in the Gaussian point cloud are updated, wherein the parameters of the Gaussian points in the same Gaussian point cluster are stored in contiguous memory space.

[0011] In one possible implementation, dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space includes:

[0012] Every N training cycles, the Gaussian point cloud is divided into multiple Gaussian point clusters based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, where N is an integer greater than or equal to 1.

[0013] In one possible implementation, before dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, the method further includes:

[0014] Every N training cycles, the sum of the gradients of the parameters of the Gaussian points in the Gaussian point cluster is calculated.

[0015] In response to the magnitude of the sum of the gradients being greater than or equal to a preset threshold, a new Gaussian point is added to the Gaussian point cluster based on the sum of the gradients.

[0016] In one possible implementation, before dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, the method further includes:

[0017] Delete the cluster of Gaussian points that have been removed by view frustum pruning for N consecutive training cycles.

[0018] In one possible implementation, the method further includes:

[0019] For any Gaussian point in the Gaussian point cloud, every N training epochs, the transparency of the Gaussian point is reset based on its visibility in those N training epochs.

[0020] In one possible implementation, outputting the rendered image corresponding to the viewpoint to be predicted through the three-dimensional Gaussian point rendering model includes:

[0021] Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frustum-clipped.

[0022] Project the Gaussian spheres in the Gaussian point cloud that have not been clipped by the view frustum into the two-dimensional space of the screen;

[0023] Based on the Gaussian projection results, a visibility table corresponding to the viewpoint to be predicted is created;

[0024] Based on the visibility table, the visible Gaussian points in the Gaussian point cloud are rasterized to obtain the rendered image corresponding to the viewpoint to be predicted.

[0025] In one possible implementation, the step of frustum clipping of the Gaussian point cloud based on the camera parameters of the viewpoint to be predicted includes:

[0026] Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frusto-clipping is performed on the Gaussian point cluster as a unit.

[0027] In one possible implementation, after the view frustum clipping of the Gaussian point cloud and before projecting the Gaussian points in the Gaussian point cloud that were not removed by the view frustum clipping into the two-dimensional space of the screen, the method further includes:

[0028] The parameters of each Gaussian point cluster in the Gaussian point cloud that has not been removed by view frustum pruning are copied to a contiguous video memory space.

[0029] In one possible implementation, creating the visibility table corresponding to the viewpoint to be predicted based on the Gaussian projection result includes:

[0030] For each two-dimensional Gaussian sphere projected onto the screen in the Gaussian projection result, a bounding box corresponding to the two-dimensional Gaussian sphere is created according to a preset transparency threshold.

[0031] The visibility of the two-dimensional Gaussian sphere to each screen region is determined based on the bounding box corresponding to the two-dimensional Gaussian sphere.

[0032] Based on the visibility of the two-dimensional Gaussian sphere to each screen region, a visibility table corresponding to the viewpoint to be predicted is created.

[0033] In one possible implementation, updating the parameters of the Gaussian points in the Gaussian point cloud based on the value of the loss function includes:

[0034] The gradient value of each Gaussian point in the Gaussian point cloud is calculated using at least one thread block based on the value of the loss function;

[0035] For any thread, the gradient value of the Gaussian point calculated by the thread is written to the shared memory corresponding to the thread block to which the thread belongs;

[0036] For any thread block, in the shared memory corresponding to the thread block, the gradient value of the Gaussian point calculated by the thread in the thread block is reduced to obtain the gradient reduction result corresponding to the thread block.

[0037] For any thread block, the gradient reduction result corresponding to the thread block is synchronized to the global memory through one of the threads in the thread block;

[0038] Based on the gradient reduction results of the Gaussian points in the Gaussian point cloud stored in the global memory, the parameters of the Gaussian points in the Gaussian point cloud are updated.

[0039] In one possible implementation, writing the gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs, for any given thread, includes:

[0040] For any thread, write the non-zero gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs.

[0041] According to another aspect of this disclosure, a three-dimensional reconstruction method is provided, comprising:

[0042] Obtain camera parameters from the target's perspective;

[0043] The camera parameters of the target viewpoint are input into a pre-trained 3D Gaussian point rendering model, and the rendered image corresponding to the target viewpoint is output through the 3D Gaussian point rendering model. The 3D Gaussian point rendering model is trained using the training method of the 3D Gaussian point rendering model.

[0044] According to another aspect of this disclosure, a training apparatus for a three-dimensional Gaussian point rendering model is provided, comprising:

[0045] The partitioning module is used to divide the Gaussian point cloud into multiple Gaussian point clusters based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space.

[0046] The first prediction module is used to input the camera parameters of the viewpoint to be predicted into the three-dimensional Gaussian point rendering model corresponding to the target scene, and output the rendered image corresponding to the viewpoint to be predicted through the three-dimensional Gaussian point rendering model.

[0047] The calculation module is used to calculate the value of the loss function corresponding to the three-dimensional Gaussian point rendering model based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted.

[0048] An update module is used to update the parameters of Gaussian points in the Gaussian point cloud according to the value of the loss function, wherein the parameters of Gaussian points in the same Gaussian point cluster are stored in contiguous memory space.

[0049] In one possible implementation, the partitioning module is used for:

[0050] Every N training cycles, the Gaussian point cloud is divided into multiple Gaussian point clusters based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, where N is an integer greater than or equal to 1.

[0051] In one possible implementation, the device further includes:

[0052] The statistics module is used to calculate the sum of the gradients of the parameters of the Gaussian points in the Gaussian point cluster every N training cycles.

[0053] A new module is added to the Gaussian point cluster in response to the sum of gradients having a magnitude greater than or equal to a preset threshold.

[0054] In one possible implementation, the device further includes:

[0055] The deletion module is used to delete Gaussian point clusters that have been removed by view frustum pruning for N consecutive training epochs.

[0056] In one possible implementation, the device further includes:

[0057] The reset module is used to reset the transparency of any Gaussian point in the Gaussian point cloud every N training cycles, based on the visibility of the Gaussian point in those N training cycles.

[0058] In one possible implementation, the first prediction module is used to:

[0059] Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frustum-clipped.

[0060] Project the Gaussian spheres in the Gaussian point cloud that have not been clipped by the view frustum into the two-dimensional space of the screen;

[0061] Based on the Gaussian projection results, a visibility table corresponding to the viewpoint to be predicted is created;

[0062] Based on the visibility table, the visible Gaussian points in the Gaussian point cloud are rasterized to obtain the rendered image corresponding to the viewpoint to be predicted.

[0063] In one possible implementation, the first prediction module is used to:

[0064] Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frusto-clipping is performed on the Gaussian point cluster as a unit.

[0065] In one possible implementation, the device further includes:

[0066] The parameter stitching module is used to copy the parameters of each Gaussian point cluster in the Gaussian point cloud that has not been removed by view frustum clipping to a contiguous video memory space.

[0067] In one possible implementation, the first prediction module is used to:

[0068] For each two-dimensional Gaussian sphere projected onto the screen in the Gaussian projection result, a bounding box corresponding to the two-dimensional Gaussian sphere is created according to a preset transparency threshold.

[0069] The visibility of the two-dimensional Gaussian sphere to each screen region is determined based on the bounding box corresponding to the two-dimensional Gaussian sphere.

[0070] Based on the visibility of the two-dimensional Gaussian sphere to each screen region, a visibility table corresponding to the viewpoint to be predicted is created.

[0071] In one possible implementation, the update module is used to:

[0072] The gradient value of each Gaussian point in the Gaussian point cloud is calculated using at least one thread block based on the value of the loss function;

[0073] For any thread, the gradient value of the Gaussian point calculated by the thread is written to the shared memory corresponding to the thread block to which the thread belongs;

[0074] For any thread block, in the shared memory corresponding to the thread block, the gradient value of the Gaussian point calculated by the thread in the thread block is reduced to obtain the gradient reduction result corresponding to the thread block.

[0075] For any thread block, the gradient reduction result corresponding to the thread block is synchronized to the global memory through one of the threads in the thread block;

[0076] Based on the gradient reduction results of the Gaussian points in the Gaussian point cloud stored in the global memory, the parameters of the Gaussian points in the Gaussian point cloud are updated.

[0077] In one possible implementation, the update module is used to:

[0078] For any thread, write the non-zero gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs.

[0079] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, comprising:

[0080] The acquisition module is used to acquire camera parameters from the target's viewpoint;

[0081] The second prediction module is used to input the camera parameters of the target viewpoint into a pre-trained three-dimensional Gaussian point rendering model, and output the rendered image corresponding to the target viewpoint through the three-dimensional Gaussian point rendering model. The three-dimensional Gaussian point rendering model is trained by the three-dimensional reconstruction device.

[0082] According to another aspect of this disclosure, a training apparatus for a three-dimensional Gaussian point drawing model is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described training method for a three-dimensional Gaussian point drawing model.

[0083] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional reconstruction method described above.

[0084] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0085] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0086] In this embodiment, the Gaussian point cloud is divided into multiple Gaussian point clusters based on the positions of Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space. The camera parameters of the viewpoint to be predicted are input into the three-dimensional Gaussian point rendering model corresponding to the target scene. The rendered image corresponding to the viewpoint to be predicted is output through the three-dimensional Gaussian point rendering model. The value of the loss function corresponding to the three-dimensional Gaussian point rendering model is calculated based on the rendered image and the reference image corresponding to the viewpoint to be predicted. The parameters of the Gaussian points in the Gaussian point cloud are updated based on the value of the loss function. The parameters of the Gaussian points in the same Gaussian point cluster are stored in contiguous video memory space, which significantly optimizes the access efficiency of video memory, reduces fragmented access of video memory, and reduces the pressure on video memory. This can significantly improve the training speed of the three-dimensional Gaussian point rendering model and reduce video memory usage, making it suitable for large-scale and complex scenes.

[0087] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0088] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0089] Figure 1 A flowchart illustrating the training method for a three-dimensional Gaussian point rendering model provided in an embodiment of this disclosure is shown.

[0090] Figure 2 This diagram illustrates a training method for a three-dimensional Gaussian point rendering model provided in an embodiment of the present disclosure.

[0091] Figure 3 A block diagram of a training apparatus for a three-dimensional Gaussian point rendering model provided in an embodiment of this disclosure is shown.

[0092] Figure 4 This is a block diagram of a training device 1900 for a three-dimensional Gaussian point drawing model, according to an exemplary embodiment. Detailed Implementation

[0093] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0094] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0095] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0096] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0097] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0098] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0099] Although 3DGS (3D Gaussian Splatting) is superior to NeRF (Neural Radiance Fields) in rendering efficiency, its training process still faces the following challenges:

[0100] Training speed bottleneck: Traditional 3DGS training methods are inefficient in large-scale scenarios and take a long time to train, making it difficult to meet the requirements of real-time performance and high efficiency.

[0101] GPU memory pressure: When processing complex scenes, 3DGS requires a large amount of GPU (Graphics Processing Unit) memory, which limits its application in resource-constrained environments.

[0102] Waste of computational resources: The 3DGS training method in related technologies has a large number of non-continuous accesses and atomic operation bottlenecks in the process of memory reading and writing and computation, which leads to waste of computational resources and affects the overall training efficiency.

[0103] To address the technical problems described above, this disclosure provides a training method for a 3D Gaussian point rendering model. The method divides the Gaussian point cloud into multiple Gaussian point clusters based on the positions of Gaussian points in the 3D space corresponding to the target scene. Camera parameters of the viewpoint to be predicted are input into the 3D Gaussian point rendering model corresponding to the target scene. The model outputs a rendered image corresponding to the viewpoint to be predicted. Based on the rendered image and a reference image corresponding to the viewpoint, the loss function of the 3D Gaussian point rendering model is calculated, and the parameters of the Gaussian points in the Gaussian point cloud are updated according to the loss function value. The parameters of Gaussian points within the same Gaussian point cluster are stored in contiguous video memory, significantly optimizing video memory access efficiency, reducing fragmented access, and thus lowering video memory pressure. This significantly improves the training speed of the 3D Gaussian point rendering model and reduces video memory usage, making it suitable for large-scale and complex scenes.

[0104] The training method for the three-dimensional Gaussian point rendering model provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0105] Figure 1 A flowchart illustrating a training method for a 3D Gaussian point rendering model provided in an embodiment of this disclosure is shown. In one possible implementation, the execution entity of the training method for the 3D Gaussian point rendering model can be a training device for the 3D Gaussian point rendering model. For example, the training method for the 3D Gaussian point rendering model can be executed by a terminal device, a server, or other electronic equipment. The terminal device can be user equipment (UE), a user terminal, a terminal, or a computing device, etc. In some possible implementations, the training method for the 3D Gaussian point rendering model can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the training method for the three-dimensional Gaussian point rendering model includes steps S11 to S14.

[0106] In step S11, the Gaussian point cloud is divided into multiple Gaussian point clusters according to the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space.

[0107] In step S12, the camera parameters of the viewpoint to be predicted are input into the three-dimensional Gaussian point rendering model corresponding to the target scene, and the rendered image corresponding to the viewpoint to be predicted is output through the three-dimensional Gaussian point rendering model.

[0108] In step S13, the value of the loss function corresponding to the three-dimensional Gaussian point rendering model is calculated based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted.

[0109] In step S14, the parameters of the Gaussian points in the Gaussian point cloud are updated according to the value of the loss function, wherein the parameters of the Gaussian points in the same Gaussian point cluster are stored in contiguous memory space.

[0110] In this embodiment of the disclosure, the target scene can represent a physical space environment that requires 3D reconstruction or the generation of a new perspective. The target scene can consist of a series of 3D objects, surfaces, and backgrounds, and can be captured and reconstructed using multi-view image data. The target scene can be an indoor environment (such as a room, office, studio, etc.) or an outdoor environment (such as a street, building exterior, natural landscape, etc.).

[0111] In a 3D Gaussian point rendering model, the target scene is represented as a set of Gaussian points, each of which can contain parameters such as position, color, and transparency. These Gaussian points together constitute a Gaussian point cloud, used to approximate the color and density field of the target scene.

[0112] To optimize the training process, embodiments of this disclosure divide the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in three-dimensional space. Specifically, spatially adjacent Gaussian points can be grouped into the same Gaussian point cluster. This division is based on the assumption that spatially adjacent Gaussian points are likely to be seen simultaneously during rendering.

[0113] In the training of traditional 3D Gaussian point rendering models, the parameters of Gaussian points (such as position, color, and transparency) may be scattered across different locations in video memory, resulting in discontinuous memory access, increased I / O (Input / Output) overhead, and reduced computational efficiency. This embodiment stores the parameters of Gaussian points within the same Gaussian point cluster in contiguous video memory. This contiguous storage method allows the GPU to access this data more efficiently in subsequent calculations (such as view frustum clipping and rasterization), reducing random access to video memory, improving cache hit rate, and thus significantly improving computational efficiency.

[0114] Therefore, by dividing Gaussian points into clusters and optimizing storage, the access mode of GPU memory changes from random to contiguous, reducing I / O bottlenecks and improving the efficiency of data reading and computation, thereby accelerating training speed. Contiguous storage reduces GPU memory fragmentation and management overhead, making GPU memory usage more efficient, especially in large-scale scenarios, significantly reducing GPU memory consumption. Due to the improved GPU memory access efficiency, the embodiments of this disclosure can better handle large-scale and complex scenarios, avoiding the problems of insufficient GPU memory or slow training speed in large-scale scenarios encountered by traditional methods.

[0115] In one possible implementation, each Gaussian point cluster can contain 64 Gaussian points.

[0116] Of course, those skilled in the art can flexibly set the number of Gaussian points in the Gaussian point cluster according to the actual application scenario requirements, and no limitation is made here.

[0117] In one possible implementation, dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space includes: every N training cycles, dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, where N is an integer greater than or equal to 1.

[0118] In this implementation, the partitioning of the Gaussian point cloud during training is not fixed but dynamically adjusted. Specifically, after every N training epochs, the system can re-partition the Gaussian point cloud into multiple Gaussian point clusters based on the latest positions of the Gaussian points in 3D space. Here, N is an integer greater than or equal to 1, representing the frequency of the partitioning operation. For example, if N = 10, it means that the Gaussian point clusters will be re-partitioned every 10 training epochs.

[0119] The clustering is based on the position of Gaussian points in 3D space. Spatially adjacent Gaussian points are grouped into the same Gaussian point cluster. This partitioning method ensures that the Gaussian points in each cluster are spatially closely related, thus enabling efficient processing in subsequent rendering and computation.

[0120] During training, the positions and parameters of Gaussian points are constantly updated (e.g., through gradient descent optimization). Therefore, the distribution of Gaussian points in 3D space may change. By repartitioning the clusters every N training epochs, we can ensure that the cluster partitioning always remains consistent with the current Gaussian point distribution, thus maintaining efficient computation and memory access. After repartitioning, the Gaussian point parameters within the same cluster are stored in contiguous memory space. This contiguous storage method reduces random memory access, improves cache hit rate, and thus significantly improves training efficiency. Dynamic cluster partitioning avoids the decrease in memory access efficiency caused by changes in Gaussian point positions, ensuring that the training process remains highly efficient.

[0121] As an example of this implementation, spatial partitioning algorithms (such as BVH (Bounding Volume Hierarchy) trees) can be used to efficiently divide Gaussian point clouds into multiple clusters. A BVH tree is a spatial indexing structure that can quickly find adjacent Gaussian points in space.

[0122] In one possible implementation, before dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, the method further includes: every N training cycles, calculating the sum of gradients of the parameters of the Gaussian points in the Gaussian point clusters; in response to the magnitude of the sum of gradients being greater than or equal to a preset threshold, adding new Gaussian points to the Gaussian point clusters based on the sum of gradients.

[0123] During the training of the 3D Gaussian point rendering model, the parameters of the Gaussian points (such as position, color, and transparency) are optimized using gradient descent. The gradient represents the direction and magnitude of parameter updates, reflecting the contribution of the current Gaussian point to the loss function. In this implementation, after every N training epochs, the system can sum the gradients of all Gaussian points in each cluster. Specifically, the gradient summation is the accumulation of gradients from all Gaussian points within the same cluster, resulting in a total gradient vector.

[0124] The magnitude of the sum of gradients (i.e., the length of the gradient) reflects the overall contribution of the Gaussian points in the Gaussian point cluster to the loss function. A larger magnitude indicates that the Gaussian points in the cluster do not adequately fit the current scene and require further optimization. In this implementation, a preset threshold can be set to determine if the gradient magnitude is sufficiently large. If the gradient magnitude is greater than or equal to this preset threshold, it means that the current Gaussian points in the cluster are insufficient to represent the scene in that region well, and more Gaussian points need to be added to improve the model's expressive power.

[0125] When the gradient magnitude is greater than or equal to a preset threshold, new Gaussian points can be added to the Gaussian point cluster based on the direction and magnitude of the gradient sum. The parameters (such as position and color) of the newly added Gaussian points can be initialized based on the gradient information to ensure they can better fit the scene in the current region. The purpose of adding Gaussian points is to improve the model's expressive ability in regions with large gradients, thereby better fitting the details of the scene and reducing rendering errors.

[0126] During training, different regions of the target scene may have varying degrees of complexity. By dynamically adding Gaussian points, the 3D Gaussian point rendering model can adaptively increase the density of Gaussian points based on gradient information, thereby better representing complex regions. Adding Gaussian points can improve the model's ability to capture details, reduce rendering errors, and thus improve the quality of the rendered image. Furthermore, by adding Gaussian points only when needed, unnecessary computation and memory overhead are avoided, ensuring that the training process remains efficient at all times.

[0127] In one possible implementation, before dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, the method further includes: deleting Gaussian point clusters that have been removed by view frustum pruning for N consecutive training cycles.

[0128] During the rendering process, the system can remove Gaussian points or clusters of Gaussian points located outside the view frustum, based on the current camera's viewpoint and view frustum range. These removed Gaussian points do not contribute to the rendered image from the current viewpoint, so their calculation can be skipped, thereby improving rendering efficiency.

[0129] During training, the system can record the number of times each Gaussian point cluster is culled by view frustum pruning. If a Gaussian point cluster is culled for N consecutive training epochs, it indicates that the cluster contributes very little to the rendering results in the current training phase, and may even be ineffective. For these Gaussian point clusters culled for N consecutive training epochs, the system can remove them from the Gaussian point cloud. This removal operation may include releasing the GPU memory occupied by these Gaussian point clusters and excluding them from subsequent calculations.

[0130] As an example of this implementation, the system can maintain a counter for each Gaussian point cluster, recording the number of times it has been removed by view frustum pruning. These counters are updated after each training cycle. If a cluster's counter reaches N (i.e., it has been removed for N consecutive training cycles), a deletion operation is triggered.

[0131] This implementation frees up GPU memory and reduces memory usage by deleting Gaussian point clusters that have not been used for a long time, thus freeing up more resources for other computing tasks. Deleting invalid Gaussian point clusters reduces the amount of data that needs to be processed in subsequent calculations, thereby improving training and rendering efficiency. Furthermore, the distribution of Gaussian points in the scene may change during training. By deleting Gaussian point clusters that have not been used for a long time, the system can dynamically adjust the distribution of the Gaussian point cloud, ensuring that it always remains consistent with the needs of the current scene.

[0132] In one possible implementation, the method further includes: for any Gaussian point in the Gaussian point cloud, every N training cycles, resetting the transparency of the Gaussian point based on its visibility in the N training cycles.

[0133] The opacity parameter of a Gaussian point determines its visibility during rendering. The higher the opacity, the less noticeable the Gaussian point; the lower the opacity, the more noticeable the Gaussian point.

[0134] During training, the system can record the visibility of each Gaussian point in each training epoch. Visibility refers to whether a Gaussian point contributes color information to the rendered image from the current viewpoint. If a Gaussian point is rendered into the image during a training epoch, it is considered visible; otherwise, it is considered invisible. After every N training epochs, the system can statistically analyze the visibility of each Gaussian point over those N epochs. For example, the system can record the number of times a Gaussian point is seen during those N epochs.

[0135] Based on the visibility statistics of Gaussian points over N training epochs, the system can dynamically adjust their transparency. Specifically: if a Gaussian point is frequently visible over N training epochs (i.e., seen a lot), the system may reduce its transparency to make it more noticeable during rendering; if a Gaussian point is rarely visible over N training epochs (i.e., seen a few times), the system may increase its transparency to make it more transparent or even invisible during rendering. The transparency reset operation can update the transparency parameters of Gaussian points based on the visibility statistics. This update can be based on preset rules or formulas, such as linearly adjusting transparency according to the visibility ratio.

[0136] As an example of this implementation, the system can maintain a visibility counter for each Gaussian point, recording the number of times it is seen over N training epochs. Opacity updates can be based on visibility proportions. For example, if a Gaussian point is seen 80% of the time over N training epochs, its opacity is set to a lower value; if it is seen only 20% of the time, its opacity is set to a higher value. Every N training epochs, the system can iterate through all Gaussian points and update the opacity parameter based on their visibility statistics.

[0137] In this implementation, by dynamically adjusting the transparency of Gaussian points, the system can better control the detail representation of different regions in the target scene. For frequently visible areas, reducing transparency enhances detail; for rarely visible areas, increasing transparency reduces unnecessary computation and rendering overhead, thereby improving training and rendering efficiency. Furthermore, during training, the distribution and visibility of Gaussian points in the scene may change. By dynamically adjusting transparency, the system can adaptively optimize the representation of the Gaussian point cloud, ensuring it always remains consistent with the needs of the current scene.

[0138] In the training process of a 3D Gaussian point rendering model, the computational flow can be divided into two main parts: forward computation and backward computation. Forward computation involves calculations from camera parameters and the reconstructed target scene to the output rendered image. Specifically, this includes frustum clipping of the Gaussian point cloud based on camera parameters, projecting the Gaussian points onto a 2D screen space, creating a visibility table, and rasterization, ultimately generating the rendered image. Backward computation involves calculations from the loss function to the parameter gradients. By comparing the differences between the rendered image and the reference image, the gradient of each Gaussian point parameter is calculated, and gradient descent is used to update parameters such as the position, color, and transparency of the Gaussian points, thereby optimizing the model's fitting effect. The combination of forward and backward computation allows the model to gradually improve rendering quality through iterative training, ultimately achieving efficient reconstruction from a 2D image to a 3D scene.

[0139] In this embodiment of the disclosure, the camera parameters of the viewpoint to be predicted can be input into the three-dimensional Gaussian point rendering model corresponding to the target scene, and the rendered image corresponding to the viewpoint to be predicted can be output through the three-dimensional Gaussian point rendering model.

[0140] In this context, the predicted viewpoint is the specific perspective from which the target scene is observed. The 3D Gaussian point rendering model needs to generate a rendered image viewed from this predicted viewpoint. The camera parameters for the predicted viewpoint can include information such as the camera's position, orientation, and field of view (view frustum). These parameters define the angle from which the target scene is observed and determine which Gaussian points will be included in the rendered image. After inputting the camera parameters of the predicted viewpoint into the 3D Gaussian point rendering model corresponding to the target scene, the model can output a rendered image corresponding to that viewpoint. This rendered image is a prediction of the target scene observed from the predicted viewpoint and includes information such as the target scene's color, lighting, and transparency.

[0141] In one possible implementation, the step of outputting the rendered image corresponding to the viewpoint to be predicted through the 3D Gaussian point rendering model includes: performing view frustum clipping on the Gaussian point cloud according to the camera parameters of the viewpoint to be predicted; projecting the Gaussian spheres in the Gaussian point cloud that were not removed by view frustum clipping onto the two-dimensional space of the screen; creating a visibility table corresponding to the viewpoint to be predicted based on the Gaussian projection result; and performing rasterization processing on the visible Gaussian points in the Gaussian point cloud according to the visibility table to obtain the rendered image corresponding to the viewpoint to be predicted.

[0142] In this implementation, a view frustum (i.e., the area visible to the camera) can be determined based on the camera parameters of the viewpoint to be predicted. Then, view frustum clipping can be performed on the Gaussian point cloud to remove Gaussian points located outside the view frustum. View frustum clipping reduces the number of Gaussian points that need to be processed, thereby improving rendering efficiency.

[0143] For Gaussian spheres not culled by the view frustum, they can be projected from 3D space onto the 2D space of the screen. Specifically, each Gaussian sphere (i.e., the representation of a Gaussian distribution in 3D space) can be transformed into the screen coordinate system through a series of mathematical transformations (such as MVP (Model-View-Projection) transformations). For example, the coordinate transformation of a Gaussian point is an MVP transformation, converting the world coordinates in 3D space to the NDC (Normalized Device Coordinates) in camera space. The formula for handling the covariance of Gaussian points in projection can be: Σ ′ =JWΣW T J T Σ, where Σ is the covariance of the Gaussian point in three-dimensional space. ′ Let J be the covariance matrix of a two-dimensional Gaussian sphere projected onto screen space, where W is the camera's view transformation and J is an affine transformation, an approximation of the camera projection transformation, obtained using a first-order Taylor expansion. After projection, each Gaussian sphere is represented on the screen as a two-dimensional Gaussian distribution, which can contain information such as position, color, and transparency.

[0144] Based on the Gaussian projection results, a visibility table can be created to record the visible Gaussian spheres in each screen region. This visibility table allows for quick location of the Gaussian spheres that need to be rendered in each screen region, thus reducing unnecessary calculations.

[0145] Based on a visibility table, visible Gaussian spheres in each screen region can be rasterized. Rasterization is the process of converting the color and transparency information of the Gaussian spheres into screen pixels. Specifically, for each screen region, all visible Gaussian spheres can be found using the visibility table. The colors and transparency of these Gaussian spheres can be overlaid to calculate the final color of the screen region. Rasterization converts the properties of the Gaussian spheres into screen pixels, generating the final rendered image.

[0146] After the above steps, a rendered image corresponding to the viewpoint to be predicted can be generated. This rendered image is the result of observing the target scene from the viewpoint to be predicted, and includes information such as the color, lighting, and transparency of the target scene.

[0147] In this implementation, the number of Gaussian points that need to be processed can be reduced by using view frustum clipping and a visibility table, thereby improving rendering efficiency. Furthermore, rasterization ensures the detail and realism of the rendered image, accurately reflecting the color and transparency information of the target scene.

[0148] In one possible implementation, the step of frustum clipping of the Gaussian point cloud based on the camera parameters of the viewpoint to be predicted includes: frustum clipping of the Gaussian point cloud in units of the Gaussian point cluster based on the camera parameters of the viewpoint to be predicted.

[0149] In this implementation, view frustum pruning can be performed on a cluster of Gaussian points, rather than on a single Gaussian point. Specifically, it can be determined whether each cluster of Gaussian points is completely outside the view frustum. If all Gaussian points in a cluster are outside the view frustum, the entire cluster will be removed.

[0150] As an example of this implementation, for each Gaussian point cluster, its bounding box can be computed. The bounding box is a simple geometric shape (such as a cuboid) that completely contains all Gaussian points within that cluster. Efficient geometric algorithms (such as the intersection test between the view frustum and the bounding box) can be used to determine whether each Gaussian point cluster lies within the view frustum. If the bounding box of a Gaussian point cluster lies entirely outside the view frustum, that cluster is culled and skipped from subsequent rendering calculations.

[0151] In this implementation, frustum pruning is performed on a cluster basis, reducing the number of Gaussian points that need to be evaluated. Compared to evaluating Gaussian points one by one, pruning on a cluster basis significantly reduces computation and improves pruning efficiency. Since Gaussian points within the same cluster are stored contiguously in GPU memory, pruning on a cluster basis optimizes GPU memory access patterns, reduces random access, and improves cache hit rate.

[0152] In one possible implementation, after the view frustum clipping of the Gaussian point cloud and before projecting the Gaussian points in the Gaussian point cloud that were not clipped into the two-dimensional space of the screen, the method further includes: copying the parameters of each cluster of Gaussian points in the Gaussian point cloud that were not clipped into the view frustum to a contiguous video memory space.

[0153] In this implementation, after view frustum culling, the parameters (such as position, color, and transparency) of the remaining Gaussian point clusters can be copied to a contiguous memory space. In subsequent calculations such as projection and rasterization, the system can efficiently access this contiguous memory space, thereby improving overall computational performance.

[0154] By storing the parameters of the Gaussian point cluster contiguously, fragmented access to GPU memory is reduced, improving memory utilization efficiency, especially in large-scale scenarios where it significantly reduces memory usage. Furthermore, the contiguous memory storage method ensures that subsequent read operations are sequential, rather than the out-of-order reads of traditional methods. This sequential memory access mode maximizes cache hit rate, reduces I / O overhead, and thus improves computational efficiency.

[0155] Furthermore, during the reverse computation phase, gradient update operations typically involve writing to GPU memory. By concatenating parameters in clusters, out-of-order writes can be avoided. Out-of-order writes lead to GPU memory access conflicts and performance degradation, while contiguous storage ensures efficient write operations.

[0156] In one possible implementation, creating a visibility table corresponding to the viewpoint to be predicted based on the Gaussian projection result includes: for each two-dimensional Gaussian sphere projected onto the screen in the Gaussian projection result, creating a bounding box corresponding to the two-dimensional Gaussian sphere according to a preset transparency threshold; determining the visibility of the two-dimensional Gaussian sphere to each screen area based on the bounding box corresponding to the two-dimensional Gaussian sphere; and creating a visibility table corresponding to the viewpoint to be predicted based on the visibility of the two-dimensional Gaussian sphere to each screen area.

[0157] In a 3D Gaussian point rendering model, after a Gaussian sphere is projected onto screen space, each Gaussian sphere can be represented as a two-dimensional Gaussian distribution, i.e., a two-dimensional Gaussian sphere, which contains information such as position, color, and transparency.

[0158] In this implementation, a preset transparency threshold (e.g., 1 / 255) can be used to determine the visibility of the Gaussian sphere. Gaussian spheres with transparency below this preset threshold can be considered to have no significant contribution to the rendering result and can be ignored. For each 2D Gaussian sphere, a bounding box can be calculated based on the preset transparency threshold. The bounding box can be a rectangular region used to represent the area that the 2D Gaussian sphere may cover on the screen.

[0159] As an example of this implementation, it can be based on the formula for the ellipse of the two-dimensional Gaussian distribution with equal probability. The boundary with an opacity of 1 / 255 is calculated to determine the visible range of the 2D Gaussian sphere on the screen. Here, λ1 and λ2 are eigenvalues ​​of the 2D Gaussian distribution, representing the extent of the Gaussian distribution's spread in the x and y directions, and c is a constant used to control the boundary of the ellipse. Based on this boundary, a rectangular bounding box can be created to represent the area that the 2D Gaussian sphere may cover on the screen. By incorporating the opacity properties of the Gaussian points into the bounding box calculation, its visibility range can be determined more accurately, allowing subsequent rasterization processing to only calculate the visible Gaussian sphere, reducing unnecessary rendering overhead.

[0160] In this implementation, the screen is divided into multiple regions (such as tiles), and each region needs to determine which 2D Gaussian spheres are visible to it. Based on the bounding box of a 2D Gaussian sphere, it can be determined whether that 2D Gaussian sphere covers a certain screen region. If it covers it, then the 2D Gaussian sphere is visible to that screen region.

[0161] A visibility table can be a data structure used to record the visible 2D Gaussian spheres in each screen region. For example, a list can be maintained for each screen region, storing all 2D Gaussian spheres visible to it. Based on the visibility determination, each 2D Gaussian sphere can be added to the visibility list of the screen regions it covers. The visibility table is used to quickly find the 2D Gaussian spheres that need to be rendered in each screen region during subsequent rasterization processing, thereby reducing unnecessary calculations.

[0162] In this implementation, a visibility table system can quickly identify the 2D Gaussian spheres that need to be processed in each screen area, avoiding traversal of all 2D Gaussian spheres and thus reducing computational load. The visibility table ensures that rasterization processing is performed only on visible 2D Gaussian spheres, improving rendering efficiency. Furthermore, in large-scale and complex scenes, the number of Gaussian spheres can be enormous. The visibility table allows for efficient management of the visibility of Gaussian spheres, ensuring the efficiency of the rendering process.

[0163] In this embodiment, the loss function of the 3D Gaussian point rendering model can be calculated based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted. The reference image can represent a real image corresponding to the viewpoint to be predicted (e.g., a real scene image in the training data) and is used to guide the optimization of the 3D Gaussian point rendering model. The loss function measures the difference between the rendered image and the reference image. By minimizing the value of the loss function, the 3D Gaussian point rendering model can progressively optimize the parameters of the Gaussian points, making the rendered image closer to the reference image. The smaller the value of the loss function, the closer the rendered image is to the reference image, and the better the model's prediction performance.

[0164] In one possible implementation, updating the parameters of the Gaussian points in the Gaussian point cloud based on the value of the loss function includes: calculating the gradient value of each Gaussian point in the Gaussian point cloud using at least one thread block based on the value of the loss function; for any thread, writing the gradient value of the Gaussian point calculated by the thread into the shared memory corresponding to the thread block to which the thread belongs; for any thread block, performing a reduction operation on the gradient values ​​of the Gaussian points calculated by the threads in the thread block in the shared memory corresponding to the thread block to obtain the gradient reduction result corresponding to the thread block; for any thread block, synchronizing the gradient reduction result corresponding to the thread block to global memory using one thread in the thread block; and updating the parameters of the Gaussian points in the Gaussian point cloud based on the gradient reduction result of the Gaussian points in the Gaussian point cloud stored in the global memory.

[0165] In this implementation, Gaussian points in the Gaussian point cloud can be assigned to multiple thread blocks, each responsible for calculating the gradient values ​​of a subset of Gaussian points. Each thread within a thread block can calculate the gradient values ​​for its assigned Gaussian points based on the loss function value. The gradient values ​​represent the direction and magnitude of parameter adjustments needed.

[0166] Each thread block can have a shared memory area to temporarily store the results of calculations performed by threads within the block. Each thread can write the gradient value of its calculated Gaussian point to the shared memory area corresponding to its thread block. Accessing shared memory is much faster than accessing global memory, thus this design can significantly improve computational efficiency.

[0167] In shared memory, reduction operations can be performed on the gradient values ​​calculated by all threads within a thread block. A reduction operation is the process of combining multiple values ​​into a single value, such as summation or finding the maximum value. Through the reduction operation, the gradient reduction result corresponding to the thread block can be obtained, which is the sum of the gradients of all Gaussian points within that thread block.

[0168] In this implementation, global memory is a shared memory region across all thread blocks in the GPU, used to store the final computation results. For each thread block, one of its threads can synchronize the gradient reduction results from shared memory to global memory. This synchronization ensures that the computation results of all thread blocks can be managed and used uniformly.

[0169] After storing the gradient reduction results of all Gaussian points in global memory, the parameters of the Gaussian points in the Gaussian point cloud (such as position, color, transparency, etc.) can be updated based on these gradient values. Specifically, gradient descent or other optimization algorithms can be used to adjust the parameters in the opposite direction of the gradient, thereby reducing the value of the loss function.

[0170] Through multiple iterations of the above process, the model will gradually optimize the parameters of the Gaussian points, eventually generating a high-quality 3D scene.

[0171] This implementation utilizes thread blocks and shared memory to efficiently compute the gradient values ​​of Gaussian points in parallel, significantly improving training speed. The use of shared memory reduces the number of global memory accesses, avoiding the bottleneck of GPU memory bandwidth and thus improving computational efficiency. This parallel computing and GPU memory optimization design enables this method to efficiently handle large-scale and complex 3D reconstruction tasks.

[0172] In one possible implementation, writing the gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs for any given thread includes: writing the non-zero gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs for any given thread.

[0173] In this implementation, each thread can compute the gradient value assigned to its Gaussian point. The gradient value is a floating-point number that represents the direction and magnitude by which the parameters of the Gaussian point need to be adjusted. Before writing the gradient value to shared memory, the thread can check if the computed gradient value is zero. If the gradient value is zero, it means that the parameters of the Gaussian point do not need to be adjusted in the current iteration, so there is no need to write it to shared memory. For non-zero gradient values, the thread can write them to the shared memory of the thread block.

[0174] In this implementation, by only writing non-zero gradient values, the number of shared memory write operations is reduced, thereby reducing the number of memory accesses.

[0175] The training method for the three-dimensional Gaussian point rendering model provided in this disclosure can be applied to the technical fields of 3D reconstruction, Gaussian point rendering, etc., and is not limited thereto.

[0176] The training method for the three-dimensional Gaussian point rendering model provided in this embodiment is illustrated below through a specific application scenario. Figure 2 This diagram illustrates a training method for a three-dimensional Gaussian point rendering model provided in an embodiment of the present disclosure.

[0177] like Figure 2 As shown, the training process of the 3D Gaussian point rendering model can be divided into two main parts: forward calculation and backward calculation.

[0178] The forward computation phase may include steps such as sample selection, frustum culling, parameter stitching, Gaussian projection, creation of a visibility table, and rasterization.

[0179] Sample selection: A sample can be selected from the training dataset of the target scene. The selected sample can include a reference image and camera parameters corresponding to the viewpoint to be predicted. The viewpoint to be predicted can represent the viewpoint when the reference image was acquired.

[0180] Forward computation for view frustum culling: Using camera parameters of the viewpoint to be predicted, view frustum culling is performed on the Gaussian point cloud of the target scene, in units of Gaussian point clusters. This step removes Gaussian points outside the view frustum range based on camera parameters, reducing the computational burden on subsequent steps.

[0181] Forward computation of parameter stitching: The parameters of each Gaussian point cluster in the Gaussian point cloud that has not been removed by view frustum pruning can be copied to a contiguous video memory space.

[0182] Forward calculation of Gaussian projection: Gaussian spheres in the Gaussian point cloud that have not been clipped by the view frustum can be projected into the two-dimensional space of the screen.

[0183] Create a visibility table: For each two-dimensional Gaussian sphere projected onto the screen in the Gaussian projection result, a bounding box corresponding to the two-dimensional Gaussian sphere can be created according to a preset transparency threshold; the visibility of the two-dimensional Gaussian sphere to each screen area can be determined according to the bounding box corresponding to the two-dimensional Gaussian sphere; and a visibility table corresponding to the viewpoint to be predicted can be created according to the visibility of the two-dimensional Gaussian sphere to each screen area.

[0184] Forward computation of rasterization: Based on the visibility table, the visible Gaussian points in the Gaussian point cloud are rasterized to obtain the rendered image corresponding to the viewpoint to be predicted.

[0185] After obtaining the rendered image corresponding to the viewpoint to be predicted, the value of the loss function (loss) corresponding to the three-dimensional Gaussian point rendering model can be calculated based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted.

[0186] Corresponding to the forward computation, the backward computation stage can include backward computation of rasterization, backward computation of Gaussian projection, and backward computation of parameter stitching. The Gaussian point parameters in the Gaussian point cloud can be updated based on the calculated gradient values.

[0187] After completing a training iteration, you can check if the required number of training iterations has been met. If not, training can continue. If the current training epoch is a multiple of 10 (i.e., "epoch % 10 != 0?" is false), you can perform steps such as generating new Gaussian points, deleting invalid Gaussian point clusters, and reorganizing Gaussian point clusters to optimize the training process.

[0188] The training method for the 3D Gaussian point rendering model provided for this application scenario can convert video footage into a high-definition, accurate 3D scene within 10-20 minutes. This increases training speed by approximately 80% while maintaining the same training accuracy as traditional methods.

[0189] This disclosure also provides a three-dimensional reconstruction method, including: acquiring camera parameters of a target viewpoint; inputting the camera parameters of the target viewpoint into a pre-trained three-dimensional Gaussian point rendering model; and outputting a rendered image corresponding to the target viewpoint through the three-dimensional Gaussian point rendering model, wherein the three-dimensional Gaussian point rendering model is trained using the training method for the three-dimensional Gaussian point rendering model.

[0190] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0191] In addition, this disclosure also provides a training device for a three-dimensional Gaussian point drawing model, a three-dimensional reconstruction device, a non-volatile computer-readable storage medium, and a computer program product. All of the above can be used to implement any of the training methods or three-dimensional reconstruction methods for a three-dimensional Gaussian point drawing model provided in this disclosure. The corresponding technical solutions and technical effects can be found in the relevant descriptions in the method section, and will not be repeated here.

[0192] Figure 3 A block diagram of a training apparatus for a three-dimensional Gaussian point rendering model provided in an embodiment of this disclosure is shown. Figure 3 As shown, the training device for the 3D Gaussian point rendering model includes:

[0193] The partitioning module 31 is used to partition the Gaussian point cloud into multiple Gaussian point clusters according to the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space.

[0194] The first prediction module 32 is used to input the camera parameters of the viewpoint to be predicted into the three-dimensional Gaussian point drawing model corresponding to the target scene, and output the rendered image corresponding to the viewpoint to be predicted through the three-dimensional Gaussian point drawing model.

[0195] The calculation module 33 is used to calculate the value of the loss function corresponding to the three-dimensional Gaussian point drawing model based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted.

[0196] The update module 34 is used to update the parameters of Gaussian points in the Gaussian point cloud according to the value of the loss function, wherein the parameters of Gaussian points in the same Gaussian point cluster are stored in contiguous memory space.

[0197] In one possible implementation, the partitioning module 31 is used for:

[0198] Every N training cycles, the Gaussian point cloud is divided into multiple Gaussian point clusters based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, where N is an integer greater than or equal to 1.

[0199] In one possible implementation, the device further includes:

[0200] The statistics module is used to calculate the sum of the gradients of the parameters of the Gaussian points in the Gaussian point cluster every N training cycles.

[0201] A new module is added to the Gaussian point cluster in response to the sum of gradients having a magnitude greater than or equal to a preset threshold.

[0202] In one possible implementation, the device further includes:

[0203] The deletion module is used to delete Gaussian point clusters that have been removed by view frustum pruning for N consecutive training epochs.

[0204] In one possible implementation, the device further includes:

[0205] The reset module is used to reset the transparency of any Gaussian point in the Gaussian point cloud every N training cycles, based on the visibility of the Gaussian point in those N training cycles.

[0206] In one possible implementation, the first prediction module 32 is used to:

[0207] Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frustum-clipped.

[0208] Project the Gaussian spheres in the Gaussian point cloud that have not been clipped by the view frustum into the two-dimensional space of the screen;

[0209] Based on the Gaussian projection results, a visibility table corresponding to the viewpoint to be predicted is created;

[0210] Based on the visibility table, the visible Gaussian points in the Gaussian point cloud are rasterized to obtain the rendered image corresponding to the viewpoint to be predicted.

[0211] In one possible implementation, the first prediction module 32 is used to:

[0212] Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frusto-clipping is performed on the Gaussian point cluster as a unit.

[0213] In one possible implementation, the device further includes:

[0214] The parameter stitching module is used to copy the parameters of each Gaussian point cluster in the Gaussian point cloud that has not been removed by view frustum clipping to a contiguous video memory space.

[0215] In one possible implementation, the first prediction module 32 is used to:

[0216] For each two-dimensional Gaussian sphere projected onto the screen in the Gaussian projection result, a bounding box corresponding to the two-dimensional Gaussian sphere is created according to a preset transparency threshold.

[0217] The visibility of the two-dimensional Gaussian sphere to each screen region is determined based on the bounding box corresponding to the two-dimensional Gaussian sphere.

[0218] Based on the visibility of the two-dimensional Gaussian sphere to each screen region, a visibility table corresponding to the viewpoint to be predicted is created.

[0219] In one possible implementation, the update module 34 is used to:

[0220] The gradient value of each Gaussian point in the Gaussian point cloud is calculated using at least one thread block based on the value of the loss function;

[0221] For any thread, the gradient value of the Gaussian point calculated by the thread is written to the shared memory corresponding to the thread block to which the thread belongs;

[0222] For any thread block, in the shared memory corresponding to the thread block, the gradient value of the Gaussian point calculated by the thread in the thread block is reduced to obtain the gradient reduction result corresponding to the thread block.

[0223] For any thread block, the gradient reduction result corresponding to the thread block is synchronized to the global memory through one of the threads in the thread block;

[0224] Based on the gradient reduction results of the Gaussian points in the Gaussian point cloud stored in the global memory, the parameters of the Gaussian points in the Gaussian point cloud are updated.

[0225] In one possible implementation, the update module 34 is used to:

[0226] For any thread, write the non-zero gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs.

[0227] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, comprising:

[0228] The acquisition module is used to acquire camera parameters from the target's viewpoint;

[0229] The second prediction module is used to input the camera parameters of the target viewpoint into a pre-trained three-dimensional Gaussian point rendering model, and output the rendered image corresponding to the target viewpoint through the three-dimensional Gaussian point rendering model. The three-dimensional Gaussian point rendering model is trained by the three-dimensional reconstruction device.

[0230] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation and technical effects can be referred to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0231] According to another aspect of this disclosure, a training apparatus for a three-dimensional Gaussian point drawing model is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described training method for a three-dimensional Gaussian point drawing model.

[0232] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional reconstruction method described above.

[0233] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0234] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0235] Figure 4 This is a block diagram illustrating a training apparatus 1900 for a three-dimensional Gaussian point rendering model according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 4 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0236] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0237] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0238] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0239] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0240] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0241] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0242] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0243] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0244] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0245] Computer program products can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0246] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0247] If the technical solution of this disclosure involves personal information, the product applying the technical solution of this disclosure has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this disclosure involves sensitive personal information, the product applying the technical solution of this disclosure has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to indicate that the user has entered the scope of personal information collection and that personal information will be collected. If the user voluntarily enters the collection scope, it is deemed to have consented to the collection of their personal information; or on the personal information processing device, with clear signs / information informing the user of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0248] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a three-dimensional Gaussian point rendering model, characterized in that, include: Based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, the Gaussian point cloud is divided into multiple Gaussian point clusters; The camera parameters of the viewpoint to be predicted are input into the three-dimensional Gaussian point rendering model corresponding to the target scene, and the rendered image corresponding to the viewpoint to be predicted is output through the three-dimensional Gaussian point rendering model. Based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted, calculate the value of the loss function corresponding to the 3D Gaussian point rendering model; Based on the value of the loss function, the parameters of the Gaussian points in the Gaussian point cloud are updated, wherein the parameters of the Gaussian points in the same Gaussian point cluster are stored in contiguous memory space.

2. The method according to claim 1, characterized in that, The step of dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space includes: Every N training cycles, the Gaussian point cloud is divided into multiple Gaussian point clusters based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space, where N is an integer greater than or equal to 1.

3. The method according to claim 2, characterized in that, Before dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of Gaussian points in the three-dimensional space corresponding to the target scene, the method further includes: Every N training cycles, the sum of the gradients of the parameters of the Gaussian points in the Gaussian point cluster is calculated. In response to the magnitude of the sum of the gradients being greater than or equal to a preset threshold, a new Gaussian point is added to the Gaussian point cluster based on the sum of the gradients.

4. The method according to claim 2, characterized in that, Before dividing the Gaussian point cloud into multiple Gaussian point clusters based on the positions of Gaussian points in the three-dimensional space corresponding to the target scene, the method further includes: Delete the cluster of Gaussian points that have been removed by view frustum pruning for N consecutive training cycles.

5. The method according to claim 2, characterized in that, The method further includes: For any Gaussian point in the Gaussian point cloud, every N training epochs, the transparency of the Gaussian point is reset based on its visibility in those N training epochs.

6. The method according to claim 1, characterized in that, The step of outputting the rendered image corresponding to the viewpoint to be predicted through the three-dimensional Gaussian point rendering model includes: Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frustum-clipped. Project the Gaussian spheres in the Gaussian point cloud that have not been clipped by the view frustum into the two-dimensional space of the screen; Based on the Gaussian projection results, a visibility table corresponding to the viewpoint to be predicted is created; Based on the visibility table, the visible Gaussian points in the Gaussian point cloud are rasterized to obtain the rendered image corresponding to the viewpoint to be predicted.

7. The method according to claim 6, characterized in that, The step of frustum clipping of the Gaussian point cloud based on the camera parameters of the viewpoint to be predicted includes: Based on the camera parameters of the viewpoint to be predicted, the Gaussian point cloud is frusto-clipping is performed on the Gaussian point cluster as a unit.

8. The method according to claim 6 or 7, characterized in that, After performing view frustum clipping on the Gaussian point cloud and before projecting the Gaussian points in the Gaussian point cloud that were not removed by view frustum clipping into the two-dimensional space of the screen, the method further includes: The parameters of each Gaussian point cluster in the Gaussian point cloud that has not been removed by view frustum pruning are copied to a contiguous video memory space.

9. The method according to claim 6, characterized in that, The step of creating a visibility table corresponding to the viewpoint to be predicted based on the Gaussian projection results includes: For each two-dimensional Gaussian sphere projected onto the screen in the Gaussian projection result, a bounding box corresponding to the two-dimensional Gaussian sphere is created according to a preset transparency threshold. The visibility of the two-dimensional Gaussian sphere to each screen region is determined based on the bounding box corresponding to the two-dimensional Gaussian sphere. Based on the visibility of the two-dimensional Gaussian sphere to each screen region, a visibility table corresponding to the viewpoint to be predicted is created.

10. The method according to claim 1, characterized in that, The step of updating the parameters of the Gaussian points in the Gaussian point cloud based on the value of the loss function includes: The gradient value of each Gaussian point in the Gaussian point cloud is calculated using at least one thread block based on the value of the loss function; For any thread, the gradient value of the Gaussian point calculated by the thread is written to the shared memory corresponding to the thread block to which the thread belongs; For any thread block, in the shared memory corresponding to the thread block, the gradient value of the Gaussian point calculated by the thread in the thread block is reduced to obtain the gradient reduction result corresponding to the thread block. For any thread block, the gradient reduction result corresponding to the thread block is synchronized to the global memory through one of the threads in the thread block; Based on the gradient reduction results of the Gaussian points in the Gaussian point cloud stored in the global memory, the parameters of the Gaussian points in the Gaussian point cloud are updated.

11. The method according to claim 10, characterized in that, The step of writing the gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs, for any given thread, includes: For any thread, write the non-zero gradient value of the Gaussian point calculated by the thread to the shared memory corresponding to the thread block to which the thread belongs.

12. A three-dimensional reconstruction method, characterized in that, include: Obtain camera parameters from the target's perspective; The camera parameters of the target viewpoint are input into a pre-trained 3D Gaussian point rendering model, and the rendered image corresponding to the target viewpoint is output through the 3D Gaussian point rendering model. The 3D Gaussian point rendering model is trained using the method described in any one of claims 1 to 11.

13. A training device for a three-dimensional Gaussian point rendering model, characterized in that, include: The partitioning module is used to divide the Gaussian point cloud into multiple Gaussian point clusters based on the position of the Gaussian points in the Gaussian point cloud corresponding to the target scene in three-dimensional space. The first prediction module is used to input the camera parameters of the viewpoint to be predicted into the three-dimensional Gaussian point rendering model corresponding to the target scene, and output the rendered image corresponding to the viewpoint to be predicted through the three-dimensional Gaussian point rendering model. The calculation module is used to calculate the value of the loss function corresponding to the three-dimensional Gaussian point rendering model based on the rendered image corresponding to the viewpoint to be predicted and the reference image corresponding to the viewpoint to be predicted. An update module is used to update the parameters of Gaussian points in the Gaussian point cloud according to the value of the loss function, wherein the parameters of Gaussian points in the same Gaussian point cluster are stored in contiguous memory space.

14. A three-dimensional reconstruction device, characterized in that, include: The acquisition module is used to acquire camera parameters from the target's viewpoint; The second prediction module is used to input the camera parameters of the target viewpoint into a pre-trained three-dimensional Gaussian point rendering model, and output the rendered image corresponding to the target viewpoint through the three-dimensional Gaussian point rendering model, wherein the three-dimensional Gaussian point rendering model is trained using the device described in claim 13.

15. A training device for a three-dimensional Gaussian point rendering model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

16. A three-dimensional reconstruction apparatus, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 12.

17. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

18. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images