Online volumetric video reconstruction method, device, equipment, and medium

Abstract

The present application relates to the field of video technology. The present application proposes an online volumetric video reconstruction method, device, equipment, and medium. The present application uses an initialized Gaussian primitive model to reconstruct the volumetric video of the original image of the target scene at the current moment. The initialized Gaussian point attribute values ​​are the parameters required for the Gaussian primitive model to obtain the volumetric video of the target scene at the previous moment. The parameters of the initialized Gaussian primitive model are the parameters updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment. This method can directly use the reconstruction key information of the previous moment to reconstruct the video at the current moment, thereby improving the reconstruction quality of the volumetric video of the target scene at the current moment. The incremental training method eliminates the need to retrain the Gaussian primitive model from scratch to update the model parameters, thereby reducing the number of iterations required for training the model, and reducing computational redundancy and reconstruction time.

Description

Technical Field

[0001] The present application relates to the field of video technology, and in particular to an online volumetric video reconstruction method, apparatus, device, and medium. Background Art

[0002] With the continuous advancement of communication networks, 3D display technology, and 3D content construction technology, volumetric video has gradually become a research hotspot in the next generation of video technology. Volumetric video allows users to view 3D scenes at different times from any viewpoint.

[0003] Volumetric video reconstruction uses computer vision and image processing techniques to reconstruct the three-dimensional structure of a scene by inputting images captured from multiple perspectives. Traditional methods include point cloud representation and mesh reconstruction. However, these traditional methods struggle to meet the requirements for reconstruction detail, data efficiency, and real-time performance. Furthermore, these methods are discrete, non-differentiable, and difficult to optimize.

[0004] In recent years, with the development of deep learning, researchers have delved into how to use neural networks to represent 3D geometry and light fields as continuous functions. Neural radiance fields are a representative example of this new approach to volumetric video synthesis. However, while neural radiance field-based methods can effectively improve the quality of volumetric video reconstruction, the dense sampling required for volume rendering in these methods results in slow training and rendering speeds, making real-time performance difficult.

[0005] Currently, 3D Gaussian Splatting achieves real-time, high-quality synthesis of new viewpoints for static scenes by combining explicit point cloud representation with implicit spherical harmonics representation, providing a real-time rendering and high-quality representation technology for volumetric video reconstruction. Existing volumetric video reconstruction methods based on 3D Gaussian Splatting mostly model spatiotemporal information using deformation field-based and 4D Gaussian basis element-based approaches. However, these methods are all offline reconstruction methods, meaning that the training process relies on the complete input of the entire video sequence, and decoding requires the transmission of all data before decoding can proceed. The speed and storage overhead of reconstructing the three-dimensional structure of a scene are difficult to meet the requirements of real-world applications.

[0006] In summary, the existing volumetric video reconstruction methods have the problems of low reconstruction quality, low reconstruction real-time performance, slow reconstruction speed and high storage overhead. These problems make the existing volumetric video reconstruction methods difficult to meet the needs of real applications. Summary of the Invention

[0007] The present application provides an online volumetric video reconstruction method, apparatus, device, and medium to address the problems of existing volumetric video reconstruction methods, such as low reconstruction result quality, low reconstruction real-time performance, slow reconstruction speed, and high storage overhead.

[0008] A first aspect of the present application provides an online volumetric video reconstruction method, the method of the present application comprising:

[0009] Get the original image of the target scene at the current moment;

[0010] The original image of the target scene at the current moment is reconstructed and rendered using the initialized Gaussian primitive model to obtain a rendered image of the target scene at the current moment, and a volumetric video of the target scene at the current moment is calculated based on the rendered image of the target scene at the current moment, wherein the parameters of the initialized Gaussian primitive model are the parameters updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment;

[0011] After obtaining the volumetric video of the target scene at the current moment, the initialized Gaussian primitive model is trained online incrementally to update the parameters of the Gaussian primitive model.

[0012] In some embodiments of the present application, the parameters of the initialized Gaussian primitive model include: initialized Gaussian point attribute values ​​and initialized Gaussian point attribute residual values, wherein the initialized Gaussian point attribute values ​​are parameters required for the Gaussian primitive model to obtain the volumetric video of the target scene at the previous moment, and the initialized Gaussian point attribute residual values ​​are Gaussian point attribute residual values ​​updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment, and the steps of performing online incremental training on the initialized Gaussian primitive model to update the parameters of the Gaussian primitive model include:

[0013] During the online incremental training process, the Gaussian point attribute residual value is updated according to the preset first total loss.

[0014] In some embodiments of the present application, the first total loss includes: error loss and structural similarity loss between the original image of the target scene at the current moment and the rendered image of the target scene, and regularization loss determined based on the initialized Gaussian point attribute residual value.

[0015] In some embodiments of the present application, the Gaussian primitive model includes Gaussian points, and the parameters of the initialized Gaussian primitive model further include: an initialized Gaussian point mask, and the initialized Gaussian primitive model is subjected to online incremental training to update the parameters of the Gaussian primitive model. The steps include:

[0016] During the online incremental training process, the Gaussian point mask and the Gaussian point attribute residual value are updated according to the preset second total loss, and based on the updated Gaussian point mask, the Gaussian point attribute value and the updated Gaussian point attribute residual value are updated, so that the transmitted updated Gaussian point attribute residual value is used as the Gaussian basis element model to obtain the Gaussian point attribute residual value required for the volumetric video of the target scene at the next moment.

[0017] In some embodiments of the present application, the second total loss includes: error loss and structural similarity loss between the original image of the target scene at the current moment and the rendered image of the target scene, regularization loss determined based on the initialized Gaussian point attribute residual value, and sparse loss determined based on the Gaussian point mask.

[0018] In some embodiments of the present application, the steps of updating the Gaussian point attribute value and transmitting the updated Gaussian point attribute residual value based on the updated Gaussian point mask include:

[0019] Binarize the updated Gaussian point mask;

[0020] Based on the binarized Gaussian point mask, the Gaussian point attribute value is updated and the updated Gaussian point attribute residual value is transmitted.

[0021] In some embodiments of the present application, after the Gaussian primitive model obtains the volumetric video of the target scene in a preset time period, a Gaussian point with a preset Gaussian point attribute value is added to the Gaussian primitive model, and the preset Gaussian point attribute value and the initialized Gaussian point attribute value are used together as new initialized Gaussian point attribute values, and the parameters of the initialized Gaussian primitive model include: the new initialized Gaussian point attribute value and the initialized Gaussian point attribute residual value.

[0022] According to a second aspect of the present application, the present application provides an online volumetric video generation device based on the online volumetric video reconstruction method of any of the above embodiments, the device of the present application comprising: an initialized Gaussian primitive model for performing volumetric video reconstruction and rendering on an original image of a target scene acquired at the current moment to obtain a rendered image of the target scene at the current moment, and calculating a volumetric video of the target scene at the current moment based on the rendered image of the target scene at the current moment, wherein the parameters of the initialized Gaussian primitive model are parameters updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment;

[0023] After obtaining the volumetric video of the target scene at the current moment, the initialized Gaussian primitive model is trained online incrementally to update the parameters of the Gaussian primitive model.

[0024] The third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in any one of the first aspects of the above embodiments are implemented.

[0025] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in any one of the first aspects of the above embodiments is implemented.

[0026] This application has the following beneficial effects:

[0027] The online volumetric video reconstruction method proposed in the above embodiment of the present application uses an initialized Gaussian element model to reconstruct the volumetric video of the original image of the target scene at the current moment, wherein the initialized Gaussian point attribute value is the initialization parameter required for the Gaussian element model to obtain the volumetric video of the target scene at the previous moment, and the parameters of the initialized Gaussian element model are the parameters updated by online incremental training after the Gaussian element model obtains the volumetric video of the target scene at the previous moment. This method can directly use the reconstruction key information of the previous moment to reconstruct the video at the current moment (the correlation between the videos of two adjacent moments is high), thereby improving the reconstruction quality of the volumetric video of the target scene at the current moment; and the Gaussian element model performs online incremental training after completing the real-time reconstruction task. The incremental training method is to further train the Gaussian element model based on the parameters updated by online learning after the Gaussian element model obtains the volumetric video of the target scene at the current moment, so that there is no need to retrain the Gaussian element model from scratch to update the model parameters, reducing the number of iterations required for training the model, thereby reducing computational redundancy and reconstruction time. Furthermore, the Gaussian basis model of this application supports volumetric video reconstruction by training, transmitting, and decoding images acquired at each moment (also known as frame-by-frame online training, transmission, and decoding). This reduces computational effort and reconstruction time, thereby increasing volumetric video reconstruction speed and reducing storage overhead. This online volumetric video reconstruction method of this application is more feasible in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0029] Figure 1 This is a flow chart of an embodiment of an online volumetric video reconstruction method provided by the present application;

[0030] Figure 2 This is a flowchart of another embodiment of the online volumetric video reconstruction method provided by this application;

[0031] Figure 3 This is a flow chart of an embodiment of a method for updating Gaussian point attribute values ​​provided by the present application;

[0032] Figure 4 This is a schematic diagram of the framework of an embodiment of an electronic device provided by the present application;

[0033] Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0035] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0036] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0037] The inventors have discovered that, compared to offline volumetric video reconstruction methods (which require the entire time series of video input for video reconstruction and model training, and require all data to be transmitted to the decoder before video decoding can be performed), online volumetric video reconstruction methods can support training, transmission, and decoding of images acquired at each moment (also known as frame-by-frame online training, transmission, and decoding), making them more suitable for applications such as immersive conferencing, 3D live streaming, and telemedicine. However, existing online volumetric video reconstruction methods based on 3D Gaussian sputtering can achieve high-quality reconstruction and real-time rendering of volumetric videos, but their reconstruction speed and storage overhead still cannot meet practical requirements. Analysis reveals that these limitations stem from the fact that existing online volumetric video reconstruction methods based on 3D Gaussian sputtering do not fully utilize key information in the reconstruction process (i.e., the model used at the current moment is newly trained, and the model parameters are unrelated to the parameters of the video reconstructed at the previous moment). This results in the need to train the model from scratch at each moment, increasing the number of training times, resulting in increased computational redundancy and increased volumetric video reconstruction time, thereby slowing down volumetric video reconstruction.

[0038] Therefore, in order to solve the problems of low reconstruction result quality, low reconstruction real-time performance, slow reconstruction speed and large storage overhead in the above-mentioned existing online volumetric video reconstruction methods, the present application proposes a new online volumetric video reconstruction solution based on 3D Gaussian sputtering. In the solution of the present application, the volumetric video of the target scene at the previous moment is obtained by using the Gaussian primitive model, and the parameters updated by online learning are used to perform real-time video reconstruction of the target scene. After the Gaussian primitive model completes the real-time reconstruction task, online incremental training is performed. The incremental training method is to further train the Gaussian primitive model based on the parameters updated by online learning after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment. The solution of the present application can meet the high-quality reconstruction of real-time video, and the reconstruction speed is fast and the storage overhead is small, so that the online volumetric video reconstruction solution of the present application can meet the needs of real-time applications.

[0039] The present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] (1) Motion Residual Transfer Mechanism (MoRI)

[0041] The motion residual transfer mechanism associates the parameters required to construct the Gaussian primitive model of the target scene volumetric video at the current moment with the parameters of the Gaussian primitive model at the previous moment, and can directly use the reconstruction information of the previous moment to reconstruct the volumetric video at the current moment; and the optimized parameters of the Gaussian primitive model at the current moment are based on the parameters of the Gaussian primitive model at the previous moment, which effectively reduces the number of required iterations, thereby reducing computational redundancy and reconstruction time.

[0042] According to one embodiment of the present application, the present application proposes an online volumetric video reconstruction method, such as Figure 1 As shown, the method includes: S1, obtaining an original image of the target scene at the current moment; S2, reconstructing and rendering the original image of the target scene at the current moment using an initialized Gaussian primitive model to obtain a rendered image of the target scene at the current moment, and calculating a volumetric video of the target scene at the current moment based on the rendered image of the target scene at the current moment, wherein the parameters of the initialized Gaussian primitive model are parameters updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment; S3, after obtaining the volumetric video of the target scene at the current moment, the initialized Gaussian primitive model performs online incremental training to update the parameters of the Gaussian primitive model.

[0043] As can be seen from the above embodiments, the online volumetric video reconstruction method proposed in the above embodiments of the present application utilizes an initialized Gaussian primitive model to reconstruct a volumetric video of the original image of the target scene at the current moment. The parameters of the initialized Gaussian primitive model are the parameters updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment. This method can directly utilize the key reconstruction information of the previous moment to reconstruct the video at the current moment (videos at two adjacent moments have high correlation), thereby improving the reconstruction quality of the volumetric video of the target scene at the current moment. Furthermore, after the Gaussian primitive model completes the real-time reconstruction task, it performs online incremental training. The incremental training method further trains the Gaussian primitive model based on the parameters updated by online learning after the Gaussian primitive model obtains the volumetric video of the target scene at the current moment. This eliminates the need to retrain the Gaussian primitive model from scratch to update the model parameters, reduces the number of iterations required for training the model, and thus reduces computational redundancy and reconstruction time. Furthermore, the Gaussian primitive model of the present application supports a method of training, transmitting, and decoding images acquired at each moment (also known as a frame-by-frame online training, transmission, and decoding method) to reconstruct the volumetric video, reducing computational effort and reconstruction time, thereby improving the reconstruction speed of the volumetric video and reducing storage overhead. The online volumetric video reconstruction method of the present application is more feasible in practical applications.

[0044] Among them, according to one embodiment of the present application, the parameters of the initialized Gaussian element model include: initialized Gaussian point attribute values ​​and initialized Gaussian point attribute residual values, wherein the initialized Gaussian point attribute values ​​are the parameters required for the Gaussian element model to obtain the volumetric video of the target scene at the previous moment, and the initialized Gaussian point attribute residual values ​​are the Gaussian point attribute residual values ​​updated by online incremental training after the Gaussian element model obtains the volumetric video of the target scene at the previous moment, and the initialized Gaussian element model performs online incremental training to update the parameters of the Gaussian element model, and the step of updating the parameters of the Gaussian element model includes: updating the Gaussian point attribute residual values ​​according to a preset first total loss during the online incremental training process.

[0045] like Figure 2 As shown in Figure 2, it shows the whole process of the online volumetric video reconstruction method, where Figure 2 The whole process in (a) is called motion residual transfer, starting from Figure 2 From the motion residual transfer part (a), it can be seen that after the Gaussian meta-model completes the volumetric video reconstruction task of the target scene at time t, the Gaussian point attribute residual value updated by online incremental training after the Gaussian meta-model completes the volumetric video reconstruction task of the target scene at time t-1 is used as the Gaussian point attribute residual value initialized by the Gaussian meta-model at time t, and the Gaussian point attribute residual value initialized at time t is optimized (i.e., updated) to obtain the Gaussian point attribute residual value after the Gaussian meta-model is updated at time t.

[0046] From the above description, it can be seen that the above embodiment of the present application uses the Gaussian residual mapping method to obtain the Gaussian point attribute residual value between the previous moment and the current moment. This method can directly use the reconstruction key information of the previous moment to reconstruct the video of the current moment (the videos of two adjacent moments have high correlation), thereby improving the quality of the reconstructed video; at the same time, after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment, the parameters updated by online learning are further trained, so that there is no need to retrain the Gaussian primitive model from scratch to update the model parameters, reducing the number of iterations required for training the model, thereby reducing computational redundancy and reconstruction time.

[0047] Among them, according to one embodiment of the present application, the original image and the rendered image of the target scene include multiple pixel values, and the original image and the rendered image of the target scene are respectively divided into multiple local image blocks, and the positions of the local image blocks of the original image and the rendered image correspond one to one. The first total loss includes: the error loss and structural similarity loss between the original image of the target scene and the rendered image of the target scene at the current moment, and the regularization loss determined according to the initialized Gaussian point attribute residual value.

[0048] According to one embodiment of the present application, the first total loss is:

[0049]

[0050] in is the first total loss, Preset structural hyperparameters, is the error loss, is the structural similarity loss, is the regularization loss, and It is set to 0.2. This value is only one of the embodiments of the present application. The value of this hyperparameter can be dynamically set according to specific experimental results and is not specifically limited here.

[0051] According to one embodiment of the present application, the error loss is:

[0052]

[0053] Where N is the total number of pixel values ​​in the original image and rendered image of the target scene at time t, represents the i-th pixel value in the rendered image of the target scene at time t, Represents the i-th pixel value in the original image of the target scene at time t.

[0054] According to one embodiment of the present application, the structural similarity loss is:

[0055]

[0056] Where M is the total number of local image blocks in the original image and rendered image of the target scene at time t, is the structural similarity function, represents the jth local image block in the rendered image of the target scene at time t, Represents the jth local image patch in the original image of the target scene at time t.

[0057] As can be seen from the above description, the above embodiments of the present application utilize error loss training models to ensure that the rendered images output by the model are as close as possible to the original images at the pixel level. Structural similarity loss training models ensure that the overall structure and details of the rendered images output by the model are as consistent as possible with the original images, thereby making the rendering results more visually realistic and natural. This dual constraint not only improves the rendering accuracy of the model but also enhances its generalization ability in different scenarios, ultimately generating high-quality rendered images that meet the needs of practical applications.

[0058]

[0059] in, represents the average value of the pixel values ​​in the jth local image block of the rendered image of the target scene at time t, represents the average value of the pixel values ​​in the jth local image block of the original image of the target scene at time t, represents the standard deviation of the pixel values ​​in the jth local image block of the rendered image of the target scene at time t, represents the standard deviation of the pixel values ​​in the jth local image block of the original image of the target scene at time t, represents the covariance of the pixel values ​​in the jth local image block of the original image and the rendered image of the target scene at time t, To preset the first constant, is the preset second constant.

[0060] From the above description, we can see that when training the model, the constant and Used to stabilize the division operation, it can effectively avoid the numerical instability problem caused when the numerator is close to zero. and The addition of prevents the denominator in the formula from being too small, thus preventing infinite or unreasonable numerical results that may occur during the division process, and improving the robustness and accuracy of the SSIM (Structural Similarity Index) calculation. This processing method not only enhances the stability of the algorithm, but also makes comparisons between different image blocks more reliable and consistent, resulting in better quality volumetric videos.

[0061] The inventors discovered through research that motion is continuous. Inspired by this, using the Gaussian point attribute residual value at the previous moment as the initialization value at the current moment can effectively reduce the initialization error, thereby reducing the number of required calculation iterations and further reducing the required reconstruction time.

[0062] According to one embodiment of the present application, the Gaussian point attribute value includes: Gaussian point coordinate value, Gaussian point scale value, Gaussian point rotation value quaternion, Gaussian point opacity, and the Gaussian point attribute residual value includes: Gaussian point coordinate residual value, Gaussian point scale residual value, Gaussian point rotation value quaternion residual value, Gaussian point opacity residual value. According to one embodiment of the present application, the initialized Gaussian point attribute residual value is:

[0063]

[0064] in, The Gaussian point attribute residual value required to obtain the volume video of the target scene at time t for the Gaussian base model is initialized. The residual value of the initialized Gaussian point coordinates required for the Gaussian element model to obtain the volume video of the target scene at time t, The residual value of the initial Gaussian point scale required to obtain the volumetric video of the target scene at time t for the Gaussian element model, The residual value of the initial Gaussian point rotation value quaternion required for the Gaussian base element model to obtain the volume video of the target scene at time t, The residual value of the initial Gaussian point opacity required for the Gaussian primitive model to obtain the volumetric video of the target scene at time t.

[0065] It can be seen from the above embodiments that the above embodiments of the present application store and transmit Gaussian point attribute residual values, so that the Gaussian basis element model supports the training, transmission and decoding of images acquired at each moment (also known as frame-by-frame online training, transmission and decoding method) to reconstruct volumetric video, thereby reducing the amount of calculation and reconstruction time, thereby improving the reconstruction speed of volumetric video and reducing storage overhead.

[0066] The parameters of the Gaussian primitive model include: coordinate parameters, scale parameters, rotation value quaternion parameters, and opacity parameters. According to one embodiment of the present application, the coordinate parameters required for initialization of the volumetric video of the target scene at time t by the Gaussian primitive model are:

[0067]

[0068]

[0069] in, Represents the coordinate parameters required for the initialization of the Gaussian element model to obtain the volumetric video of the target scene at time t, It represents the coordinate parameters required by the Gaussian element model to obtain the volume video of the target scene at time t-1, which is the coordinate value of the initialized Gaussian point; It represents the residual value of the Gaussian point coordinates updated by online incremental training after obtaining the volumetric video of the target scene at time t-1, that is, the residual value of the initialized Gaussian point coordinates.

[0070] According to one embodiment of the present application, the initialization scale parameter required by the Gaussian primitive model to obtain the volumetric video of the target scene at time t is:

[0071]

[0072]

[0073] in, represents the initial scale parameter required by the Gaussian primitive model to obtain the volumetric video of the target scene at time t, It represents the scale parameter required by the Gaussian primitive model to obtain the volumetric video of the target scene at time t-1, which is the initial Gaussian point scale value; It represents the residual value of the Gaussian point scale updated by online incremental training after obtaining the volumetric video of the target scene at time t-1, that is, the residual value of the initialized Gaussian point scale.

[0074] According to one embodiment of the present application, the initialization rotation value quaternion parameter required by the Gaussian basis element model to obtain the volumetric video of the target scene at time t is:

[0075]

[0076]

[0077] in, Represents the initialization rotation value quaternion parameter required by the Gaussian primitive model to obtain the volumetric video of the target scene at time t, The quaternion parameter representing the rotation value required by the Gaussian base model to obtain the volumetric video of the target scene at time t-1 is the initial Gaussian point rotation value quaternion. It represents the residual value of the Gaussian point rotation value quaternion updated by online incremental training after obtaining the volumetric video of the target scene at time t-1, that is, the residual value of the initialized Gaussian point rotation value quaternion.

[0078] According to one embodiment of the present application, the Gaussian primitive model obtains the initial opacity parameter required for the volumetric video of the target scene at time t:

[0079]

[0080]

[0081] Among them, among them, Represents the initial opacity parameter required by the Gaussian primitive model to obtain the volumetric video of the target scene at time t, It represents the opacity parameter required by the Gaussian element model to obtain the volumetric video of the target scene at time t-1, which is the initial Gaussian point opacity; It represents the residual value of the Gaussian point opacity updated by online incremental training after obtaining the volume video of the target scene at time t-1, that is, the residual value of the initialized Gaussian point opacity. is the normalization function.

[0082] From the above description, we can see that since motion is continuous, inspired by this, using the Gaussian residual value of the previous frame as the initialization value of the next frame can effectively reduce the initialization error, thereby reducing the number of required calculation iterations and thus reducing the required reconstruction time.

[0083] The use of a motion chain transfer initialization strategy (i.e., the initialized Gaussian point attribute residual value is the Gaussian point attribute residual value updated by online incremental training after the Gaussian basis element model obtains the volumetric video of the target scene at the previous moment) will lead to error accumulation as the model is trained at subsequent moments, thereby generating unexpected motion artifacts in the static area of ​​the target scene.

[0084] To this end, according to one embodiment of the present application, the regularization loss is:

[0085]

[0086] in, Control hyperparameters for preset locations, Control hyperparameters for preset scales, To preset the rotation control hyperparameters, Opaque control hyperparameters for presets.

[0087] From the above description, it can be seen that the above embodiment of the present application constrains the training process by introducing a temporal consistency regularization loss, that is, the temporal consistency regularization function can ensure that the static area of ​​the target scene is not affected by the cumulative error (the static area refers to the area with the same content at the previous moment and the current moment), while retaining the temporal dynamic characteristics of the moving area. The Gaussian point attribute residual value can maintain good temporal continuity between each moment, reduce error accumulation and reduce unnecessary artifacts in the static area, and improve the quality of the reconstructed video.

[0088] According to one embodiment of the present application, when the current moment is the first moment, the Gaussian point attribute values ​​of the Gaussian primitive model are updated using the error loss and structural similarity loss between the original image of the target scene at the first moment and the rendered image of the target scene.

[0089] As can be seen from the above description, the above embodiment of the present application uses the Gaussian point attribute values ​​converged by training as the basis for reconstructing the volumetric video of the target scene at the subsequent moment, which can improve the reconstruction quality of the volumetric video at the subsequent moment.

[0090] (2) Dynamic Gaussian Sparsification Mechanism

[0091] The dynamic Gaussian sparsification mechanism (DGS) uses the differences of Gaussian points at different moments to dynamically determine the less important Gaussian points in the Gaussian basis model at each moment, thereby reducing the Gaussian point attribute residual values ​​that need to be transmitted at each moment and reducing the required storage overhead.

[0092] The inventors also found through research that in Gaussian space, there are many Gaussian points with low transparency or small size, which have little impact on rendering quality and performance. These unnecessary Gaussian points require the transmission of redundant Gaussian point attribute residual values ​​at every moment, resulting in excessive storage overhead and serious space redundancy.

[0093] To this end, the present application proposes a dynamic Gaussian sparse method, which dynamically judges the importance of Gaussian points at each moment and thins out unimportant Gaussian points (sparsening unimportant Gaussian points means setting the Gaussian point attribute values ​​corresponding to unimportant Gaussian points to 0), thereby reducing the transmission of Gaussian point attribute residual values ​​(that is, the Gaussian points are not actually removed, and therefore will not affect the reconstruction performance at subsequent moments). According to one embodiment of the present application, a Gaussian basis model includes Gaussian points, and the parameters of the initialized Gaussian basis model also include: an initialized Gaussian point mask, and the initialized Gaussian basis model performs online incremental training to update the parameters of the Gaussian basis model. The step also includes: during the online incremental training process, updating the Gaussian point mask and the Gaussian point attribute residual values ​​according to a preset second total loss, and transmitting the updated Gaussian point attribute residual values ​​based on the updated Gaussian point mask, so as to use the transmitted updated Gaussian point attribute residual values ​​as the Gaussian basis model to obtain the Gaussian point attribute residual values ​​required for the volumetric video of the target scene at the next moment.

[0094] Still refer to Figure 2 As shown, Figure 2 The whole process in (b) is called dynamic Gaussian sparsification, starting from Figure 2 As can be seen from the dynamic Gaussian sparsification part (b), after the Gaussian base model completes the volumetric video reconstruction task of the target scene at time t, the dynamic Gaussian sparsification method initializes a Gaussian point mask for each Gaussian point in the Gaussian base model as , Represented as the index of the Gaussian point. This Gaussian point mask determines whether the corresponding Gaussian point is sparse.

[0095] From the above description, it can be seen that the above embodiment of the present application initializes the Gaussian point mask in the Gaussian basis model, updates the Gaussian point mask, and then determines the Gaussian point attribute residual value required by the Gaussian basis model to obtain the volumetric video of the target scene at the next moment based on the updated Gaussian point mask, thereby reducing the Gaussian point attribute residual value required to be transmitted at the current moment to obtain the volumetric video of the target scene at the next moment, thereby reducing the storage overhead while improving the video reconstruction speed.

[0096] According to one embodiment of the present application, the initialized Gaussian point mask is normalized to obtain a Gaussian point soft mask, and the Gaussian point mask and the Gaussian point attribute residual value are updated according to a preset second total loss.

[0097] According to one embodiment of the present application, a Gaussian point soft mask is obtained in the following manner:

[0098]

[0099] in, represents the k-th Gaussian point soft mask at time t, Represents the k-th Gaussian point mask initialized at time t.

[0100] As can be seen from the above description, the above embodiment of the present application performs normalization processing on the initialized Gaussian point mask, which can significantly improve the training stability and convergence efficiency of the model. Normalization unifies the update values ​​of different dimensions to the same scale range, avoiding the gradient imbalance problem caused by excessive or small update amplitudes of certain parameters. This processing method can not only accelerate the convergence process of the model, but also effectively prevent the occurrence of gradient explosion or gradient vanishing phenomena during training, ensuring that the model parameter updates are smoother and more consistent. It helps to improve the accuracy and robustness of the Gaussian primitive model, making it more stable and reliable in complex scenarios.

[0101] Among them, according to one embodiment of the present application, the second total loss includes: the error loss and structural similarity loss between the original image of the target scene at the current moment and the rendered image of the target scene, the regularization loss determined according to the initialized Gaussian point attribute residual value, and the sparse loss determined according to the Gaussian point mask.

[0102] According to one embodiment of the present application, the second total loss is:

[0103]

[0104] in, is the sparse loss. And:

[0105]

[0106] Where G is the total number of Gaussian points in the Gaussian basis model at time t.

[0107] From the above description, it can be seen that by dynamically setting a mask for each Gaussian point in the Gaussian primitive model at each moment, and reducing the Gaussian point attribute residual value required by the Gaussian primitive model to obtain the volumetric video of the target scene at the next moment according to the mask, the storage overhead is reduced while the video reconstruction speed is improved.

[0108] According to one embodiment of the present application, Figure 3 As shown, the steps of transmitting the updated Gaussian point attribute residual value based on the updated Gaussian point mask include: P1, binarizing the updated Gaussian point mask; P2, updating the Gaussian point attribute value and transmitting the updated Gaussian point attribute residual value based on the binarized Gaussian point mask.

[0109] According to one embodiment of the present application, a preset threshold is used to perform binarization processing on the updated Gaussian point soft mask to obtain a Gaussian point hard mask.

[0110] Schematically, in order to soft mask the updated Gaussian points Binarize and use a preset threshold Since the binarization process is non-differentiable, a straight-through estimator is used to predict the gradient. According to an embodiment of the present application, the Gaussian point soft mask is used as follows: Convert to Gaussian point hard mask :

[0111]

[0112] in, is a symbolic function used to stop gradient calculation, is the indicator function. Gaussian point hard mask Is a binary mask whose value can only be 0 or 1. If =0, the updated attribute residual value of the kth Gaussian point is set to 0; otherwise it remains unchanged. hour, =1, then the kth Gaussian point is an important Gaussian point; otherwise, =0, at this time the kth Gaussian point is an unimportant Gaussian point.

[0113] As can be seen from the above description, the above embodiment of the present application uses a mask to thin out unimportant Gaussian points, thereby reducing the transmission of Gaussian point attribute residual values. At the same time, because the present application changes the Gaussian point attribute residual values ​​to be transmitted based on the mask, the Gaussian points are not actually removed, and therefore will not affect the target scene video reconstruction performance at subsequent moments. In addition, the formula of the above embodiment achieves a smooth transition from soft mask to hard mask while maintaining the trainability of the model.

[0114] According to one embodiment of the present application, the Gaussian point scale value and the Gaussian point opacity are updated based on the binarized Gaussian point mask.

[0115] According to one embodiment of the present application, the Gaussian point scale value and the Gaussian point opacity are updated in the following manner:

[0116]

[0117] in, Indicates the scale value of the kth Gaussian point after the update at time t, Indicates the scale value of the kth Gaussian point initialized at time t.

[0118]

[0119] in, Indicates the opacity of the kth Gaussian point after the update at time t, Indicates the opacity of the kth Gaussian point initialized at time t.

[0120] From the above description, it can be seen that the above embodiment of the present application thins out unimportant Gaussian points through masks, sets the Gaussian point opacity and Gaussian point scale values ​​of unimportant Gaussian points to 0, and reduces the transmission of Gaussian point attribute residual values ​​based on the Gaussian point masks, thereby improving the reconstruction speed of volumetric video and reducing storage overhead.

[0121] According to one embodiment of the present application, the Gaussian point attribute residual value required by the Gaussian basis element model to obtain the volumetric video of the target scene at time t+1 is:

[0122]

[0123] in, represents the hard mask of all Gaussian points at time t, Represents the Gaussian point attribute residual value after update at time t.

[0124] It can be seen from the above embodiments that the above embodiments of the present application reduce the transmission of Gaussian point attribute residual values ​​by storing and transmitting the non-zero values ​​of the updated Gaussian point attribute residual values.

[0125] Still refer to Figure 2 As shown, Figure 2 The whole process in (d) is called reconstruction, starting from Figure 2 As can be seen from the reconstruction part (d), the parameters required for the Gaussian primitive model to obtain the volumetric video of the target scene at time t-1 are The Gaussian point attribute residual values ​​(i.e., sparse Gaussian point attribute residual values) updated by online incremental training after obtaining the volumetric video of the target scene at time t-1 are fused to obtain the parameters required by the Gaussian point model to obtain the volumetric video of the target scene at time t. , and use A rendered image of the target scene at time t is obtained, and based on the rendered image of the target scene at time t, a volumetric video GT of the target scene at time t is calculated.

[0126] According to one embodiment of the present application, will be compressed.

[0127] As can be seen from the above description, compressing the updated Gaussian point attribute residual value at time t for storage and transmission can further reduce the required storage overhead and improve the reconstruction speed of subsequent videos.

[0128] As can be seen from the above embodiments, the above embodiments of the present application reduce the transmission of Gaussian point attribute residual values ​​by storing and transmitting the non-zero values ​​of the updated Gaussian point attribute residual values. At the same time, since the present application changes the Gaussian point attribute residual values ​​according to the mask, the Gaussian points are not actually removed, and therefore will not affect the target scene video reconstruction performance at subsequent moments. Since the Gaussian point representation of the scene is essentially order-independent, this means that rearranging the Gaussian points and residual values ​​will not affect the rendering results. Therefore, only the Gaussian point attribute residual values ​​need to be stored and transmitted. and Gaussian point hard mask , the volumetric video reconstruction task of the target scene at subsequent moments can be achieved.

[0129] In order to ensure the smoothness and actual consistency of the volumetric video of the target scene reconstructed at different times, according to one embodiment of the present application, the Gaussian point soft mask at the current time Will be passed on to a subsequent time.

[0130] (3) Adding Gaussian points (i.e. Gaussian Addition)

[0131] The inventors also found through research that since new objects such as flames and water droplets may appear at subsequent moments, it is difficult to reconstruct the video of new objects appearing at subsequent moments by relying solely on the Gaussian point attribute values ​​and Gaussian point attribute residual values ​​residual mapping at previous moments.

[0132] In order to perform high-quality video reconstruction of new objects appearing at subsequent moments, the present invention proposes a Gaussian point addition strategy to efficiently process newly appearing objects.

[0133] Among them, according to one embodiment of the present application, after obtaining the volumetric video of the target scene at the current moment, Gaussian points with preset Gaussian point attribute values ​​are added to the Gaussian base element model, and the preset Gaussian point attribute values ​​and the initialized Gaussian point attribute values ​​are used as new initialized Gaussian point attribute values, and the parameters of the initialized Gaussian base element model include: new initialized Gaussian point attribute values ​​and initialized Gaussian point attribute residual values.

[0134] As can be seen from the above description, the above embodiment of the present application can improve the reconstructed video quality of newly appearing objects at subsequent moments by adding Gaussian points with preset Gaussian point attribute values ​​to form new initialized Gaussian point attribute values.

[0135] The inventors have also discovered through research that newly appearing objects exhibit only very small offsets between adjacent moments. Therefore, according to one embodiment of the present application, after obtaining a volumetric video of the target scene for a preset time period, the Gaussian primitive model adds Gaussian points with preset Gaussian point attribute values ​​to the Gaussian primitive model. The preset Gaussian point attribute values ​​and the initialized Gaussian point attribute values ​​are combined as new initialized Gaussian point attribute values. The parameters of the initialized Gaussian primitive model include the new initialized Gaussian point attribute values ​​and the initialized Gaussian point attribute residual values. The preset time period consists of a preset number of moments.

[0136] Schematically,

[0137] in, represents the volumetric video set of the target scene at time t, represents the volumetric video of the target scene at time t.

[0138] According to one embodiment of the present application, Gaussian points with preset Gaussian point attribute values ​​are added to the Gaussian basis element model under the control of a Gaussian point addition indicator function. The Gaussian point addition indicator function is:

[0139]

[0140]

[0141] in, Add indicator functions for Gaussian points, To set a time, , it means that after obtaining the volumetric video of the target scene at each of the five moments, a Gaussian point with a preset Gaussian point attribute value is added to the Gaussian basis model at time t. In this case, after obtaining the volumetric video of the target scene at time t, the parameters of the Gaussian basis model when adding Gaussian points with the preset Gaussian point attribute value to the Gaussian basis model are:

[0142]

[0143] in, It indicates that the Gaussian primitive model after adding Gaussian points obtains the initialization parameters required for the volumetric video of the target scene at time t. Represents the Gaussian point mask initialized at time t after adding the Gaussian point, It means that the Gaussian basis element model after adding Gaussian points obtains the parameters required for the volumetric video of the target scene at time t-1. It represents the Gaussian point attribute residual value updated by online incremental training after the Gaussian basis element model after adding Gaussian points obtains the volume video of the target scene at time t-1.

[0144] For example, still referring to Figure 2 As shown, Figure 2 The whole process in (c) is called adding Gaussian points, starting from Figure 2 As can be seen from the (c) Adding Gaussian Points part, after the Gaussian base model obtains the volumetric video of the target scene at five moments 1-5, a Gaussian point with a preset Gaussian point attribute value is added to the Gaussian base model at the 5th moment; after the Gaussian base model obtains the volumetric video of the target scene at five moments 6-10, a Gaussian point with a preset Gaussian point attribute value is added to the Gaussian base model at the 10th moment.

[0145] It should be noted that the preset time interval of 5 is only one embodiment of the present application. The specific value can be set according to the changing characteristics of the target scene to be reconstructed and is not specifically limited here. After adding the new Gaussian point, the Gaussian basis model continues to be trained according to the motion residual transfer mechanism (i.e., MoRI) and dynamic Gaussian sparsification mechanism (i.e., DGS) of the above-mentioned embodiment, and the details will not be repeated here.

[0146] As can be seen from the above description, after adding Gaussian points in the above embodiment of the present application, these newly added Gaussian points will be integrated into the initial Gaussian point set for online learning at subsequent moments. Since new objects generally exhibit a tendency to move continuously, compared to directly discarding them, the present application can better reconstruct the volumetric video of the target scene by reusing the attribute residual values ​​of the newly added Gaussian points, while effectively reducing the number of training calculation iterations and storage requirements. (That is, these new Gaussian points will also be used at subsequent moments, and by reusing these Gaussian points, storage requirements and reconstruction time are reduced.)

[0147] According to one embodiment of the present application, the parameters required for obtaining the volumetric video of the target scene at the next moment by using the Gaussian basis element model after adding Gaussian points are compressed using quantization and entropy coding methods.

[0148] As can be seen from the above description, the above embodiment of the present application further compresses the parameters required to obtain the volumetric video of the target scene at the next moment using the Gaussian basis model to achieve lower storage requirements. Because Gaussian points are highly sensitive to positional attributes (i.e., Gaussian point coordinate attribute values ​​and their corresponding residual values), this application uses 16-bit quantization for Gaussian point position information and 8-bit quantization for other attribute information. For the Gaussian point attribute residual values ​​updated using the second preset loss, all residual information is quantized using 8 bits.

[0149] In addition, based on the online volumetric video reconstruction method of any of the above embodiments of the present application, the present application proposes an online volumetric video reconstruction device, and the device of the present application includes: an initialized Gaussian primitive model, which is used to perform volumetric video reconstruction and rendering on the original image of the target scene at the current moment to obtain a rendered image of the target scene at the current moment, and based on the rendered image of the target scene at the current moment, calculate the volumetric video of the target scene at the current moment, wherein the parameters of the initialized Gaussian primitive model are the parameters updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment; after obtaining the volumetric video of the target scene at the current moment, the initialized Gaussian primitive model performs online incremental training to update the parameters of the Gaussian primitive model. It should be noted that the functions performed by the Gaussian primitive model in the video reconstruction process are consistent with those in the above method embodiments and will not be repeated here.

[0150] According to one embodiment of the present application, the original image of the target scene at the current moment is the original image of the target scene at the current moment from multiple perspectives.

[0151] It can be seen from the above embodiments that by reconstructing the volumetric video of the target scene using images from multiple perspectives, a high-quality volumetric video of the target scene from any perspective can be reconstructed.

[0152] Among them, according to an implementation example of the present application, the online volumetric video reconstruction method or device proposed in the present application is configured on medical equipment or live broadcast equipment, etc., which can reconstruct the video of the patient's target area or the high-quality reconstruction of the live broadcast scene in real time.

[0153] In order to verify the effectiveness of the online volumetric video reconstruction solution of any of the above embodiments of the present application, the inventors conducted a series of experiments, and the experimental results are as follows:

[0154] Table 1: Comparative experiment of the online volumetric video reconstruction scheme proposed in this application and the existing volumetric video reconstruction scheme on the dataset N3DV

[0155]

[0156] Table 2: Comparative experiment of the online volumetric video reconstruction scheme proposed in this application and the existing volumetric video reconstruction scheme on the MeetRoom dataset

[0157]

[0158] Table 1 compares the online volumetric video reconstruction solution proposed in this application with existing volumetric video reconstruction solutions on the N3DV dataset. Table 2 compares the online volumetric video reconstruction solution proposed in this application with existing volumetric video reconstruction solutions on the MeetRoom dataset. Both tables show that the online volumetric video reconstruction solution proposed in this application achieves the best performance in terms of reconstruction quality (PSNR), required storage (Storage), training time (Train), and rendering speed (Render). Furthermore, it achieves competitive performance across all metrics compared to offline methods.

[0159] Table 3: Ablation experiment of the motion residual transfer mechanism proposed in this application

[0160]

[0161] Table 3 shows ablation experiments on the motion residual transfer mechanism. ResMap, MCI, and Norm correspond to the experimental results for the introduction of Gaussian point attribute residuals, the motion chain transfer initialization strategy (i.e., the initial Gaussian point attribute residuals are updated by online incremental training after the Gaussian basis model obtains the volumetric video of the target scene at the previous moment), and temporal consistency regularization (introducing the first preset loss). The two datasets used in the experiments are Neural 3D Video (N3DV) and MeetRoom. N3DV contains six indoor 2D videos captured with 18-21 cameras, with a resolution of 2704*2028 and a frame rate of 30 FPS. Following the experimental setup of existing work, this method downsamples the videos by a factor of 2, uses the middle view as the test view, and the remaining views as the training view. MeetRoom contains four indoor 2D videos captured with 13 cameras, with a resolution of 1280*720 and a frame rate of 30 FPS. The middle view is used as the test view, and the remaining views are used as the training view.

[0162] Table 4: Ablation experiments of the dynamic Gaussian point sparse mechanism (DGS) proposed in this application

[0163]

[0164] Table 4 shows an ablation experiment using the dynamic Gaussian point sparser (DGS) mechanism. Static pruning permanently removes unimportant Gaussian points at the first moment. The experimental results demonstrate the effectiveness of this dynamic sparser method. The "without DGS" experiment does not sparse Gaussian points, retaining all Gaussian point attribute residuals. DGS sparses Gaussian points and then transmits the corresponding Gaussian point attribute residuals, thereby reducing the number of parameters required for transmission.

[0165] Table 5: Ablation experiment of the Gaussian addition strategy (FGGA) proposed in this application

[0166]

[0167] Table 5 shows the ablation experiment of the Gaussian addition strategy (FGGA). The experiment shows that without using the strategy of adding Gaussian points in this application, although the required storage and training time are reduced, it is impossible to reconstruct high-quality videos of newly appeared objects. Experiments were conducted. Taking all factors into consideration, the strategy of adding Gaussian points in this application can reduce the required storage and training time while performing high-quality video reconstruction of newly appeared objects.

[0168] In summary, compared with the existing static, offline, and online volumetric video reconstruction methods, the online volumetric video reconstruction method of the present application can not only reconstruct volumetric video with high quality and reconstruct and transmit it frame by frame, but also effectively reduce the required training time and storage overhead, thereby realizing high-quality, low-latency, and low-storage online volumetric video reconstruction. First, the present application proposes a motion residual transfer mechanism, which inherits and reuses the Gaussian point attribute residual information of the previous moment by analyzing the motion continuity of the target scene between adjacent moments, avoids repeated calculations at subsequent moments, and realizes the redundancy removal of the calculation of Gaussian point attribute residual information, thereby effectively improving the training speed of the Gaussian basis element model and the efficiency of video reconstruction; secondly, the present application proposes a dynamic Gaussian point sparsification method, which dynamically selects key Gaussian points for representation of the target scene content at each moment, eliminates redundant Gaussian points, thereby effectively reducing spatial redundancy and reducing the storage required at each moment; finally, the present application proposes a Gaussian point addition strategy, which adds Gaussian points every preset moment according to the small changes in the volumetric video of multiple consecutive moments in time, avoids redundant updates of repeated Gaussian points, and performs efficient video reconstruction on newly appeared objects to further reduce incremental redundancy, thereby reducing computing and storage requirements.

[0169] Based on the inventive concept of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the above embodiments when executing the computer program. Figure 4 Provide detailed explanation.

[0170] like Figure 4 As shown in FIG, the electronic device 100 of the present application is shown, which may specifically include a processor 110 and a memory 120. The memory 120 is coupled to the processor 110.

[0171] Processor 110 is used to control the operation of the electronic device. Processor 110 may also be referred to as a CPU (Central Processing Unit). Processor 110 may be an integrated circuit chip with signal processing capabilities. Processor 110 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor, or processor 110 may be any conventional processor.

[0172] The memory 120 is used to store computer programs and can be RAM, ROM, or other types of storage terminals. Specifically, the memory 120 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 120 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage terminals and flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 120 is used to store at least one program code.

[0173] The processor 110 is configured to execute the computer program stored in the memory 120 to implement the methods described in the various method embodiments of the present application.

[0174] In some embodiments, the electronic device may further include a peripheral terminal interface 130 and at least one peripheral terminal. The processor 110, memory 120, and peripheral terminal interface 130 may be connected via a bus or signal lines. Each peripheral terminal may be connected to the peripheral terminal interface 130 via a bus, signal lines, or circuit boards. Specifically, the peripheral terminal includes at least one of a radio frequency circuit 140, a display screen 150, an audio circuit 160, and a power supply 170.

[0175] The peripheral terminal interface 130 can be used to connect at least one peripheral terminal related to I / O (Input / Output) to the processor 110 and the memory 120. In some embodiments, the processor 110, the memory 120, and the peripheral terminal interface 130 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 110, the memory 120, and the peripheral terminal interface 130 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0176] The RF circuit 140 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 140 communicates with communication networks and other IoT devices via electromagnetic signals, and is therefore the communication circuitry of the electronic device. The RF circuit 140 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 140 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identification module card, and the like. The RF circuit 140 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 140 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0177] The display screen 150 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 150 is a touch screen display, the display screen 150 is also capable of collecting touch signals on or above the surface of the display screen 150. The touch signals can be input as control signals to the processor 110 for processing. In this case, the display screen 150 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 150, disposed on the front panel of the electronic device; in other embodiments, there can be at least two display screens 150, disposed on different surfaces of the electronic device or in a foldable design; in still other embodiments, the display screen 150 can be a flexible display, disposed on a curved or foldable surface of the electronic device. Furthermore, the display screen 150 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 150 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0178] The audio circuit 160 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 110 for processing, or input them into the radio frequency circuit 140 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively set in different parts of the electronic device. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 110 or the radio frequency circuit 140 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 160 may also include a headphone jack.

[0179] Power supply 170 is used to power various components in the electronic device. Power supply 170 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 170 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0180] For a detailed description of the functions and execution processes of the various functional modules or components in the electronic device embodiments of the present application, reference can be made to the descriptions in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some data can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0182] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0183] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0184] Based on the inventive concept of the above embodiments, the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are described. Figure 5 The execution process of the above embodiment in a computer-readable storage medium is described.

[0185] like Figure 5As shown, it shows the computer-readable storage medium of the present application. If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the computer-readable storage medium 200. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions / computer programs for enabling an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks or optical disks, as well as electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. that have the above-mentioned storage media.

[0186] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned various method embodiments of the present application, and will not be repeated here.

[0187] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

[0188] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

Claims

1. An online volumetric video reconstruction method, characterized in that: The method comprises: Get the original image of the target scene at the current moment; An initialized Gaussian primitive model is used to reconstruct and render the original image of the target scene at the current moment to obtain a rendered image of the target scene at the current moment, and based on the rendered image of the target scene at the current moment, a volumetric video of the target scene at the current moment is calculated, wherein the parameters of the initialized Gaussian primitive model include: initialized Gaussian point attribute values ​​and initialized Gaussian point attribute residual values, and the initialized Gaussian point attribute values ​​are the parameters required by the Gaussian primitive model to obtain the volumetric video of the target scene at the previous moment, and the initialized Gaussian point attribute residual values ​​are the Gaussian point attribute residual values ​​updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment; After obtaining the volumetric video of the target scene at the current moment, the initialized Gaussian primitive model is subjected to online incremental training to update the parameters of the Gaussian primitive model, wherein, during the online incremental training process, the Gaussian point attribute residual value is updated according to a preset first total loss.

2. The online volumetric video reconstruction method according to claim 1, characterized in that The first total loss includes: an error loss and a structural similarity loss between the original image of the target scene and the rendered image of the target scene at the current moment, and a regularization loss determined according to the initialized Gaussian point attribute residual value.

3. The online volumetric video reconstruction method according to claim 2, characterized in that The Gaussian primitive model includes Gaussian points, and the initialized parameters of the Gaussian primitive model further include: an initialized Gaussian point mask, and the initialized Gaussian primitive model is subjected to online incremental training to update the parameters of the Gaussian primitive model. The steps include: During the online incremental training process, the Gaussian point mask and the Gaussian point attribute residual value are updated according to the preset second total loss, and based on the updated Gaussian point mask, the Gaussian point attribute value and the updated Gaussian point attribute residual value are updated, so that the updated Gaussian point attribute residual value transmitted is used as the Gaussian basis element model to obtain the Gaussian point attribute residual value required for the volumetric video of the target scene at the next moment.

4. The online volumetric video reconstruction method according to claim 3, characterized in that The second total loss includes: the error loss and structural similarity loss between the original image of the target scene and the rendered image of the target scene at the current moment, the regularization loss determined according to the initialized Gaussian point attribute residual value, and the sparse loss determined according to the Gaussian point mask.

5. The online volumetric video reconstruction method according to claim 3, characterized in that The step of updating the Gaussian point attribute value and transmitting the updated Gaussian point attribute residual value based on the updated Gaussian point mask includes: performing binarization processing on the updated Gaussian point mask; Based on the binarized Gaussian point mask, the Gaussian point attribute value is updated and the updated Gaussian point attribute residual value is transmitted.

6. The online volumetric video reconstruction method according to claim 1, characterized in that After the Gaussian primitive model obtains the volumetric video of the target scene in a preset time period, a Gaussian point with a preset Gaussian point attribute value is added to the Gaussian primitive model, and the preset Gaussian point attribute value and the initialized Gaussian point attribute value are used as the new initialized Gaussian point attribute value, and the parameters of the initialized Gaussian primitive model include: the new initialized Gaussian point attribute value and the initialized Gaussian point attribute residual value.

7. An online volumetric video generation device based on the online volumetric video reconstruction method according to claim 1, characterized in that: The device includes: an initialized Gaussian primitive model, which is used to perform volumetric video reconstruction and rendering on an original image of a target scene acquired at the current moment to obtain a rendered image of the target scene at the current moment, and calculate the volumetric video of the target scene at the current moment based on the rendered image of the target scene at the current moment, wherein the parameters of the initialized Gaussian primitive model include: initialized Gaussian point attribute values ​​and initialized Gaussian point attribute residual values, and the initialized Gaussian point attribute values ​​are the parameters required by the Gaussian primitive model to obtain the volumetric video of the target scene at the previous moment, and the initialized Gaussian point attribute residual values ​​are the Gaussian point attribute residual values ​​updated by online incremental training after the Gaussian primitive model obtains the volumetric video of the target scene at the previous moment; After obtaining the volumetric video of the target scene at the current moment, the initialized Gaussian primitive model is subjected to online incremental training to update the parameters of the Gaussian primitive model, wherein, during the online incremental training process, the Gaussian point attribute residual value is updated according to a preset first total loss.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images