Three-dimensional dynamic scene compression method based on multi-resolution codebook, computer equipment and program product

Abstract

The invention relates to a three-dimensional dynamic scene compression method based on a multi-resolution codebook, computer equipment and a program product, and the three-dimensional dynamic scene compression method comprises the steps: carrying out the reconstruction of an input view of the same dynamic scene at different moments and different visual angles, and obtaining a three-dimensional dynamic scene based on a 3D Gaussian point in a space, each 3D Gaussian point has a Gaussian point attribute; optimizing Gaussian point attributes according to the input view and the rendered image of the three-dimensional scene at the same moment and under the same view angle; taking the Gaussian point position and the dynamic scene time as input to obtain Gaussian point attribute variation, and superposing the Gaussian point attribute variation to Gaussian point attributes to obtain three-dimensional scenes at different moments; a plurality of feature codebooks are correspondingly provided according to the number of resolutions of the deformation field feature plane, the feature codebooks are used for synchronous optimization with the deformation field feature plane, and feature codebook values of the feature codebooks are used for one-to-many mapping and replacement with feature values in the deformation field feature plane under the corresponding resolutions.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a three-dimensional dynamic scene compression method, computer device and program product based on a multi-resolution codebook. Background Art

[0002] 3D dynamic scene reconstruction is an important task in the field of vision and graphics. It can model changing 3D scenes and render 2D images from any perspective at different times. Given a series of pictures taken by a monocular camera at different times and from different perspectives, it can model the entire 3D dynamic scene and generate high-quality images from any novel perspective. Since the scene changes over time, using only static 3D scene reconstruction methods often produces poor results. In order to model 3D dynamic scenes, existing methods have explored a variety of methods for accurately modeling 3D dynamic scenes. However, they have problems such as long training time and large storage space.

[0003] Neural Radiance Fields (NeRF) proposed to use neural networks to model 3D scenes. Due to its excellent performance in reconstructing geometry and appearance, it has become an excellent scene representation. Due to its reliability, there are currently many methods that use it to reconstruct 3D dynamic scenes. D-NeRF (D-NeRF: Neural Radiance Fields for Dynamic Scenes) combines NeRF with deformation fields and uses input views to jointly optimize the two, effectively improving the reconstruction quality of 3D dynamic scenes. Hexplane (HexPlane: A Fast Representation for Dynamic Scenes) uses 6 feature planes and a small decoder to model 3D dynamic scenes, further improving the quality of scene rendered images. However, despite valuable efforts, these methods still have a lot of room for improvement in terms of training time, rendering time, storage space, etc.

[0004] Recently, the introduction of 3D Gaussian Splatting (3DGS) provides a new idea for 3D dynamic scene reconstruction. 4DGS (4D Gaussian Splatting) first introduced 3DGS to the task of 3D dynamic scene reconstruction and achieved results that surpassed previous work. However, since 3DGS is an explicit representation and requires a large number of parameters to be stored when constructing the scene, 4DGS is difficult to achieve satisfactory results in terms of storage space. Summary of the invention

[0005] Based on this, it is necessary to provide a three-dimensional dynamic scene compression method based on a multi-resolution codebook to address the above technical problems.

[0006] The present application provides a three-dimensional dynamic scene compression method based on a multi-resolution codebook, including:

[0007] Reconstructing a three-dimensional dynamic scene based on 3D Gaussian points in space using input views of the same dynamic scene at different times and different perspectives, each of the 3D Gaussian points having Gaussian point attributes;

[0008] Optimizing the Gaussian point attributes according to the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle;

[0009] The deformation field based on 4D Gaussian realizes the dynamic change of the scene over time, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, superimposing the Gaussian point attribute change to the Gaussian point attribute, and obtaining the three-dimensional scene at different times;

[0010] The deformation field feature plane of the deformation field is compressed, comprising: providing a plurality of feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks being used to be optimized synchronously with the deformation field feature plane, and feature codebook values ​​of the feature codebooks being used to perform one-to-many mapping and replacement with feature values ​​in the deformation field feature plane at corresponding resolutions.

[0011] Optionally, the three-dimensional dynamic scene compression method includes deleting Gaussian points, specifically including:

[0012] Obtaining the importance value score of each of the 3D Gaussian points, comprising: for any 3D Gaussian point, taking the sum of the volume rendering weights of all pixels of the rendered image under a single viewing angle as a candidate value, and taking the maximum value of the candidate values ​​of each of the 3D Gaussian points under all viewing angles as the importance value score of the 3D Gaussian point;

[0013] According to the descending order of the importance value scores, some 3D Gaussian points with the importance value scores ranked later are deleted.

[0014] Optionally, the three-dimensional dynamic scene compression method includes compressing the Gaussian point attributes, specifically including:

[0015] Providing a plurality of attribute codebooks corresponding to different Gaussian point attributes, wherein attribute codebook values ​​in the attribute codebooks are used for one-to-many mapping and replacement with Gaussian point attribute values ​​of different 3D Gaussian points;

[0016] Optimizing the 3D Gaussian consists of a first stage and a second stage performed sequentially:

[0017] In the first stage, the Gauss point attributes are optimized according to the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle;

[0018] In the second stage, according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle, the Gauss point attributes and the attribute codebook are optimized simultaneously to obtain a mapping table of a one-to-many relationship between the attribute codebook values ​​and the corresponding Gauss point attribute values ​​mapped and replaced, and during the rendering process, the attribute codebook values ​​are used to replace the corresponding Gauss point attribute values ​​in a one-to-many manner according to the mapping table.

[0019] Optionally, the Gaussian point attributes include scaling attributes, rotation attributes and color attributes, and the attribute codebook correspondingly includes a scaling codebook, a rotation codebook and a color codebook.

[0020] Optionally, optimizing the Gaussian point attribute, and optimizing the Gaussian point attribute and the attribute codebook simultaneously, are both performed using the following loss function:

[0021] L gs =(1-λ)L1+λL D-SSIM

[0022] Where, L gs represents the loss function;

[0023] L1, represents the average of the absolute differences between the input view and the rendered image of the 3D scene at the same viewing angle;

[0024] L D-SSIM , represents the structural similarity loss between the rendered image and the input image.

[0025] Optionally, the Gaussian point position and the dynamic scene time are both used as inputs, and the Gaussian point attribute change is obtained through the deformation field, specifically including:

[0026] The Gaussian point position and the dynamic scene time are both used as inputs, and a feature vector is obtained from a deformation field feature plane in the deformation field; the feature vector is input into multiple multi-layer perceptrons to obtain a Gaussian point attribute change.

[0027] Optionally, compressing the deformation field characteristic plane of the deformation field is performed on some characteristic values ​​in the deformation field characteristic plane, and specifically includes:

[0028] Providing a characteristic importance value score plane having the same shape as the deformation field characteristic plane of the deformation field, wherein the characteristic importance value score plane is used to accompany the deformation field characteristic plane optimization and to obtain a characteristic value score;

[0029] According to the order of the eigenvalue scores, part of the eigenvalues ​​in the deformation field feature plane are mapped to the feature codebook values ​​in a many-to-one manner.

[0030] Optionally, the three-dimensional dynamic scene compression method further includes:

[0031] Obtaining a mapping relationship between the characteristic codebook value and the corresponding characteristic value in the deformation field characteristic plane;

[0032] Providing a difference codebook, the difference codebook is used to store the quantization parameter difference between the following two: the original value of the corresponding eigenvalue replaced in the deformation field feature plane, and the feature codebook value used to replace the corresponding eigenvalue in the deformation field feature plane;

[0033] When the feature codebook value is used to replace the corresponding feature value in the deformation field feature plane, the quantization parameter difference in the difference codebook is retrieved to reduce the difference caused by the replacement process.

[0034] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook described in the present application.

[0035] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook described in the present application.

[0036] The three-dimensional dynamic scene compression method based on the multi-resolution codebook of the present application has at least the following effects:

[0037] The present application realizes compression of deformation field parameters by performing one-to-many mapping and replacement of the feature codebook values ​​of the feature codebook with the corresponding feature values ​​in the deformation field feature plane, effectively reducing the storage space of the dynamic scene. The feature codebook and the deformation field feature plane are optimized synchronously, taking into account the generation quality of the scene rendering image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a three-dimensional dynamic scene compression method based on a multi-resolution codebook in one embodiment of the present application;

[0039] Figure 2 This is a model architecture diagram of step S200 in a three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0040] Figure 3 This is a model architecture diagram of step S300 in a three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0041] Figure 4 This is a model architecture diagram of step S400 in a three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0042] Figure 5 This is a model architecture diagram of step 500 in a three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0043] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] See also Figure 1 An embodiment of the present application provides a three-dimensional dynamic scene compression method based on a multi-resolution codebook, comprising steps S100 to S400, wherein:

[0046] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0047] Step S200 , optimizing Gaussian point attributes according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0048] Step S300, realizing the dynamic change of the scene over time based on the deformation field of 4D Gaussian, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, superimposing the Gaussian point attribute change to the Gaussian point attribute, and obtaining the three-dimensional scene at different times.

[0049] Step S400, compressing the deformation field feature plane of the deformation field, including: providing multiple feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks are used to be optimized synchronously with the deformation field feature plane, and the feature codebook values ​​of the feature codebooks are used to perform one-to-many mapping and replacement with the feature values ​​in the deformation field feature plane at the corresponding resolution.

[0050] See also Figure 1 to Figure 3 In one embodiment, the present application provides a three-dimensional dynamic scene compression method based on a multi-resolution codebook, adding optional sub-steps and subsequent steps on the basis of the previous embodiment. It includes steps S100 to S500. Among them:

[0051] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0052] Step S200 , optimizing Gaussian point attributes according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0053] This embodiment mainly adopts a three-dimensional explicit expression model based on 3DGS, that is, the three-dimensional scene reconstructed according to the input view uses 3D Gaussian points in space for accurate geometric modeling and visual rendering. In step S100, the implementation method of reconstructing the three-dimensional dynamic scene through input views at different times and different perspectives can be referred to the relevant document on 4D Gaussian in the prior art, "CVPR 2024 4DGaussian Splatting for Real-Time Dynamic Scene Rendering".

[0054] 3DGS projects Gaussian points in space onto a 2D plane according to the viewing direction and obtains the final image through volume rendering. However, since multiple attributes of each Gaussian point need to be optimized, the final reconstructed scene consumes a lot of storage space.

[0055] See also Figure 2 , step S210, the three-dimensional dynamic scene compression method includes deleting Gaussian points. As an optional sub-step, step S210 specifically includes steps S211 to S212:

[0056] Step S211, obtaining the importance value score of each 3D Gaussian point, including: for any 3D Gaussian point, taking the sum of the volume rendering weights of all pixels in the rendered image under a single viewing angle as a candidate value, and taking the maximum value of the candidate values ​​of each 3D Gaussian point under all viewing angles as the importance value score of the 3D Gaussian point;

[0057] Step S212, according to the order of importance value scores from large to small, delete some 3D Gaussian points (referred to as Gaussian points for short) with the importance value scores ranked low.

[0058] Step S210 is a spatial Gaussian point compression method based on pruning, that is, unimportant Gaussian points in the three-dimensional space are discarded, thereby reducing the storage space occupied by the Gaussian points in the scene.

[0059] Step S211 prunes the Gaussian points in the space by calculating the importance score of each Gaussian point. In this way, the number of Gaussian points in the space can be effectively reduced while ensuring the quality of image rendering, and the various properties of Gaussian can be compactly represented. Under a single viewing angle, the sum of the volume rendering weights of the Gaussian points to the global pixels is used as a candidate value for the importance score of the Gaussian point, and the maximum value is selected from the candidate values ​​of all viewing angles as the importance score of the Gaussian point. Compared with the method of using the weight accumulation method under all viewing angles as the importance score, this method better preserves the details in the scene.

[0060] Specifically, the process of rendering an image is described by the following formula:

[0061]

[0062] Where:

[0063] r, represents the ray under the rendering perspective;

[0064] Indicates the ray color at the rendering perspective;

[0065] Indicates the color of the Gaussian point;

[0066] T i (1-exp(-σ i δ i ), represents the weight in the volume rendering process, referred to as volume rendering weight;

[0067]

[0068] σ i , represents the opacity corresponding to the Gaussian point in space;

[0069] δ i , represents the distance between adjacent sampling points in the ray direction, δ i =t i+1 -t i , t i is the sampling point, used to represent the sampling of Gaussian points.

[0070] According to equation (1), the importance of each Gaussian point can be intuitively represented by the weight in the volume rendering process. Step S211 uses the maximum value of the sum of the volume rendering weights of the Gaussian point under all viewing angles as the importance value score of the Gaussian point:

[0071]

[0072] Where:

[0073] Score(G i), represents the i-th Gaussian point G i The importance score of

[0074] I, represents all rendering perspectives;

[0075] j, represents one of the rendering perspectives I;

[0076] r represents the ray under the rendering perspective.

[0077] After obtaining the importance value scores of the Gaussian points, step S212 determines the Gaussian points that can be removed through a custom threshold and performs a pruning operation. The custom threshold can be, for example, a percentage threshold. See the following formula:

[0078] m i =Score(G i )<τ prune (3)

[0079] m i , represents a threshold-based binary mask used to select Gaussian points that need to be pruned (i.e. deleted);

[0080] τ prune , represents the percentage threshold.

[0081] Step S212 is now completed. Step S220 uses the weight and maximum value under a single perspective as the important value score of the Gaussian point, which can better maintain the details in the scene. After pruning, the number of Gaussian points in the space is significantly reduced, reducing the final storage space.

[0082] See also Figure 2 , step S220, the three-dimensional dynamic scene compression method includes compressing Gaussian point attributes. As an optional sub-step, step S220 specifically includes steps S221 to S223:

[0083] Step S221, providing multiple attribute codebooks corresponding to different Gaussian point attributes (i.e. Figure 2 The attribute codebook values ​​in the attribute codebook are used for one-to-many mapping and replacement with the Gaussian point attribute values ​​of different 3D Gaussian points;

[0084] Step S222, optimizing the 3D Gaussian includes a first stage and a second stage performed in sequence: in the first stage, optimizing Gaussian point attributes according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle;

[0085] Step S223, in the second stage, according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle, the Gaussian point attributes and the attribute codebook are optimized simultaneously to obtain a mapping table of a one-to-many relationship between the attribute codebook value and the corresponding Gaussian point attribute value to be mapped and replaced, and during the rendering process, the corresponding Gaussian point attribute value is replaced in a one-to-many manner using the attribute codebook value according to the mapping table.

[0086] Step S220 compresses the attributes of the Gaussian points based on vector quantization. The Gaussian point attributes include scaling attributes, rotation attributes, and color attributes (sh coefficients). Correspondingly, the attribute codebook provided in step S221 includes: scaling codebook, rotation codebook, and color codebook.

[0087] The codebook size provided in step S221 can be set as needed, and the attribute codebook can be understood as a pre-provided multidimensional vector. The first-stage optimization in step S222 is based on the technical idea of ​​3D Gaussian optimization in the prior art. On the basis of the first stage in step S223, the Gaussian point attributes and the attribute codebook are optimized simultaneously. During the optimization process, the K-means clustering algorithm is used to cluster the attribute vectors into K clusters, and the attribute codebook value corresponding to the cluster is set according to the importance score of the Gaussian point in the cluster, thereby realizing the mapping relationship between the Gaussian point attributes and the attribute codebook value. By setting the appropriate codebook size, the relationship between the compression ratio and the rendering quality can be effectively balanced.

[0088] Since a one-to-many mapping table of attribute codebook values ​​and corresponding Gaussian point attribute values ​​has been obtained in step S222, taking color attribute as an example, the color codebook in the attribute codebook uses a vector quantization method to compress color information, further reducing storage space while ensuring generation quality.

[0089] For step S223, during the rendering process, the original Gaussian point attributes (such as color attribute values, etc.) are replaced by a mapping table according to the attribute codebook, and the codebook and other Gaussian attributes are optimized according to the loss function used in 3DGS. The optimization of Gaussian point attributes and the optimization of Gaussian point attributes and attribute codebooks are performed using the following loss function:

[0090] L gs =(1-λ)L1+λL D-SSIM

[0091] Where, L gs represents the loss function;

[0092] L1, represents the average of the absolute differences between the input view and the rendered image of the 3D scene at the same viewing angle;

[0093] L D-SSIM, represents the structural similarity loss between the rendered image and the input image.

[0094] Step S300, realizing dynamic changes of the scene over time based on the 4D Gaussian deformation field, includes:

[0095] The Gaussian point position and dynamic scene time are used as inputs, and feature vectors are obtained from the deformation field feature plane in the deformation field; the feature vectors are input into multiple multi-layer perceptrons ( Figure 3 MLP in) to obtain the Gaussian point attribute change ( Figure 3 The ΔV in the figure is used to add the Gaussian point attribute changes to the Gaussian point attributes to obtain the three-dimensional scenes at different times.

[0096] For the specific implementation of step S300, reference may be made to the related literature on 4D Gaussian in the prior art.

[0097] See also Figure 4 and Figure 5 Step S400, compressing the deformation field feature plane of the deformation field, including: providing a plurality of feature codebooks (i.e. Figure 5 The codebook shown in , where codebook 1 and codebook 2 correspond to different resolutions), the feature codebook is used to optimize synchronously with the deformation field feature plane, and the feature codebook value of the feature codebook is used to perform one-to-many mapping and replacement with the feature value in the deformation field feature plane at the corresponding resolution.

[0098] In practical applications, the deformation field of 4DGS (4D Gaussian) often occupies a large storage space, and it is difficult to effectively reduce the final storage requirement by simply compressing the Gaussian points. Step S400 proposes a deformation field compression method based on a multi-resolution codebook, which effectively reduces the storage space of the deformation field.

[0099] Specifically, the number of feature codebooks is first determined according to the number of feature plane resolutions in the deformation field of 4DGS. Then, feature codebooks of different sizes are used for feature planes of deformation fields with different resolutions for vector quantization. Vector quantization is to perform a many-to-one mapping of the feature values ​​in the deformation field feature planes with the feature codebook values.

[0100] Further, in step S400, the deformation field feature plane is compressed, and the compression is performed on some eigenvalues ​​in the deformation field feature plane, specifically including: providing a feature importance value score plane with the same shape as the deformation field feature plane, the feature importance value score plane is used to accompany the deformation field feature plane optimization and to obtain the eigenvalue scores; according to the sorting of the eigenvalue scores, some eigenvalues ​​in the deformation field feature plane are mapped to the feature codebook values ​​accordingly in a many-to-one manner.

[0101] This process does not directly calculate the importance value scores of the features in the feature plane, but obtains the eigenvalue scores corresponding to each feature by optimizing the importance value score plane with the same shape as the feature plane, thereby guiding the subsequent quantization process.

[0102] In order to map each eigenvalue to the correct codebook value (feature codebook value), we randomly select a batch of eigenvalues ​​at each iteration and calculate the Euclidean distance between the selected eigenvalue and each codebook value in the codebook to determine which codebook value the eigenvalue is associated with, and modify the codebook value through the eigenvalue importance score.

[0103] Similar to the idea of ​​Gaussian point pruning, after the eigenvalue scores are sorted, they are divided into three parts: the first part is the part with higher eigenvalue scores, which is retained, that is, step S400 is not performed on the first part. The second part is the part with lower eigenvalue scores, which is set to a constant or set to zero. The third part is the part with a middle eigenvalue score, which is vector quantized, that is, step S400 is performed on the third part.

[0104] In the third part, in order to ensure the final generation quality, the loss L is introduced vq To reduce the difference between the characteristic codebook value (codebook vector) and the characteristic value (actual vector) in the deformation field characteristic plane.

[0105]

[0106] Among them, N represents the number of eigenvalues ​​involved in each calculation, f i represents the eigenvalue in the characteristic plane of the deformation field, r i Represents the signature code value.

[0107] Step S500, using the difference codebook to reduce the difference between the feature codebook value and the actual feature plane feature value. As an optional subsequent step, step S500 specifically includes steps S510 to S530:

[0108] Step S510, obtaining a mapping relationship between a feature codebook value and a corresponding feature value in a deformation field feature plane;

[0109] Step S520, providing a difference codebook ( Figure 5 The codebook in '), the difference codebook is used to store the quantization parameter difference between the following two: the original value of the corresponding eigenvalue replaced in the deformation field feature plane, and the feature codebook value used to replace the corresponding eigenvalue in the deformation field feature plane;

[0110] Step S530, when the feature code book value is used to replace the corresponding feature value in the deformation field feature plane, the quantization parameter difference in the difference code book is retrieved to reduce the difference caused by the replacement process. The mapping relationship of step S510 is the "partial feature value in the deformation field feature plane is mapped many-to-one with the feature code book value" in step S400. In step S520, a fixed-size difference code book is additionally designed to store the difference between the quantized parameter and the original parameter. The difference code book can be accessed by feature code books of all resolutions, thereby reducing the difference between the quantized value (feature code book value) and the actual value (feature value in the deformation field feature plane) to ensure the quality of the scene rendering picture.

[0111] The three-dimensional dynamic scene compression method based on a multi-resolution codebook provided in each embodiment of the present application can compress the 3D Gaussian sputtering dynamic scene and perform lightweight representation of the three-dimensional dynamic scene, while maintaining the rendering quality of the three-dimensional dynamic scene during compression. Steps S210 and S220 perform 3D Gaussian compression, and make the representation of the scene more compact through methods such as pruning and codebook quantization. Step S400 performs deformation field compression, and compresses the parameters of the deformation field through a multi-resolution codebook, effectively reducing the storage space of the dynamic scene.

[0112] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional dynamic scene compression method based on a multi-resolution codebook is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0114] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0115] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0116] Step S200 , optimizing Gaussian point attributes according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0117] Step S300, realizing the dynamic change of the scene over time based on the deformation field of 4D Gaussian, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, superimposing the Gaussian point attribute change to the Gaussian point attribute, and obtaining the three-dimensional scene at different times.

[0118] Step S400, compressing the deformation field feature plane of the deformation field, including: providing multiple feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks are used to be optimized synchronously with the deformation field feature plane, and the feature codebook values ​​of the feature codebooks are used to perform one-to-many mapping and replacement with the feature values ​​in the deformation field feature plane at the corresponding resolution.

[0119] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0120] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0121] Step S200 , optimizing Gaussian point attributes according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0122] Step S300, realizing the dynamic change of the scene over time based on the deformation field of 4D Gaussian, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, superimposing the Gaussian point attribute change to the Gaussian point attribute, and obtaining the three-dimensional scene at different times.

[0123] Step S400, compressing the deformation field feature plane of the deformation field, including: providing a plurality of feature codebooks correspondingly according to the number of resolutions of the deformation field feature plane, the feature codebooks are used to optimize synchronously with the deformation field feature plane, and the feature codebook values ​​of the feature codebooks are used to perform one-to-many mapping and replacement with the feature values ​​in the deformation field feature plane at the corresponding resolution. In this embodiment, the computer program product includes a program code portion, which is used to execute the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product can be stored on a computer-readable recording medium. The computer program product can also be provided for downloading via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method can be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality can be provided for downloading with the aid of a hardware description language.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0125] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0126] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional dynamic scene compression method based on a multi-resolution codebook, characterized in that: include: Reconstructing a three-dimensional dynamic scene based on 3D Gaussian points in space using input views of the same dynamic scene at different times and different perspectives, each of the 3D Gaussian points having Gaussian point attributes; Optimizing the Gaussian point attributes according to the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle; The deformation field based on 4D Gaussian realizes the dynamic change of the scene over time, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, superimposing the Gaussian point attribute change to the Gaussian point attribute, and obtaining the three-dimensional scene at different times; The deformation field feature plane of the deformation field is compressed, comprising: providing a plurality of feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks being used to be optimized synchronously with the deformation field feature plane, and feature codebook values ​​of the feature codebooks being used to perform one-to-many mapping and replacement with feature values ​​in the deformation field feature plane at corresponding resolutions.

2. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The three-dimensional dynamic scene compression method includes deleting Gaussian points, specifically including: Obtaining the importance value score of each of the 3D Gaussian points, comprising: for any 3D Gaussian point, taking the sum of the volume rendering weights of all pixels of the rendered image under a single viewing angle as a candidate value, and taking the maximum value of the candidate values ​​of each of the 3D Gaussian points under all viewing angles as the importance value score of the 3D Gaussian point; According to the descending order of the importance value scores, some 3D Gaussian points with the importance value scores ranked later are deleted.

3. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The three-dimensional dynamic scene compression method includes compressing the Gaussian point attributes, specifically including: Providing a plurality of attribute codebooks corresponding to different Gaussian point attributes, wherein attribute codebook values ​​in the attribute codebooks are used for one-to-many mapping and replacement with Gaussian point attribute values ​​of different 3D Gaussian points; Optimizing the 3D Gaussian consists of a first stage and a second stage performed sequentially: In the first stage, the Gauss point attributes are optimized according to the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle; In the second stage, according to the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle, the Gauss point attributes and the attribute codebook are optimized simultaneously to obtain a mapping table of a one-to-many relationship between the attribute codebook values ​​and the corresponding Gauss point attribute values ​​mapped and replaced, and during the rendering process, the attribute codebook values ​​are used to replace the corresponding Gauss point attribute values ​​in a one-to-many manner according to the mapping table.

4. The three-dimensional dynamic scene compression method according to claim 3, characterized in that: The Gaussian point attributes include scaling attributes, rotation attributes and color attributes, and the attribute codebook correspondingly includes a scaling codebook, a rotation codebook and a color codebook.

5. The three-dimensional dynamic scene compression method according to claim 3, characterized in that: Optimizing the Gaussian point attribute, and optimizing the Gaussian point attribute and the attribute codebook simultaneously, are both performed using the following loss function: L gs =(1-λ)L1+λL D-SSIM Where, L gs represents the loss function; L1, represents the average of the absolute differences between the input view and the rendered image of the 3D scene at the same viewing angle; L D-SSIM , represents the structural similarity loss between the rendered image and the input image.

6. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The Gaussian point position and the dynamic scene time are both used as inputs, and the Gaussian point attribute change is obtained through the deformation field, specifically including: The Gaussian point position and the dynamic scene time are both used as inputs, and a feature vector is obtained from a deformation field feature plane in the deformation field; the feature vector is input into multiple multi-layer perceptrons to obtain a Gaussian point attribute change.

7. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The compressing of the deformation field characteristic plane of the deformation field is performed on some characteristic values ​​in the deformation field characteristic plane, and specifically includes: Providing a characteristic importance value score plane having the same shape as the deformation field characteristic plane of the deformation field, wherein the characteristic importance value score plane is used to accompany the deformation field characteristic plane optimization and to obtain a characteristic value score; According to the order of the eigenvalue scores, part of the eigenvalues ​​in the deformation field feature plane are mapped to the feature codebook values ​​in a many-to-one manner.

8. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The three-dimensional dynamic scene compression method also includes: Obtaining a mapping relationship between the characteristic codebook value and the corresponding characteristic value in the deformation field characteristic plane; Providing a difference codebook, the difference codebook is used to store the quantization parameter difference between: the original value of the corresponding eigenvalue replaced in the deformation field feature plane, and the feature codebook value used to replace the corresponding eigenvalue in the deformation field feature plane; When the feature codebook value is used to replace the corresponding feature value in the deformation field feature plane, the quantization parameter difference in the difference codebook is retrieved to reduce the difference caused by the replacement process.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instruction is executed by a processor, the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook described in any one of claims 1 to 8 are implemented.

Images