Method for three-dimensional dynamic scene compression based on multi-resolution codebook, computer device and program product

Abstract

The application relates to a three-dimensional dynamic scene compression method based on a multi-resolution codebook, a computer device and a program product, the three-dimensional dynamic scene compression method comprising the following steps: reconstructing a three-dimensional dynamic scene based on 3D Gaussian points in space by using input views of different perspectives at different time instants of the same dynamic scene, wherein each 3D Gaussian point has a Gaussian point attribute; optimizing the Gaussian point attribute according to the input views and a rendered image of the three-dimensional scene at the same time instant and the same perspective; taking the Gaussian point position and the dynamic scene time as inputs, obtaining a Gaussian point attribute change amount, superimposing the Gaussian point attribute change amount on the Gaussian point attribute, and obtaining the three-dimensional scene at different time instants; and providing a plurality of feature codebooks according to the resolution number of a deformation field feature plane, wherein the feature codebooks are used for synchronous optimization of the deformation field feature plane, and feature codebook values of the feature codebooks are used for one-to-many mapping and replacement with feature values in the deformation field feature plane at the corresponding resolution.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a three-dimensional dynamic scene compression method, computer device, and program product based on a multi-resolution codebook. Background Art

[0002] 3D dynamic scene reconstruction is an important task in the field of vision and graphics. It can model changing 3D scenes and render 2D images from arbitrary perspectives at different times. Given a series of images taken by a monocular camera at different times and from different perspectives, it can model the entire 3D dynamic scene and generate high-quality images from any novel perspective. Because the scene changes over time, using only static 3D scene reconstruction methods often produces poor results. To model 3D dynamic scenes, existing methods have explored a variety of methods for accurately modeling 3D dynamic scenes. However, they suffer from problems such as long training time and large storage space.

[0003] Neural Radiance Fields (NeRF) proposes to use neural networks to model 3D scenes. Due to its excellent performance in reconstructing geometry and appearance, it has become an excellent scene representation. Due to its reliability, there are currently many methods that use it to reconstruct 3D dynamic scenes. D-NeRF (D-NeRF: Neural Radiance Fields for Dynamic Scenes) combines NeRF with deformation fields and uses input views to jointly optimize the two, effectively improving the reconstruction quality of 3D dynamic scenes. Hexplane (HexPlane: A Fast Representation for Dynamic Scenes) uses 6 feature planes and a small decoder to model 3D dynamic scenes, further improving the quality of scene rendered images. However, despite valuable efforts, these methods still have much room for improvement in terms of training time, rendering time, storage space, etc.

[0004] The recent introduction of 3D Gaussian Splatting (3DGS) has provided a new approach to 3D dynamic scene reconstruction. 4DGS (4D Gaussian Splatting) first introduced 3DGS to the task, achieving results that surpass previous work. However, because 3DGS is an explicit representation that requires storing a large number of parameters when constructing the scene, 4DGS struggles to achieve satisfactory results in terms of storage space. Summary of the Invention

[0005] Therefore, it is necessary to provide a three-dimensional dynamic scene compression method based on a multi-resolution codebook in view of the above technical problems.

[0006] The three-dimensional dynamic scene compression method based on a multi-resolution codebook comprises the following steps:

[0007] The input views of the same dynamic scene at different time instants and different viewpoints are used to reconstruct a three-dimensional dynamic scene based on 3D Gaussian points, each of which has a Gaussian point attribute.

[0008] The Gaussian point attribute is optimized according to the input views and a rendered image of the three-dimensional scene at the same time instant and the same viewpoint.

[0009] The dynamic change of the scene over time is realized based on a 4D Gaussian deformation field, which comprises the following steps: taking the Gaussian point position and the dynamic scene time as inputs, obtaining a Gaussian point attribute change amount through the deformation field, and adding the Gaussian point attribute change amount to the Gaussian point attribute to obtain a three-dimensional scene at different time instants.

[0010] The deformation field feature plane of the deformation field is compressed, which comprises the following steps: providing a plurality of feature codebooks according to the resolution number of the deformation field feature plane, the feature codebooks being used to optimize the deformation field feature plane synchronously, and the feature codebook values of the feature codebooks being used to map and replace feature values in the deformation field feature plane at the corresponding resolution in a one-to-many manner.

[0011] Optionally, the three-dimensional dynamic scene compression method comprises deleting Gaussian points, specifically comprising the following steps:

[0012] The importance value score of each 3D Gaussian point is obtained, which comprises the following steps: for any 3D Gaussian point, taking the sum of the volume rendering weights of all pixels in the rendered image at a single viewpoint as a candidate value, and taking the maximum value of the candidate values of all 3D Gaussian points at all viewpoints as the importance value score of the 3D Gaussian point.

[0013] According to the order from large to small of the importance value scores, some 3D Gaussian points with low importance value scores are deleted.

[0014] Optionally, the three-dimensional dynamic scene compression method comprises compressing the Gaussian point attribute, specifically comprising the following steps:

[0015] A plurality of attribute codebooks corresponding to different Gaussian point attributes are provided, and the attribute codebook values in the attribute codebooks are used to map and replace the Gaussian point attribute values of different 3D Gaussian points in a one-to-many manner.

[0016] The optimization of the 3D Gaussian comprises a first stage and a second stage executed in sequence:

[0017] In the first stage, the Gaussian point attributes are optimized based on the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle;

[0018] In the second stage, based on the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle, the Gaussian point attributes and the attribute codebook are optimized simultaneously to obtain a mapping table of a one-to-many relationship between the attribute codebook values ​​and the corresponding Gaussian point attribute values ​​that are mapped and replaced. During the rendering process, the attribute codebook values ​​are used to replace the corresponding Gaussian point attribute values ​​in a one-to-many manner according to the mapping table.

[0019] Optionally, the Gaussian point attributes include scaling attributes, rotation attributes, and color attributes, and the attribute codebook correspondingly includes a scaling codebook, a rotation codebook, and a color codebook.

[0020] Optionally, optimizing the Gaussian point attribute, and simultaneously optimizing the Gaussian point attribute and the attribute codebook, are both performed using the following loss function:

[0021] L gs =(1-λ)L1+λL D-SSIM

[0022] Where, L gs represents the loss function;

[0023] L1 represents the average of the absolute differences between the input view and the rendered image of the 3D scene at the same viewing angle;

[0024] L D-SSIM , represents the structural similarity loss between the rendered image and the input image.

[0025] Optionally, the Gaussian point position and the dynamic scene time are both used as inputs, and the Gaussian point attribute change is obtained through the deformation field, specifically including:

[0026] The Gaussian point position and the dynamic scene time are both used as inputs, and a feature vector is obtained from the deformation field feature plane in the deformation field; the feature vector is input into multiple multi-layer perceptrons to obtain the Gaussian point attribute change.

[0027] Optionally, compressing the deformation field characteristic plane of the deformation field is performed on some eigenvalues ​​in the deformation field characteristic plane, and specifically includes:

[0028] providing a characteristic importance score plane having the same shape as the deformation field characteristic plane of the deformation field, wherein the characteristic importance score plane is used to be optimized along with the deformation field characteristic plane and to obtain a characteristic value score;

[0029] According to the order of the eigenvalue scores, some eigenvalues ​​in the deformation field feature plane are mapped to the feature codebook values ​​in a many-to-one manner.

[0030] Optionally, the three-dimensional dynamic scene compression method further includes:

[0031] Obtaining a mapping relationship between the characteristic codebook value and the corresponding characteristic value in the deformation field characteristic plane;

[0032] Providing a difference codebook, the difference codebook being used to store quantization parameter differences between: original values ​​of corresponding eigenvalues ​​to be replaced in the deformation field feature plane, and feature codebook values ​​used to replace corresponding eigenvalues ​​in the deformation field feature plane;

[0033] When the corresponding eigenvalues ​​in the deformation field feature plane are replaced by the eigencodebook values, the quantization parameter difference in the difference codebook is retrieved to reduce the difference caused by the replacement process.

[0034] The present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook described in the present application.

[0035] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook described in the present application.

[0036] The three-dimensional dynamic scene compression method based on the multi-resolution codebook of this application has at least the following effects:

[0037] This application achieves compression of deformation field parameters by performing a one-to-many mapping and replacement of feature codebook values ​​with corresponding feature values ​​in the deformation field feature plane, effectively reducing the storage space of dynamic scenes. The feature codebook and deformation field feature plane are optimized synchronously, ensuring the quality of the generated scene rendered image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of a flow chart of a three-dimensional dynamic scene compression method based on a multi-resolution codebook in one embodiment of the present application;

[0039] Figure 2 This is a model architecture diagram of step S200 in the three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0040] Figure 3 This is a model architecture diagram of step S300 in the three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0041] Figure 4 This is a model architecture diagram of step S400 in the three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0042] Figure 5 This is a model architecture diagram of step 500 in a three-dimensional dynamic scene compression method based on a multi-resolution codebook in an embodiment of the present application;

[0043] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] See also Figure 1 An embodiment of the present application provides a three-dimensional dynamic scene compression method based on a multi-resolution codebook, comprising steps S100 to S400, wherein:

[0046] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0047] Step S200 : optimizing Gaussian point attributes based on the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0048] Step S300, based on the 4D Gaussian deformation field, realizes the dynamic change of the scene over time, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, and superimposing the Gaussian point attribute change to the Gaussian point attribute to obtain the three-dimensional scene at different times.

[0049] Step S400 compresses the deformation field feature plane of the deformation field, including providing multiple feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks being optimized synchronously with the deformation field feature plane, and feature codebook values ​​in the feature codebooks being used for one-to-many mapping and replacement with feature values ​​in the deformation field feature plane at corresponding resolutions.

[0050] See also Figures 1-3 In one embodiment, the present application provides a three-dimensional dynamic scene compression method based on a multi-resolution codebook, which adds optional sub-steps and subsequent steps based on the previous embodiment. It includes steps S100 to S500. Among them:

[0051] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0052] Step S200 : optimizing Gaussian point attributes based on the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0053] This embodiment primarily utilizes a 3DGS-based 3D explicit representation model. Specifically, the 3D scene reconstructed from the input view uses 3D Gaussian points in space for precise geometric modeling and visual rendering. For the implementation of reconstructing the 3D dynamic scene in step S100 using input views at different times and perspectives, see the related prior art document on 4D Gaussian splatting, "CVPR 2024 4DGaussian Splatting for Real-Time Dynamic Scene Rendering."

[0054] 3DGS projects Gaussian points in space onto a 2D plane based on the viewing angle and obtains the final image through volume rendering. However, due to the need to optimize multiple attributes of each Gaussian point, the final reconstructed scene consumes a lot of storage space.

[0055] See also Figure 2 In step S210, the three-dimensional dynamic scene compression method includes deleting Gaussian points. As an optional sub-step, step S210 specifically includes steps S211 to S212:

[0056] Step S211, obtaining an importance score for each 3D Gaussian point, including: for any 3D Gaussian point, taking the sum of the volume rendering weights of all pixels in the rendered image at a single view as a candidate value, and taking the maximum value of the candidate values ​​of each 3D Gaussian point at all view angles as the importance score of the 3D Gaussian point;

[0057] In step S212 , according to the order of importance value scores from large to small, some 3D Gaussian points (referred to as Gaussian points for short) with the importance value scores being ranked low are deleted.

[0058] Step S210 is a spatial Gaussian point compression method based on pruning, that is, discarding unimportant Gaussian points in the three-dimensional space, thereby reducing the storage space occupied by Gaussian points in the scene.

[0059] Step S211 prunes the Gaussian points in space by calculating the importance score of each Gaussian point. This method effectively reduces the number of Gaussian points in space while ensuring image rendering quality, and compactly represents the various properties of Gaussian. At a single viewpoint, the sum of the volume rendering weights of the Gaussian points for all pixels is used as a candidate value for the importance score of the Gaussian point, and the maximum value among the candidate values ​​at all viewpoints is selected as the importance score of the Gaussian point. Compared to using the method of accumulating weights at all viewpoints as the importance score, this method better preserves the details in the scene.

[0060] Specifically, the process of rendering an image is described by the following formula:

[0061]

[0062] Where:

[0063] r, represents the ray under the rendering perspective;

[0064] Indicates the ray color at the rendering perspective;

[0065] Indicates the color of the Gaussian point;

[0066] T i (1-exp(-σ i δ i ), represents the weight in the volume rendering process, referred to as volume rendering weight;

[0067]

[0068] σ i , represents the opacity corresponding to the Gaussian point in space;

[0069] δ i , represents the distance between adjacent sampling points in the ray direction, δ i =t i+1 -t i , t i is the sampling point, which is used to represent the sampling of Gaussian points.

[0070] According to equation (1), the importance of each Gaussian point can be intuitively expressed using the weight in the volume rendering process. Step S211 uses the maximum value of the sum of the volume rendering weights of the Gaussian point under all viewing angles as the importance value score of the Gaussian point:

[0071]

[0072] Where:

[0073] Score(G iG i , represents the importance score of the i-th Gaussian point G i

[0074] I, represents all the rendering views;

[0075] j, represents one of the rendering views I;

[0076] r, represents a ray at the rendering view I.

[0077] After obtaining the importance score of the Gaussian points, step S212 determines the Gaussian points that can be removed and performs pruning operation by using a self-defined threshold, which can be a percentage threshold. See the following formula:

[0078] m i = Score(G i ) < τ prune (3)

[0079] m i , represents a binary mask based on the threshold, which is used to select the Gaussian points that need to be pruned (i.e. deleted);

[0080] τ prune , represents a percentage threshold.

[0081] Thus, step S212 is completed. Step S220 uses the weight and the maximum value at a single view as the importance score of the Gaussian points, which can better maintain the details in the scene. After performing pruning, the number of Gaussian points in the space is significantly reduced, which reduces the final storage space.

[0082] Referring to Figure 2 , step S220, the three-dimensional dynamic scene compression method includes compressing the Gaussian point attributes. As a sub-step of the optional implementation, step S220 specifically includes steps S221-S223:

[0083] Step S221, provides a plurality of attribute codebooks (i.e. the codebooks shown in Figure 2 ) corresponding to different Gaussian point attributes, and the attribute codebook values in the attribute codebooks are used for one-to-many mapping and replacement with the Gaussian point attribute values of different 3D Gaussian points;

[0084] Step S222, optimizing the 3D Gaussian includes a first stage and a second stage performed in sequence: in the first stage, the Gaussian point attributes are optimized according to the input view and the rendering image of the three-dimensional scene at the same time and the same view;

[0085] Step S223, in the second stage, based on the input view and the rendered image of the three-dimensional scene at the same time and the same perspective, the Gaussian point attributes and the attribute codebook are optimized simultaneously to obtain a mapping table of the attribute codebook values ​​and the corresponding Gaussian point attribute values ​​to be mapped and replaced, and a one-to-many relationship between the two. During the rendering process, the attribute codebook values ​​are used to replace the corresponding Gaussian point attribute values ​​in a one-to-many manner according to the mapping table.

[0086] Step S220 compresses the attributes of the Gaussian points based on vector quantization. The Gaussian point attributes include scaling attributes, rotation attributes, and color attributes (sh coefficients). Correspondingly, the attribute codebooks provided in step S221 include scaling codebooks, rotation codebooks, and color codebooks.

[0087] The codebook size provided in step S221 can be set as needed, and the attribute codebook can be understood as a pre-provided multidimensional vector. The first-stage optimization in step S222 is based on the technical idea of ​​3D Gaussian optimization in the prior art. In step S223, based on the first stage, the Gaussian point attributes and the attribute codebook are optimized simultaneously. During the optimization process, the K-means clustering algorithm is used to cluster the attribute vectors into K clusters, and the attribute codebook value corresponding to the cluster is set according to the important value score of the Gaussian point in the cluster, thereby realizing the mapping relationship between the Gaussian point attributes and the attribute codebook value. By setting the appropriate codebook size, the relationship between the compression ratio and the rendering quality can be effectively balanced.

[0088] Since a one-to-many mapping table between attribute codebook values ​​and corresponding Gaussian point attribute values ​​has been obtained in step S222, taking color attributes as an example, the color codebook in the attribute codebook uses a vector quantization method to compress color information, further reducing storage space while ensuring generation quality.

[0089] In step S223, during the rendering process, the original Gaussian point attributes (such as color attribute values) are replaced using a mapping table according to the attribute codebook, and the codebook and other Gaussian attributes are optimized according to the loss function used in 3DGS. The following loss function is used to optimize Gaussian point attributes and optimize Gaussian point attributes and attribute codebook simultaneously:

[0090] L gs =(1-λ)L1+λL D-SSIM

[0091] Where, L gs represents the loss function;

[0092] L1 represents the average of the absolute differences between the input view and the rendered image of the 3D scene at the same viewing angle;

[0093] L D-SSIM, represents the structural similarity loss between the rendered image and the input image.

[0094] Step S300, realizing dynamic changes of the scene over time based on the 4D Gaussian deformation field, includes:

[0095] The Gaussian point position and dynamic scene time are both used as input, and the feature vector is obtained from the deformation field feature plane in the deformation field; the feature vector is input into multiple multi-layer perceptrons ( Figure 3 MLP in) to obtain the Gaussian point attribute change ( Figure 3 The ΔV in the Gaussian point is used to add the Gaussian point attribute changes to obtain the three-dimensional scenes at different times.

[0096] For the specific implementation of step S300 , reference may be made to the related literature on 4D Gaussian in the prior art.

[0097] See also Figure 4 and Figure 5 Step S400 compresses the deformation field feature plane of the deformation field, including: providing a plurality of feature codebooks (ie, Figure 5 The codebook shown in , where codebook 1 and codebook 2 correspond to different resolutions), the feature codebook is used to optimize synchronously with the deformation field feature plane, and the feature codebook value of the feature codebook is used to perform a one-to-many mapping and replacement with the feature value in the deformation field feature plane at the corresponding resolution.

[0098] In practical applications, the 4DGS (4D Gaussian) deformation field often occupies a large amount of storage space. Simply compressing the Gaussian points is not enough to effectively reduce the final storage requirements. Step S400 proposes a deformation field compression method based on a multi-resolution codebook, which effectively reduces the storage space of the deformation field.

[0099] Specifically, the number of feature codebooks is first determined based on the number of feature plane resolutions in the 4DGS deformation field. Then, vector quantization is performed using feature codebooks of different sizes for feature planes of different deformation field resolutions. Vector quantization involves a many-to-one mapping of eigenvalues ​​in the deformation field feature planes to the feature codebook values.

[0100] Furthermore, in step S400, the deformation field feature plane is compressed, and the compression is performed on some eigenvalues ​​in the deformation field feature plane. Specifically, the compression includes: providing a feature importance value score plane with the same shape as the deformation field feature plane, the feature importance value score plane is used to accompany the deformation field feature plane optimization and to obtain eigenvalue scores; according to the sorting of the eigenvalue scores, some eigenvalues ​​in the deformation field feature plane are mapped to the feature codebook values ​​accordingly in a many-to-one manner.

[0101] This process does not directly calculate the importance value scores of the features in the feature plane, but obtains the eigenvalue scores corresponding to each feature by optimizing the importance value score plane with the same shape as the feature plane, thereby guiding the subsequent quantization process.

[0102] In order to map each eigenvalue to the correct codebook value (feature codebook value), we randomly select a batch of eigenvalues ​​at each iteration and calculate the Euclidean distance between the selected eigenvalue and each codebook value in the codebook to determine which codebook value the eigenvalue is associated with, and modify the codebook value through the eigenvalue importance score.

[0103] Similar to the concept of Gaussian point pruning, after the eigenvalue scores are sorted, they are divided into three parts: the first part, which contains the higher eigenvalue scores, is retained, i.e., step S400 is not performed on the first part. The second part, which contains the lower eigenvalue scores, is set to a constant or zero. The third part, which contains the intermediate eigenvalue scores, is vector quantized, i.e., step S400 is performed on the third part.

[0104] In the third part, in order to ensure the final generation quality, the loss L is introduced vq To reduce the difference between the characteristic codebook value (codebook vector) and the characteristic value (actual vector) in the deformation field characteristic plane.

[0105]

[0106] Among them, N represents the number of eigenvalues ​​involved in each calculation, f i represents the eigenvalue in the characteristic plane of the deformation field, r i Represents the signature code value.

[0107] Step S500, using the difference codebook to narrow the gap between the feature codebook value and the actual feature plane feature value. As an optional subsequent step, step S500 specifically includes steps S510 to S530:

[0108] Step S510, obtaining a mapping relationship between a feature codebook value and a corresponding feature value in a deformation field feature plane;

[0109] Step S520: Provide a difference codebook ( Figure 5 The codebook in '), the difference codebook is used to store the quantization parameter difference between the following two: the original value of the corresponding eigenvalue replaced in the deformation field feature plane, and the feature codebook value used to replace the corresponding eigenvalue in the deformation field feature plane;

[0110] In step S530, when the corresponding eigenvalues ​​in the deformation field feature plane are replaced by the feature codebook values, the quantization parameter difference in the difference codebook is retrieved to reduce the difference caused by the replacement process. The mapping relationship of step S510 is the same as "part of the eigenvalues ​​in the deformation field feature plane is mapped one-to-many to the feature codebook values" in step S400. In step S520, a fixed-size difference codebook is additionally designed to store the difference between the quantized parameters and the original parameters. The difference codebook can be accessed by feature codebooks of all resolutions, thereby reducing the difference between the quantized value (feature codebook value) and the actual value (feature value in the deformation field feature plane) to ensure the quality of the scene rendering image.

[0111] The multi-resolution codebook-based three-dimensional dynamic scene compression method provided in each embodiment of the present application can compress 3D Gaussian sputtering dynamic scenes and perform lightweight representation of three-dimensional dynamic scenes, while maintaining the rendering quality of the three-dimensional dynamic scene during compression. Steps S210 and S220 perform 3D Gaussian compression, making the scene representation more compact through methods such as pruning and codebook quantization. Step S400 performs deformation field compression, compressing the deformation field parameters through a multi-resolution codebook, effectively reducing the storage space of the dynamic scene.

[0112] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a three-dimensional dynamic scene compression method based on a multi-resolution codebook is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0114] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0115] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0116] Step S200 : optimizing Gaussian point attributes based on the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0117] Step S300, based on the 4D Gaussian deformation field, realizes the dynamic change of the scene over time, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, and superimposing the Gaussian point attribute change to the Gaussian point attribute to obtain the three-dimensional scene at different times.

[0118] Step S400 compresses the deformation field feature plane of the deformation field, including providing multiple feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks being optimized synchronously with the deformation field feature plane, and feature codebook values ​​in the feature codebooks being used for one-to-many mapping and replacement with feature values ​​in the deformation field feature plane at corresponding resolutions.

[0119] In one embodiment, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the following steps:

[0120] Step S100 , using input views of the same dynamic scene at different times and different perspectives, reconstruct a three-dimensional dynamic scene based on 3D Gaussian points in space, where each 3D Gaussian point has Gaussian point attributes.

[0121] Step S200 : optimizing Gaussian point attributes based on the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle.

[0122] Step S300, based on the 4D Gaussian deformation field, realizes the dynamic change of the scene over time, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, and superimposing the Gaussian point attribute change to the Gaussian point attribute to obtain the three-dimensional scene at different times.

[0123] Step S400 compresses the deformation field feature plane of the deformation field, including providing multiple feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks being optimized synchronously with the deformation field feature planes, and feature codebook values ​​in the feature codebooks being used to perform a one-to-many mapping and replacement with feature values ​​in the deformation field feature planes at corresponding resolutions. In this embodiment, the computer program product includes a program code portion for executing the steps of the multi-resolution codebook-based three-dimensional dynamic scene compression method in various embodiments of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet, and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality may be provided for download via a hardware description language.

[0124] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0125] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification. When technical features in different embodiments are reflected in the same figure, it can be regarded as that figure also discloses the combination examples of the various embodiments involved.

[0126] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A three-dimensional dynamic scene compression method based on a multi-resolution codebook, characterized in that: include: Reconstructing a three-dimensional dynamic scene based on 3D Gaussian points in space using input views of the same dynamic scene at different times and different perspectives, where each 3D Gaussian point has Gaussian point properties; Optimizing the Gaussian point attributes based on the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle; A 4D Gaussian deformation field is used to achieve dynamic changes in the scene over time, including: taking the Gaussian point position and the dynamic scene time as input, obtaining the Gaussian point attribute change through the deformation field, and superimposing the Gaussian point attribute change on the Gaussian point attribute to obtain the three-dimensional scene at different times; Compressing the deformation field feature plane of the deformation field includes: providing a plurality of feature codebooks corresponding to the number of resolutions of the deformation field feature plane, the feature codebooks being used for synchronous optimization with the deformation field feature plane, and feature codebook values ​​of the feature codebooks being used for one-to-many mapping and replacement with feature values ​​in the deformation field feature plane at corresponding resolutions.

2. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The three-dimensional dynamic scene compression method includes deleting Gaussian points, specifically including: Obtaining an importance score for each of the 3D Gaussian points, including: for any 3D Gaussian point, taking a sum of volume rendering weights of all pixels in a rendered image at a single viewing angle as a candidate value, and taking a maximum value of the candidate values ​​of the 3D Gaussian points at all viewing angles as the importance score of the 3D Gaussian point; According to the order of the importance value scores from large to small, some 3D Gaussian points with the importance value scores ranked later are deleted.

3. The three-dimensional dynamic scene compression method according to claim 1, wherein: The three-dimensional dynamic scene compression method includes compressing the Gaussian point attributes, specifically including: Providing multiple attribute codebooks corresponding to different Gaussian point attributes, wherein attribute codebook values ​​in the attribute codebooks are used for one-to-many mapping and replacement with Gaussian point attribute values ​​of different 3D Gaussian points; Optimizing the 3D Gaussian consists of a first and a second stage performed sequentially: In the first stage, the Gaussian point attributes are optimized based on the input view and a rendered image of the three-dimensional scene at the same time and the same viewing angle; In the second stage, based on the input view and the rendered image of the three-dimensional scene at the same time and the same viewing angle, the Gaussian point attributes and the attribute codebook are optimized simultaneously to obtain a mapping table of a one-to-many relationship between the attribute codebook values ​​and the corresponding Gaussian point attribute values ​​that are mapped and replaced. During the rendering process, the attribute codebook values ​​are used to replace the corresponding Gaussian point attribute values ​​in a one-to-many manner according to the mapping table.

4. The three-dimensional dynamic scene compression method according to claim 3, characterized in that: The Gaussian point attributes include scaling attributes, rotation attributes, and color attributes, and the attribute codebook correspondingly includes a scaling codebook, a rotation codebook, and a color codebook.

5. The three-dimensional dynamic scene compression method according to claim 3, wherein: Optimizing the Gaussian point attributes, and simultaneously optimizing the Gaussian point attributes and the attribute codebook, are both performed using the following loss function: L gs =(1-λ)L1+λL D-SSIM Where, L gs represents the loss function; L1 represents the average of the absolute differences between the input view and the rendered image of the 3D scene at the same viewing angle; L D-SSIM , represents the structural similarity loss between the rendered image and the input image.

6. The three-dimensional dynamic scene compression method according to claim 1, wherein: The Gaussian point position and dynamic scene time are both used as inputs, and the Gaussian point attribute change is obtained through the deformation field, specifically including: The Gaussian point position and the dynamic scene time are both used as inputs, and a feature vector is obtained from the deformation field feature plane in the deformation field; the feature vector is input into multiple multi-layer perceptrons to obtain the Gaussian point attribute change.

7. The three-dimensional dynamic scene compression method according to claim 1, characterized in that: The compressing of the deformation field characteristic plane of the deformation field is performed on some characteristic values ​​in the deformation field characteristic plane, and specifically includes: providing a characteristic importance score plane having the same shape as the deformation field characteristic plane of the deformation field, wherein the characteristic importance score plane is used to be optimized along with the deformation field characteristic plane and to obtain a characteristic value score; According to the order of the eigenvalue scores, some eigenvalues ​​in the deformation field feature plane are mapped to the feature codebook values ​​in a many-to-one manner.

8. The three-dimensional dynamic scene compression method according to claim 1, wherein: The three-dimensional dynamic scene compression method further includes: Obtaining a mapping relationship between the characteristic codebook value and the corresponding characteristic value in the deformation field characteristic plane; Providing a difference codebook, the difference codebook being used to store quantization parameter differences between: original values ​​of corresponding eigenvalues ​​to be replaced in the deformation field feature plane, and feature codebook values ​​used to replace corresponding eigenvalues ​​in the deformation field feature plane; When the corresponding eigenvalues ​​in the deformation field feature plane are replaced by the eigencodebook values, the quantization parameter difference in the difference codebook is retrieved to reduce the difference caused by the replacement process.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that The processor executes the computer program to implement the steps of the three-dimensional dynamic scene compression method based on multi-resolution codebook according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the three-dimensional dynamic scene compression method based on a multi-resolution codebook according to any one of claims 1 to 8 are implemented.

Images