An end-to-end joint optimization method for dynamic neural radiated field representation and compression

Abstract

The present disclosure provides an end-to-end joint optimization method for dynamic neural radiance field representation and compression, which comprises: representing a dynamic neural radiance field by using a coefficient feature grid and a base feature grid, compensating an error area of the dynamic neural radiance field by using a residual feature grid and the coefficient feature grid, and determining a represented dynamic neural radiance field; performing model training processing and analog quantization operation processing on the represented dynamic neural radiance field; performing optimization processing on the dynamic neural radiance field subjected to the analog quantization operation processing according to a predicted compression data amount of the represented dynamic neural radiance field and a preset loss function, and determining a trained dynamic neural radiance field; performing uniform quantization processing and encoding processing on the trained dynamic neural radiance field, and determining an actual compression data amount of the dynamic neural radiance field. Through the present disclosure, the dynamic neural radiance field is efficiently modeled and compressed, the compression rate of the dynamic neural radiance field is improved, and the distortion degree after restoration is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to an end-to-end joint optimization method for dynamic neural radiation field representation and compression. Background Technology

[0002] Realistic volumetric video provides immersive experiences for virtual reality and remote sensing. The outstanding performance of neural radiation fields in realistic static scenes has inspired research into volumetric video. Today, dynamic neural radiation fields show great potential in representing realistic volumetric video. However, current techniques still have many limitations. For example, dynamic neural radiation field methods can only represent limited segments of dynamic scenes, and storing and transmitting volumetric video using neural radiation fields remains challenging for video sequences involving arbitrary motion or long time series.

[0003] Currently, it is difficult to find an efficient method for representing and compressing neural radiation fields to generate and store long sequences of dynamic neural radiation fields. Directly extending static neural radiation field methods to dynamic scenes is impractical, as it ignores the spatiotemporal continuity of the scene, leading to an excessive number of network parameters. Some methods attempt to recreate features in each frame by transforming features back to the canonical space. However, relying solely on the canonical space limits the effectiveness for sequences with significant motion or topological changes. Furthermore, extending neural radiation fields to the 4D spatiotemporal domain faces challenges related to suboptimal rendering quality and large model storage, especially in streaming scenarios. Additionally, traditional image encoders are not suitable for the high-dimensional feature domain where neural radiation field features reside, thus necessitating deep learning-based methods for compression. Currently, there is no method for end-to-end joint optimization of dynamic neural radiation field representation and compression, resulting in the loss of dynamic details and reduced compression efficiency.

[0004] Therefore, those skilled in the art are dedicated to developing a method for end-to-end joint optimization of dynamic neural radiation field representation and compression to improve the reconstruction quality and compression rate of dynamic neural radiation fields and significantly improve rate distortion performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this disclosure is to provide an end-to-end joint optimization method for dynamic neural radiation field representation and compression.

[0006] To achieve the above objectives, according to one aspect of this disclosure, an end-to-end joint optimization method for dynamic neural radiation field representation and compression is provided, comprising:

[0007] The dynamic neural radiation field is represented by a coefficient feature grid and a basic feature grid. The error region of the dynamic neural radiation field is compensated by the residual feature grid and the coefficient feature grid. The represented dynamic neural radiation field is then determined.

[0008] The represented dynamic neural radiation field is subjected to model training and simulation quantization to determine the predicted compressed data volume of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization.

[0009] The dynamic neural radiation field, which has undergone simulation quantization, is optimized according to a preset loss function to determine the trained dynamic neural radiation field.

[0010] The trained dynamic neural radiation field is subjected to uniform quantization and encoding to determine the actual compressed data volume of the dynamic neural radiation field.

[0011] Optionally, the step of representing the dynamic neural radiation field using a coefficient feature grid and a basic feature grid, and compensating for the error region of the dynamic neural radiation field using a residual feature grid and the coefficient feature grid, to determine the represented dynamic neural radiation field, includes:

[0012] The dynamic neural radiation field is represented using a feature voxel grid;

[0013] The dynamic neural radiation field represented by the aforementioned feature voxel grid is subjected to basis decomposition to determine the coefficient feature grid and the basic feature grid.

[0014] The reconstructed dynamic neural radiation field is determined based on the coefficient feature grid and the basic feature grid.

[0015] Optionally, determining the reconstructed dynamic neural radiation field based on the coefficient feature grid and the basic feature grid includes:

[0016] Based on the position of the light rays, interpolation processing is performed on the coefficient feature grid and the basic feature grid to determine the interpolated coefficient feature grid and the interpolated basic feature grid;

[0017] The interpolated coefficient feature grid and the interpolated basic feature grid are subjected to a Hadman product to determine the Hadman product of the coefficient feature grid and the basic feature grid.

[0018] The Hadman product of the coefficient feature grid and the basic feature grid is input into a multilayer perceptron to determine the reconstructed dynamic neural radiation field.

[0019] Optionally, the step of representing the dynamic neural radiation field using a coefficient feature grid and a basic feature grid, compensating for the error region of the dynamic neural radiation field using a residual feature grid and the coefficient feature grid, and determining the represented dynamic neural radiation field further includes:

[0020] The reconstructed dynamic neural radiation field is represented by a continuous set of feature grids;

[0021] The initial frame of each of the feature grid groups is represented using the long reference feature grid of the key frame, and each remaining frame of each of the feature grid groups is represented using the residual feature grid and the coefficient feature grid, thereby determining the represented dynamic neural radiation field.

[0022] Optionally, the step of performing model training and simulation quantization on the represented dynamic neural radiation field to determine the predicted compressed data volume of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization includes:

[0023] The represented dynamic neural radiation field is subjected to model training processing, and a preset entropy model is used to perform data distribution prediction processing on the represented dynamic neural radiation field to determine the amount of predicted compressed data of the represented dynamic neural radiation field.

[0024] When training the model on the represented dynamic neural radiation field, uniform noise is used to perform a simulation quantization operation on the represented dynamic neural radiation field to determine the dynamic neural radiation field after the simulation quantization operation.

[0025] Optionally, the preset loss function includes a distortion loss function, the amount of predicted compressed data, and the L1 norm of the residual feature grid.

[0026] Optionally, the step of uniformly quantizing and encoding the trained dynamic neural radiation field to determine the actual compressed data volume of the dynamic neural radiation field includes:

[0027] The feature grid of the trained dynamic neural radiation field is uniformly quantized to determine the uniformly quantized dynamic neural radiation field.

[0028] The dynamic neural radiation field, after uniform quantization, is interval encoded to determine the actual compressed data volume of the dynamic neural radiation field.

[0029] According to a second aspect of this disclosure, an end-to-end joint optimization system for dynamic neural radiation field representation and compression is provided, comprising:

[0030] The representation module is used to represent the dynamic neural radiation field using a coefficient feature grid and a basic feature grid, to compensate for the error region of the dynamic neural radiation field using a residual feature grid and the coefficient feature grid, and to determine the represented dynamic neural radiation field.

[0031] The model training module is used to perform model training and simulation quantization processing on the represented dynamic neural radiation field to determine the amount of predicted compressed data of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization.

[0032] An optimization module is used to optimize the dynamic neural radiation field that has undergone simulation quantization processing according to a preset loss function, and to determine the dynamic neural radiation field after training.

[0033] The compression module is used to perform uniform quantization and encoding processing on the trained dynamic neural radiation field to determine the actual compressed data volume of the dynamic neural radiation field.

[0034] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method provided in the first aspect of this disclosure.

[0035] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0036] A memory on which computer programs are stored;

[0037] A processor for executing the computer program in the memory to implement the steps of the method provided in the first aspect of this disclosure.

[0038] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects:

[0039] The above technical solution uses coefficient feature grids and basic feature grids to represent the dynamic neural radiation field, and uses residual feature grids and coefficient feature grids to compensate for the error region of the dynamic neural radiation field, thus determining the represented dynamic neural radiation field. Then, the represented dynamic neural radiation field is subjected to model training and encoding processing, thereby optimizing the representation and compression of the dynamic neural radiation field, improving the reconstruction quality and compression rate of the dynamic neural radiation field. The representation and compression process of the dynamic neural radiation field is jointly optimized end-to-end, and differentiable analog quantization operations are used to promote the end-to-end training of the dynamic neural radiation field, thus achieving efficient modeling and compression of the dynamic neural radiation field.

[0040] In embodiments of this disclosure, a preset entropy model is also used to predict the amount of compressed data for dynamic neural radiation fields, thereby facilitating end-to-end training of dynamic neural radiation fields. Attached Figure Description

[0041] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0042] Figure 1 This is a flowchart illustrating an end-to-end joint optimization method for dynamic neural radiation field representation and compression according to an exemplary embodiment.

[0043] Figure 2 This is a block diagram illustrating an end-to-end joint optimization system for dynamic neural radiation field representation and compression, according to an exemplary embodiment. Detailed Implementation

[0044] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all fall within the protection scope of the present disclosure.

[0045] Figure 1 This is a flowchart illustrating an end-to-end joint optimization method for dynamic neural radiation field representation and compression according to an exemplary embodiment.

[0046] like Figure 1 As shown, this disclosure provides an end-to-end joint optimization method for dynamic neural radiation field representation and compression, including S11 to S14.

[0047] S11 uses coefficient feature grids and basic feature grids to represent the dynamic neural radiation field, and uses residual feature grids and coefficient feature grids to compensate for the error region of the dynamic neural radiation field, thus determining the represented dynamic neural radiation field.

[0048] Among them, the coefficient feature grid is used to represent spatial variation, and the basic feature grid is used to represent the commonalities of the signal.

[0049] The error region of the dynamic neural radiation field includes the error compensation region between different frames and the new observation region.

[0050] The residual feature mesh is used to represent the error compensation region between the base mesh of the keyframe and the base mesh of the current frame, as well as the region of new observation.

[0051] In one possible embodiment, S11 may include S21 to S23.

[0052] S21 uses a characteristic voxel grid to represent the dynamic neural radiation field.

[0053] S22, perform basis decomposition on the dynamic neural radiation field represented by the feature voxel grid to determine the coefficient feature grid and the basic feature grid.

[0054] For example, a complete dynamic neural radiation field can be decomposed into a coefficient feature grid C and a basic feature grid B through basis decomposition.

[0055] S23, determine the reconstructed dynamic neural radiation field based on the coefficient feature grid and the basic feature grid.

[0056] In one possible embodiment, S23 may include S31 to S33.

[0057] S31, based on the position of the light rays, interpolate the coefficient feature grid and the basic feature grid to determine the interpolated coefficient feature grid and the interpolated basic feature grid.

[0058] S32, perform a Haldman product on the interpolated coefficient feature grid and the interpolated basic feature grid to determine the Haldman product of the coefficient feature grid and the basic feature grid.

[0059] S33, input the Hadman product of the coefficient feature grid and the basic feature grid into the multilayer perceptron to determine the reconstructed dynamic neural radiation field.

[0060] From steps S31 to S32, the reconstructed dynamic neural radiation field can be represented as follows:

[0061]

[0062] Where Φ represents a multilayer perceptron, and interp(·) represents an interpolation function. Let represent the Haldman product, C represent the coefficient feature grid, B represent the basic feature grid, x represent the three-dimensional position coordinates in space, d represent the ray, c represent the color of the corresponding point, and σ represent the density of the corresponding point.

[0063] Here, ray d includes the starting point and direction of the ray.

[0064] The multilayer perceptron Φ used in this disclosure is a small multilayer perceptron (MLP).

[0065] The reconstructed dynamic neural radiation field of this disclosure can be used to acquire color and depth images.

[0066] In one possible embodiment, S11 may also include S24 to S25.

[0067] S24 uses a continuous set of feature grids to represent the reconstructed dynamic neural radiation field.

[0068] In this approach, a long sequence of dynamic neural radiation fields that change over time is represented as multiple consecutive sets of feature grids.

[0069] S25, the initial frame of each feature grid group is represented by the long reference feature grid of the key frame, and each remaining frame of each feature grid group is represented by the residual feature grid and the coefficient feature grid, thus determining the represented dynamic neural radiation field.

[0070] Following the example above, the initial frame of each feature grid group is represented by a long reference feature network of the key frame. Based on the basic commonalities and short-term similarities of features during signal extraction, each remaining frame of each feature grid group, except for the initial frame, can be represented by a residual feature grid and a coefficient feature grid with the same basic dimension, thereby obtaining the represented dynamic neural radiation field.

[0071] The residual feature mesh represents the error compensation region and the newly observed region between the base mesh of the keyframe and the base mesh of the current frame. The residual feature mesh can enhance the representation by incorporating residual information and reducing temporal redundancy.

[0072] In another possible embodiment, if the residual feature mesh of a certain frame is obtained, the dynamic neural radiation field of that frame can be recovered by summing the residual feature mesh and the base mesh of the key frame.

[0073] S12, perform model training and simulation quantization on the represented dynamic neural radiation field to determine the amount of predicted compressed data for the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization.

[0074] As an example, the represented dynamic neural radiation field is trained using a model, and a pre-defined entropy model is used to predict the data distribution of the represented dynamic neural radiation field, thereby determining the amount of compressed data for the prediction of the represented dynamic neural radiation field.

[0075] Specifically, a new data distribution is constructed by using a pre-defined entropy model to represent the dynamic neural radiation field, and the actual data distribution of each feature grid is predicted based on the new data distribution, thereby predicting the bit rate after entropy encoding, i.e., predicting the amount of compressed data.

[0076] In this disclosure, the predicted compressed data of the predicted dynamic neural radiation field can be integrated into a preset loss function. During the model training process of the dynamic neural radiation field, the low-entropy distribution of features is encouraged, and the bit rate of the dynamic neural radiation field is constrained to replace the actual entropy coding process, thus maintaining the differentiability of the distribution model of the original data of the dynamic neural radiation field.

[0077] As another example, when training the model on the represented dynamic neural radiation field, uniform noise is used to perform a simulated quantization operation on the represented dynamic neural radiation field to determine the simulated quantized dynamic neural radiation field.

[0078] By adding uniform noise to the dynamic neural radiation field and performing simulated quantization, the generalization performance of the dynamic neural radiation field can be improved, while maintaining the differentiability of the original data distribution model. After simulated quantization, the dynamic neural radiation field does not require further noise addition after model training.

[0079] S13, optimize the dynamic neural radiation field after simulation quantization based on the preset loss function to determine the trained dynamic neural radiation field.

[0080] The dynamic neural radiation field is optimized by backpropagation using a preset loss function, which includes the distortion loss function, the amount of predicted compressed data, and the L1 norm of the residual feature grid.

[0081] In one possible embodiment, the distortion loss function, the amount of predicted compressed data, and the L1 norm of the residual feature grid are weighted and summed to form a preset loss function.

[0082] Loss=D+λ1R+λ2r

[0083]

[0084] Where Loss represents the preset loss function, and D represents the distortion loss function. c represents the color of the reconstructed image. r Let R represent the true image color, λ1 represent the predicted compressed data volume, r represent the L1 norm of the residual feature network, and λ2 represent the L1 norm loss of the residual feature network.

[0085] In this disclosure, λ1 = λ2 = 0.000001, and λ1 and λ2 are used to balance the loss ratio of the predicted compressed data volume and the L1 norm of the residual feature grid.

[0086] Among them, the distortion loss function is used to represent the reconstruction quality of the dynamic neural radiation field; the predicted compressed data volume is the predicted compressed data volume of the dynamic neural radiation field, which is used to constrain the data volume of the compressed dynamic neural radiation field; the L1 norm of the residual feature network is used to constrain the residual feature network.

[0087] S14 performs uniform quantization and encoding on the trained dynamic neural radiation field to determine the actual compressed data volume of the dynamic neural radiation field.

[0088] In one possible embodiment, the feature grid of the trained dynamic neural radiation field is uniformly quantized to determine the uniformly quantized dynamic neural radiation field.

[0089] The accuracy of uniform quantization can be adjusted by changing the quantization parameters.

[0090] In another possible embodiment, the dynamic neural radiation field, after being uniformly quantized, is interval encoded to determine the actual compressed data volume of the dynamic neural radiation field.

[0091] The actual compressed data volume of the dynamic neural radiation field is the compression bit rate of the dynamic neural radiation field after interval coding.

[0092] In this disclosure, before performing interval encoding on the dynamic neural radiation field, it is necessary to convert the dynamic neural radiation field to a non-negative interval.

[0093] The process of uniform quantization and interval encoding of dynamic neural radiation fields is as follows:

[0094]

[0095] Where E is the interval encoder, D is the interval decoder, Q is the uniform quantization process, and q represents the quantization parameter. This represents the actual compressed data volume of the dynamic neural radiation field, where x represents the quantized data of the dynamic neural radiation field, specifically including the basic feature grid B and the coefficient feature grid C.

[0096] The above technical solution uses coefficient feature grids and basic feature grids to represent the dynamic neural radiation field, and uses residual feature grids and coefficient feature grids to compensate for the error region of the dynamic neural radiation field, thus determining the represented dynamic neural radiation field. The represented dynamic neural radiation field is then subjected to model training and encoding processes, simultaneously optimizing the representation and compression of the dynamic neural radiation field, improving the reconstruction quality and compression rate. The representation and compression process of the dynamic neural radiation field is jointly optimized end-to-end, and differentiable analog quantization operations and a pre-set entropy model are used to promote end-to-end training of the dynamic neural radiation field, achieving efficient modeling and compression of the dynamic neural radiation field.

[0097] In one possible embodiment, the model-trained dynamic neural radiation field of this disclosure can be evaluated for its reconstruction quality using peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and the performance of the model-trained dynamic neural radiation field can also be evaluated using the amount of stored data and rate-distortion (RD) performance.

[0098] The dynamic neural radiation field disclosed herein is represented by coefficient feature grids and basic feature grids, which has the characteristics of low entropy and high reconstruction quality. The compressed data volume of the dynamic neural radiation field after compression is also small. Therefore, the end-to-end joint optimization method of dynamic neural radiation field representation and compression provided in this disclosure can efficiently realize the modeling and compression of long sequence dynamic neural radiation fields, and significantly improve rate distortion performance.

[0099] Figure 2 This is a block diagram illustrating an end-to-end joint optimization system for dynamic neural radiation field representation and compression, according to an exemplary embodiment.

[0100] Based on the same concept, this disclosure also provides an end-to-end joint optimization system 100 for dynamic neural radiation field representation and compression, such as Figure 2 As shown, it includes: a representation module 110, a model training module 120, an optimization module 130, and a compression module 140.

[0101] The representation module 110 is used to represent the dynamic neural radiation field using a coefficient feature grid and a basic feature grid, to compensate for the error region of the dynamic neural radiation field using a residual feature grid and the coefficient feature grid, and to determine the represented dynamic neural radiation field.

[0102] The model training module 120 is used to perform model training and simulation quantization processing on the represented dynamic neural radiation field to determine the predicted compressed data volume of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization processing.

[0103] The optimization module 130 is used to optimize the dynamic neural radiation field after simulation quantization based on the predicted compressed data volume of the represented dynamic neural radiation field and a preset loss function, and to determine the trained dynamic neural radiation field.

[0104] Compression module 140 is used to perform uniform quantization and encoding processing on the trained dynamic neural radiation field to determine the actual compressed data volume of the dynamic neural radiation field.

[0105] The above technical solution uses coefficient feature grids and basic feature grids to represent the dynamic neural radiation field, and uses residual feature grids and coefficient feature grids to compensate for the error region of the dynamic neural radiation field, thus determining the represented dynamic neural radiation field. Then, the represented dynamic neural radiation field is subjected to model training and encoding processing, thereby optimizing the representation and compression of the dynamic neural radiation field, improving the reconstruction quality and compression rate of the dynamic neural radiation field. The representation and compression process of the dynamic neural radiation field is jointly optimized end-to-end, and differentiable analog quantization operations are used to promote the end-to-end training of the dynamic neural radiation field, thus achieving efficient modeling and compression of the dynamic neural radiation field.

[0106] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0107] Based on the same concept described above, in another embodiment of this disclosure, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to perform an end-to-end joint optimization method for dynamic neural radiation field representation and compression.

[0108] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0109] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0110] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0111] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0112] In this embodiment of the disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of an end-to-end joint optimization method for dynamic neural radiation field representation and compression in any of the above embodiments.

[0113] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Although preferred embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0118] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An end-to-end joint optimization method for dynamic neural radiation field representation and compression, characterized in that, include: The dynamic neural radiation field is represented by a coefficient feature grid and a basic feature grid. The error region of the dynamic neural radiation field is compensated by a residual feature grid and the coefficient feature grid. The represented dynamic neural radiation field is determined. The coefficient feature grid is used to represent spatial variation, the basic feature grid is used to represent the commonality of the signal, and the residual feature grid is used to represent the error compensation region between the basic grid of the key frame and the basic grid of the current frame and the newly observed region. The represented dynamic neural radiation field is subjected to model training and simulation quantization to determine the predicted compressed data volume of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization. The dynamic neural radiation field, which has undergone simulation quantization, is optimized according to a preset loss function to determine the trained dynamic neural radiation field. The dynamic neural radiation field after training is uniformly quantized and encoded to determine the actual compressed data volume of the dynamic neural radiation field. The step of performing model training and simulation quantization on the represented dynamic neural radiation field to determine the predicted compressed data volume of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization includes: The represented dynamic neural radiation field is subjected to model training processing, and a preset entropy model is used to perform data distribution prediction processing on the represented dynamic neural radiation field to determine the amount of predicted compressed data of the represented dynamic neural radiation field. When training the model on the represented dynamic neural radiation field, uniform noise is used to perform a simulation quantization operation on the represented dynamic neural radiation field to determine the dynamic neural radiation field after the simulation quantization operation.

2. The method according to claim 1, characterized in that, The process of representing the dynamic neural radiation field using coefficient feature grids and basic feature grids, compensating for the error region of the dynamic neural radiation field using residual feature grids and the coefficient feature grids, and determining the represented dynamic neural radiation field includes: Based on the position of the light rays, interpolation processing is performed on the coefficient feature grid and the basic feature grid to determine the interpolated coefficient feature grid and the interpolated basic feature grid; The interpolated coefficient feature grid and the interpolated basic feature grid are subjected to a Hadman product to determine the Hadman product of the coefficient feature grid and the basic feature grid. The Hadman product of the coefficient feature grid and the basic feature grid is input into a multilayer perceptron to determine the reconstructed dynamic neural radiation field.

3. The method according to claim 2, characterized in that, The process of representing the dynamic neural radiation field using coefficient feature grids and basic feature grids, compensating for the error region of the dynamic neural radiation field using residual feature grids and the coefficient feature grids, and determining the represented dynamic neural radiation field further includes: The reconstructed dynamic neural radiation field is represented by a continuous set of feature grids; The initial frame of each of the feature grid groups is represented using the long reference feature grid of the key frame, and each remaining frame of each of the feature grid groups is represented using the residual feature grid and the coefficient feature grid, thereby determining the represented dynamic neural radiation field.

4. The method according to claim 1, characterized in that, The preset loss function includes the distortion loss function, the amount of predicted compressed data, and the L1 norm of the residual feature grid.

5. The method according to claim 1, characterized in that, The process of uniformly quantizing and encoding the trained dynamic neural radiation field to determine the actual compressed data volume of the dynamic neural radiation field includes: The feature grid of the trained dynamic neural radiation field is uniformly quantized to determine the uniformly quantized dynamic neural radiation field. The dynamic neural radiation field, after uniform quantization, is interval encoded to determine the actual compressed data volume of the dynamic neural radiation field.

6. An end-to-end joint optimization system for dynamic neural radiation field representation and compression, characterized in that, include: The representation module is used to represent the dynamic neural radiation field using a coefficient feature grid and a basic feature grid, to compensate for the error region of the dynamic neural radiation field using a residual feature grid and the coefficient feature grid, and to determine the represented dynamic neural radiation field. The coefficient feature grid is used to represent spatial variation, the basic feature grid is used to represent the commonality of the signal, and the residual feature grid is used to represent the error compensation region and the newly observed region between the basic grid of the key frame and the basic grid of the current frame. The model training module is used to perform model training and simulation quantization processing on the represented dynamic neural radiation field to determine the amount of predicted compressed data of the represented dynamic neural radiation field and the dynamic neural radiation field after simulation quantization. An optimization module is used to optimize the dynamic neural radiation field that has undergone simulation quantization processing according to a preset loss function, and to determine the dynamic neural radiation field after training. A compression module is used to perform uniform quantization and encoding processing on the trained dynamic neural radiation field to determine the actual compressed data volume of the dynamic neural radiation field. The model training module is used for: The represented dynamic neural radiation field is subjected to model training processing, and a preset entropy model is used to perform data distribution prediction processing on the represented dynamic neural radiation field to determine the amount of predicted compressed data of the represented dynamic neural radiation field. When training the model on the represented dynamic neural radiation field, uniform noise is used to perform a simulation quantization operation on the represented dynamic neural radiation field to determine the dynamic neural radiation field after the simulation quantization operation.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images