Encoding and decoding methods and related devices for dynamic scenes
By introducing time-varying features and deformation indication information into the rendering units in dynamic scenes and compressing and encoding them, the problem of excessive storage requirements and bandwidth in 3D Gaussian sputtering technology is solved, achieving a balance between low storage overhead and high reconstruction quality.
Patent Information
- Application Number
- CN202510604361.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-10
- Publication Date
- 2026-11-17
AI Technical Summary
Existing 3D Gaussian sputtering technology has excessively high storage and bandwidth requirements when reconstructing dynamic scenes, making it difficult to balance low storage overhead and high reconstruction quality.
By introducing time-varying features and deformation indication information, the rendering units in dynamic scenes are compressed and encoded, storing only the reference frame data and the target time-varying features, thus reducing storage and transmission bandwidth requirements.
It achieves a balance between low storage overhead and high reconstruction quality, ensuring detailed reconstruction quality of dynamic scenes while reducing storage space and transmission bandwidth requirements.
Smart Images

Figure CN123123583A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method and apparatus for encoding and decoding dynamic scenes. Background Technology
[0002] In recent years, 3D / 4D Gaussian splatting (GS) technology has made significant progress in the field of dynamic scene reconstruction due to its efficient scene representation capabilities and real-time rendering potential. Taking 3D Gaussian splatting (3DGS) as an example, it uses 3D Gaussian primitives to construct and represent the local geometry and appearance of a scene. A 3D Gaussian primitive is a parameterized, resolvable geometric primitive that describes the local geometry and appearance of a scene through parameters such as position, covariance matrix, color, and transparency. However, 3D Gaussian primitives themselves contain multiple parameters, and as scene complexity increases, a large number of 3D Gaussian primitives are needed to accurately represent objects and details in the scene. These primitive parameters accumulate rapidly, leading to a significant increase in storage requirements. Simultaneously, to achieve high-quality rendering, a sufficient number of 3D Gaussian primitives are needed to accurately describe scene details. While increasing the number of 3D Gaussian primitives can improve rendering quality, it also further increases storage overhead. Therefore, when using 3DGS technology to reconstruct and represent dynamic scenes, balancing low storage overhead (or low bandwidth) with high reconstruction quality has become a pressing issue. Summary of the Invention
[0003] This application provides a method and related apparatus for encoding and decoding dynamic scenes, which, for the scene representation data (data of the reference frame, deformation indication information of multiple rendering units, and time-varying features of the target) determined for dynamic scenes, can balance the requirements of low storage overhead and high reconstruction quality. The technical solution is as follows:
[0004] In a first aspect, a method for encoding dynamic scenes is provided. The method includes: acquiring data of a reference frame of the dynamic scene, time-varying features and deformation indication information of multiple rendering units in the current frame, wherein the data of the reference frame includes position information and attribute information of the multiple rendering units in the reference frame, the time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering units in the current frame relative to the reference frame, and the deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering units in the current frame has changed relative to the reference frame; compressing the time-varying features of the multiple rendering units based on the deformation indication information to obtain target time-varying features; and encoding the data of the reference frame, the deformation indication information, and the target time-varying features into a bitstream.
[0005] The dynamic scene refers to a three-dimensional scene that changes over time.
[0006] In one possible implementation, dynamic scenes can be represented and reconstructed using a dynamic 3D Gaussian splashing scheme, and the rendering unit can be specifically a 3D Gaussian unit.
[0007] Therefore, this application introduces corresponding time-varying features for each of the multiple rendering units in a dynamic scene, enabling a more flexible and comprehensive description of the deformation of each rendering unit. Simultaneously, it introduces corresponding deformation indication information for each rendering unit to intuitively reflect whether the position and attribute information of each rendering unit in the current frame has changed relative to the reference frame. Thus, by using time-varying features and deformation indication information, the deformation of each rendering unit can be described in detail, improving rendering quality. When storing or encoding scene representation data for a dynamic scene, the deformation indication information is used to compress the time-varying features of multiple rendering units to obtain the target time-varying features. This allows only the data of the reference frame of the dynamic scene, the target time-varying features, and the deformation indication information of multiple rendering units to be stored / encoded, eliminating the need to store the complete time-varying features of each rendering unit. This reduces the storage space requirements for storing time-varying features and the bandwidth requirements for transmitting them. Thus, based on reference frame data, deformation indication information, and target time-varying characteristics, this application can not only accurately describe the details of the dynamic scene to ensure the quality of reconstructing / rendering the dynamic scene, but also reduce the requirements for storage space and transmission bandwidth, achieving the effect of low storage overhead and high reconstruction quality at the same time.
[0008] In one possible implementation, obtaining the time-varying features and deformation indication information of multiple rendering units in the dynamic scene includes: obtaining the time-varying features and deformation indication information based on a target neural network, wherein the target neural network is trained based on the position information of the multiple rendering units in the reference frame and the image of the current frame.
[0009] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0010] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0011] In one possible implementation, the step of encoding the deformation indication information into the bitstream includes: obtaining a first feature and / or a second feature through a first neural network based on the deformation indication information of the plurality of rendering units, wherein the first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame; and encoding the first feature and / or the second feature into the bitstream.
[0012] Among them, the target time-varying feature is used to describe the change of target information of at least one target rendering unit. The target rendering unit refers to the rendering unit whose target information changes relative to the reference frame. The target information includes the position information and attribute information of the target rendering unit that have changed relative to the reference frame.
[0013] In one possible implementation, the time-varying features of the plurality of rendering units include positional features and / or attribute features; the step of compressing the time-varying features of the plurality of rendering units based on the deformation indication information to obtain the target time-varying features includes: if the deformation indication information of the target rendering unit indicates that the attribute information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying features do not include the attribute features of the target rendering unit, and the target rendering unit is any one of the plurality of rendering units; if the deformation indication information of the target rendering unit indicates that the position information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying features do not include the positional features of the target rendering unit; if the deformation indication information of the target rendering unit indicates that both the positional information and attribute information of the target rendering unit in the current frame have not changed relative to the reference frame, then the target time-varying features do not include the time-varying features of the target rendering unit.
[0014] Secondly, a method for decoding dynamic scenes is provided. The method includes: parsing data of a reference frame of the dynamic scene, deformation indication information of multiple rendering units in the dynamic scene, and target time-varying features from a bitstream. The data of the reference frame includes position information and attribute information of the multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. Based on the deformation indication information and the target time-varying features, the time-varying features are determined, and the time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. Based on the data of the reference frame and the time-varying features of the multiple rendering units, the position information and / or attribute information of the multiple rendering units in the current frame are determined respectively.
[0015] The dynamic scene refers to a three-dimensional scene that changes over time.
[0016] In one possible implementation, dynamic scenes can be represented and reconstructed using a dynamic 3D Gaussian splashing scheme, and the rendering unit can be specifically a 3D Gaussian unit.
[0017] Therefore, this application significantly reduces the bandwidth requirements for transmitting the time-varying features of multiple rendering units in a dynamic scene by encoding and transmitting the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features. After parsing the deformation indication information and target time-varying features of multiple rendering units from the bitstream, the time-varying features of each rendering unit can be recovered based on the deformation indication information. Then, combined with the data of the reference frame of the dynamic scene, the position information and attribute information of each rendering unit in the current frame can be accurately determined. Thus, based on the data of the reference frame, deformation indication information, and target time-varying features, this application can not only accurately describe the details of the dynamic scene to ensure the quality of reconstructing / rendering the dynamic scene, but also reduce the transmission bandwidth requirements, achieving the effect of low storage overhead and high reconstruction quality simultaneously.
[0018] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0019] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0020] In one possible implementation, parsing deformation indication information of multiple rendering units in the dynamic scene from the bitstream includes: parsing a first feature and / or a second feature from the bitstream, wherein the first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame; and obtaining the deformation indication information through a first neural network based on the first feature and / or the second feature.
[0021] In one possible implementation, determining the position information and / or attribute information of the plurality of rendering units in the current frame based on the data of the reference frame and the time-varying characteristics of the plurality of rendering units includes: inputting the data of the reference frame and the time-varying characteristics of the plurality of rendering units into a second neural network to obtain the position change amount and / or attribute change amount of the plurality of rendering units in the current frame, respectively, output by the second neural network. The position change amount indicates the change in the position information of the rendering unit in the current frame relative to the reference frame, and the attribute change amount indicates the change in the attribute information of the rendering unit in the current frame relative to the reference frame. Based on the data of the reference frame and the position change amount and / or attribute change amount of the plurality of rendering units in the current frame, the position information and / or attribute information of the plurality of rendering units in the current frame are determined.
[0022] In one possible implementation, the time-varying features of the plurality of rendering units include positional features and / or attribute features, and the second neural network includes a positional analysis subnetwork and / or an attribute analysis subnetwork; the step of inputting the data of the reference frame and the time-varying features of the plurality of rendering units into the second neural network to obtain the positional change and / or attribute change of the plurality of rendering units in the current frame, respectively, output by the second neural network, includes: inputting the positional information or positional information and attribute information of the plurality of rendering units in the reference frame, and the positional features of the plurality of rendering units into the positional analysis subnetwork to obtain the positional change of the current frame relative to the reference frame, respectively, output by the positional analysis subnetwork; and / or, inputting the attribute information and attribute features of the plurality of rendering units in the reference frame into the attribute analysis subnetwork to obtain the attribute change of the current frame relative to the reference frame, respectively, output by the attribute analysis subnetwork.
[0023] Thirdly, a dynamic scene encoding apparatus is provided, which has the function of implementing the dynamic scene encoding method behavior described in the first aspect above. The dynamic scene encoding apparatus includes at least one module, which is used to implement the dynamic scene encoding method provided in the first aspect above.
[0024] In one possible implementation, the encoding device for the dynamic scene includes:
[0025] The acquisition module is used to acquire data of a reference frame of a dynamic scene, time-varying features and deformation indication information of multiple rendering units in the current frame. The data of the reference frame includes the position information and attribute information of the multiple rendering units in the reference frame. The time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame.
[0026] A compression module is used to compress the time-varying features of the multiple rendering units based on the deformation indication information to obtain the target time-varying features;
[0027] The encoding module is used to encode the data of the reference frame, the deformation indication information, and the target time-varying features into the bitstream.
[0028] In one possible implementation, the acquisition module is specifically used for:
[0029] The time-varying features and deformation indication information are obtained based on the target neural network, which is trained based on the position information of the multiple rendering units in the reference frame and the image of the current frame.
[0030] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0031] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0032] In one possible implementation, the encoding module is specifically used for:
[0033] Based on the deformation indication information of the plurality of rendering units, a first feature and / or a second feature are obtained through a first neural network. The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame.
[0034] The first feature and / or the second feature are encoded into the bitstream.
[0035] In one possible implementation, the time-varying features of the plurality of rendering units include positional features and / or attribute features; the compression module is specifically used for:
[0036] If the deformation indication information of the target rendering unit indicates that the attribute information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the attribute feature of the target rendering unit, and the target rendering unit is any one of the plurality of rendering units.
[0037] If the deformation indication information of the target rendering unit indicates that the position information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the position feature of the target rendering unit;
[0038] If the deformation indication information of the target rendering unit indicates that the position information and attribute information of the target rendering unit in the current frame have not changed relative to the reference frame, then the target time-varying feature does not include the time-varying feature of the target rendering unit.
[0039] In one possible implementation, the rendering unit is a 3D Gaussian unit.
[0040] In one possible implementation, the dynamic scene refers to a three-dimensional scene that changes over time.
[0041] Fourthly, a dynamic scene decoding device is provided, which has the function of implementing the dynamic scene decoding method behavior described in the second aspect above. The dynamic scene decoding device includes at least one module for implementing the dynamic scene decoding method provided in the second aspect above.
[0042] In one possible implementation, the decoding device for the dynamic scene includes:
[0043] The decoding module is used to parse the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features from the bitstream. The data of the reference frame includes the position information and attribute information of the multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame.
[0044] The first determining module is used to determine the time-varying features of the plurality of rendering units based on the deformation indication information and the target time-varying features, wherein the time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame;
[0045] The second determining module is used to determine the position information and / or attribute information of the multiple rendering units in the current frame based on the data of the reference frame and the time-varying characteristics of the multiple rendering units.
[0046] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0047] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0048] In one possible implementation, the decoding module is specifically used for:
[0049] The first feature and / or the second feature are parsed from the bitstream. The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame.
[0050] Based on the first feature and / or the second feature, the deformation indication information is obtained through a first neural network.
[0051] In one possible implementation, the second determining module includes:
[0052] The first determining unit is used to input the data of the reference frame and the time-varying features of the plurality of rendering units into the second neural network to obtain the position change amount and / or attribute change amount of the plurality of rendering units in the current frame, respectively, output by the second neural network. The position change amount indicates the change amount of the position information of the rendering unit in the current frame relative to the reference frame, and the attribute change amount indicates the change amount of the attribute information of the rendering unit in the current frame relative to the reference frame.
[0053] The second determining unit is used to determine the position information and / or attribute information of the plurality of rendering units in the current frame based on the data of the reference frame and the position change and / or attribute change of the plurality of rendering units in the current frame.
[0054] In one possible implementation, the time-varying features of the plurality of rendering units include positional features and / or attribute features, and the second neural network includes a positional analysis subnetwork and / or an attribute analysis subnetwork; the first determining unit is specifically used for:
[0055] The position information or position information and attribute information of the plurality of rendering units in the reference frame, as well as the position features of the plurality of rendering units, are input into the position analysis sub-network to obtain the position change of the plurality of rendering units in the current frame relative to the reference frame, respectively, output by the position analysis sub-network; and / or
[0056] The attribute information and attribute features of the multiple rendering units in the reference frame are input into the attribute analysis sub-network to obtain the attribute change amount of the current frame relative to the reference frame for each of the multiple rendering units, as output by the attribute analysis sub-network.
[0057] In one possible implementation, the rendering unit is a 3D Gaussian unit.
[0058] In one possible implementation, the dynamic scene refers to a three-dimensional scene that changes over time.
[0059] Fifthly, a computer device is provided, the computer device including a processor and a memory;
[0060] The memory is used to store a computer program that executes the encoding method provided in the first aspect above, or to store a computer program that executes the decoding method provided in the second aspect above.
[0061] The processor is configured to execute a computer program stored in the memory to implement the dynamic scene encoding method described in the first aspect above, or to implement the dynamic scene decoding method described in the second aspect above.
[0062] In one possible implementation, the computer device may further include a communication bus for establishing a connection between the processor and the memory.
[0063] In a sixth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the dynamic scene encoding method described in the first aspect, or the steps of the dynamic scene decoding method described in the second aspect.
[0064] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer, cause the computer to perform the steps of the dynamic scene encoding method described in the first aspect, or the steps of the dynamic scene decoding method described in the second aspect.
[0065] Alternatively, a computer program is provided that, when run on a computer, causes the computer to perform the steps of the dynamic scene encoding method described in the first aspect above, or the steps of the dynamic scene decoding method described in the second aspect above.
[0066] Eighthly, a coding and decoding system is provided, the coding and decoding system including an encoding device and a decoding device, the encoding device being used to implement the steps of the dynamic scene encoding method described in the first aspect above, and the decoding device being used to implement the steps of the dynamic scene decoding method described in the second aspect above.
[0067] In a ninth aspect, an encoded bitstream is provided, the bitstream being generated according to the encoding method for dynamic scenes described in the first aspect above.
[0068] In a tenth aspect, an encoded bitstream is provided, the bitstream including reference frame data of a dynamic scene, deformation indication information of multiple rendering units in the dynamic scene, and time-varying features of the target;
[0069] The reference frame data includes the position information and attribute information of the plurality of rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the multi-frame images has changed relative to the reference frame. The multi-frame images are used to describe the dynamic scene. The target time-varying feature is obtained by compressing the time-varying features of the plurality of rendering units based on the deformation indication information. The time-varying feature is used to describe the changes in the position information and attribute information of the corresponding rendering unit in the multi-frame images relative to the reference frame.
[0070] In an eleventh aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a bitstream generated according to the encoding method for dynamic scenes described in the first aspect above.
[0071] In a twelfth aspect, an apparatus for storing a bitstream is provided, the apparatus comprising: a receiver and at least one storage medium;
[0072] The receiver is used to receive the bitstream generated according to the encoding method of the dynamic scene described in the first aspect above;
[0073] The at least one storage medium is used to store the bitstream.
[0074] In a thirteenth aspect, an apparatus for transmitting a code stream is provided, the apparatus comprising: a transmitter and a receiver;
[0075] The receiver is used to receive the bitstream generated according to the encoding method of the dynamic scene described in the first aspect above;
[0076] The transmitter is used to transmit the bitstream.
[0077] In a fourteenth aspect, an apparatus for transmitting a code stream is provided, the apparatus comprising: a transmitter and at least one storage medium;
[0078] The at least one storage medium is used to store the bitstream generated according to the encoding method of the dynamic scene described in the first aspect above;
[0079] The transmitter is used to obtain the bitstream from the storage medium and transmit the bitstream.
[0080] In a fifteenth aspect, a system for distributing bitstreams is provided, the system comprising:
[0081] At least one storage medium for storing a bitstream generated according to the encoding method for the dynamic scene described in the first aspect above;
[0082] A streaming media device is configured to acquire a target bitstream from the at least one storage medium and send the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.
[0083] The technical effects achieved by the third to fifteenth aspects mentioned above are similar to those achieved by the corresponding technical means in the first and second aspects, and will not be repeated here. Attached Figure Description
[0084] Figure 1 This is a schematic diagram of the process for obtaining the deformation parameters of a 3D Gaussian element at time t in a related technology.
[0085] Figure 2 This is a flowchart illustrating the process of obtaining the deformation parameters of a 3D Gaussian element at time t in another related technique.
[0086] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;
[0087] Figure 4 This is a flowchart illustrating a dynamic scene encoding method provided in an embodiment of this application;
[0088] Figure 5 This is a schematic diagram illustrating the classification of the deformation field of a three-dimensional Gaussian element provided in an embodiment of this application;
[0089] Figure 6 This is a flowchart illustrating a dynamic scene decoding method provided in an embodiment of this application;
[0090] Figure 7 This is a schematic diagram of the structure of an encoding device for a dynamic scene provided in an embodiment of this application;
[0091] Figure 8 This is a schematic diagram of the structure of a dynamic scene decoding device provided in an embodiment of this application. Detailed Implementation
[0092] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0093] To facilitate understanding, before explaining the encoding and decoding method for dynamic scenes provided in this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.
[0094] First, the terminology used in the embodiments of this application will be introduced.
[0095] Explicit field representation, also known as explicit field representation, is a method for directly describing the properties of each point in space. In three-dimensional (D) space (or higher-dimensional space), each point can be uniquely identified by a location coordinate (e.g., (x, y, z)); these location coordinates are assigned one or more specific attribute values, thus comprehensively and explicitly describing the state of the entire space. That is, in explicit field representation, each point in space corresponds to a location coordinate and one or more attribute parameters. These attribute parameters can represent various physical quantities, including but not limited to density, color, temperature, velocity, transparency, and brightness.
[0096] Deformation field: A mathematical tool or model used to describe the dynamic changes of point clouds or scenes in three-dimensional space over time. In dynamic 3D Gaussian splatting (3DGS) or other related dynamic scene modeling techniques, the deformation field simulates the motion, shape changes, and other dynamic behaviors of objects by assigning a set of time-varying offsets or transformation parameters to each basic unit (e.g., a 3D Gaussian primitive).
[0097] Time-varying features: In machine learning, time-varying features refer to data or features that change over time. The values of these features are not static but change over time, thus exhibiting time dependence.
[0098] Secondly, the application scenarios of the embodiments of this application will be introduced.
[0099] 3D Gaussian splashing (also known as 3D Gaussian sputtering) is based on explicit scene representation, decomposing the scene into a large number of 3D Gaussian primitives and using an efficient computational framework to achieve fast rendering. 3D Gaussian splashing overcomes the bottlenecks of traditional neural rendering methods (such as Neural Radiation Fields, NeRF) that rely on implicit representation and extensive sampling optimization, achieving a significant balance between rendering speed, real-time performance, and image quality, making it a popular research direction in the fields of 3D reconstruction and virtual reality.
[0100] 3D Gaussian representation is a method for explicitly parametrically modeling 3D scenes. Its core idea is to describe the geometric, color, and lighting properties of a scene using a set of 3D Gaussian primitives. Currently, the most common primitive form for 3D Gaussian primitives is the ellipsoid; therefore, 3D Gaussian primitives can also be called 3D Gaussian ellipsoids.
[0101] The geometry of each three-dimensional Gaussian ellipsoid is determined by the coordinates of its center position (u). x ,u y ,u z The shape of each 3D Gaussian ellipsoid is determined by the covariance matrix Σ, while its color is represented by opacity and spherical harmonic coefficients (SHs). To ensure the positive semi-definite property of the covariance matrix during optimization, it is typically split into a scaling matrix and a rotation matrix. The actual optimization parameters of the scaling matrix are stored as a 3D vector s representing the scaling ratio, and the actual optimization parameters of the rotation matrix are stored as rotation quaternions q. A scalar α represents the opacity of the 3D Gaussian ellipsoid (0 ≤ α ≤ 1), controlling the degree of occlusion of the background by the 3D Gaussian ellipsoid. SHs are used to model the incident light distribution (i.e., ambient light) on the surface of the 3D Gaussian ellipsoid, supporting global illumination effects.
[0102] The main process of 3D Gaussian splashing is as follows: Input a set of multi-view images (objects or environments photographed from different perspectives) or video sequences (e.g., dynamic scenes, but requiring inter-frame consistency); Based on the structure-from-motion (SfM) algorithm, recover the camera pose (position and orientation) and sparse point cloud of the scene from the multi-view images; Convert the sparse point cloud into an initial set of 3D Gaussian ellipsoids, each ellipsoid representing a local region of the scene; These initial 3D Gaussian ellipsoids are cloned, split, pruned, and refined to improve their quality and ensure the accuracy of the scene's geometry, color, and lighting; Perform rasterization rendering through a fast differentiable rasterizer to project the optimized 3D Gaussian ellipsoid set onto the target viewpoint, generating a new viewpoint image.
[0103] Because 3D Gaussian splashing reconstructs scenes using an explicitly parameterized set of 3D Gaussian ellipsoids, its high precision and parameterization complexity directly lead to a surge in model storage requirements. In other words, for some complex scenes (such as outdoor 360°), to ensure high-quality reconstruction, a large number of 3D Gaussian ellipsoids need to be constructed and stored. Each 3D Gaussian ellipsoid has multiple attribute parameters, and storing these 3D Gaussian ellipsoids, along with their various attribute parameters, consumes a huge amount of storage space.
[0104] As an example, when modeling the Bicycle scene in the MipNeRF-360 dataset, the data storage of the 3D Gaussian ellipsoid set is as high as 1.4GB, which places high demands on storage space and transmission bandwidth.
[0105] Dynamic 3D Gaussian splatting aims to handle continuous scene changes over time (e.g., moving objects, deformable surfaces). Based on its core methodology, existing research can be categorized into three main approaches:
[0106] The first type of scheme is frame-wise 3D Gaussianplatting, where each frame is treated as an independent 3D Gaussian splash.
[0107] The core idea of this type of scheme is to treat each frame of a dynamic scene as an independent static scene, and model each frame using 3D Gaussian elements (e.g., an ellipsoid). The Gaussian element parameters (such as position, color, covariance matrix, etc.) between different frames are not explicitly correlated; interpolation or alignment algorithms are used to achieve inter-frame transitions.
[0108] The second type of solution: the solution based on deformable 3D Gaussian plating.
[0109] The core idea of this type of scheme is to map the three-dimensional Gaussian ellipsoid parameters of all frames of a dynamic scene to a canonical space through the deformation field. Only the ellipsoid parameters of the canonical space and the parameters of the deformation field need to be stored, thereby reducing the amount of storage.
[0110] Specifically, the deformation-based 3D Gaussian splashing scheme establishes two spaces: a canonical space and a deformation space. The canonical space is a time-independent representation of the dynamic scene, modeling the common geometric structure of all objects in the scene over time using a set of static 3D Gaussian primitives (such as ellipsoids). For example, in a human motion scene, the canonical space might correspond to the standard human posture (such as a T-pose); in fluid simulation, the canonical space might correspond to the initial shape of the fluid (such as a stationary water sphere). The deformation space is a time-dependent representation of the dynamic scene, modeling the dynamic changes of the scene by deforming the ellipsoid of the canonical space into the geometric state of the current frame. For example, in a human motion scene, the deformation space might correspond to a running posture (such as arm swinging and leg bending); in fluid simulation, the deformation space might correspond to the falling and breaking of a water sphere. Since the deformation space is modeled using 3D Gaussian primitives deformed from the canonical space, the deformed 3D Gaussian primitives are used to represent the content of a specific frame; therefore, each frame has a corresponding deformation method.
[0111] The main difference between the existing solutions lies in the different ways of implementing deformation, but its essence can be abstracted as a mapping from time-dependent input to changes in 3D Gaussian meta-attributes.
[0112] The third type of solution: the 4D Gaussian splatting solution.
[0113] The core idea of this type of solution is to model the dynamic scene as a continuous Gaussian distribution in four-dimensional space (3D space + time), and directly parameterize the trajectory of the four-dimensional Gaussian ellipsoid in spacetime, avoiding explicit deformation fields or frame-level storage.
[0114] Specifically, the 4D Gaussian splashing approach expands 3D Gaussian primitives into 4D Gaussian primitives, introducing a temporal dimension to model the dynamic changes of the scene. Each 4D Gaussian primitive independently describes the state of a time period and a local space. Through the distribution and optimization of a large number of 4D Gaussian primitives, high-quality reconstruction of complex dynamic scenes is achieved. However, due to the lack of deformation mapping relationships between each 4D Gaussian primitive, there is a large amount of redundancy in the temporal dimension, which is difficult to compress through simple transformations.
[0115] Compared to the first and third types of solutions, the deformation-based 3D Gaussian splashing approach reduces the storage volume of raw data by decoupling the normal space and deformation space; simultaneously, it ensures motion continuity through deformation mapping, avoiding inter-frame flicker artifacts. Furthermore, both deformation parameters and normal space parameters can be further optimized using compression or neural network techniques. Based on these advantages, the deformation-based 3D Gaussian splashing approach has become the mainstream solution for dynamic scene modeling / reconstruction in the industry.
[0116] In summary, 3D Gaussian sputtering technology, especially the deformation-based 3D Gaussian sputtering scheme, can simulate and reconstruct scenes by distributing a large number of three-dimensional Gaussian units, capturing rich details and dynamic changes in the scene, thus providing high-quality reconstruction results. However, its problems are also quite prominent: to achieve high-quality reconstruction, this technology typically needs to store a large number of 3D Gaussian unit parameters, including but not limited to the center position, color, size, and possible deformation information of the 3D Gaussian units, which leads to high storage overhead. In resource-constrained environments, such as mobile devices or limited network bandwidth, this high storage overhead becomes a key factor restricting its widespread application.
[0117] Therefore, although 3D Gaussian sputtering technology has great potential in the field of dynamic scene reconstruction, it currently cannot simultaneously meet the requirements of low storage overhead and high reconstruction quality. This has prompted researchers to continuously explore new technologies and methods in order to find a better balance between the two.
[0118] Next, we will briefly explain the dynamic scene reconstruction scheme that balances low storage overhead and high reconstruction quality in related technologies.
[0119] One related technology provides a 4D Gaussian scheme, the core idea of which is to encode time-varying features in multiple two-dimensional feature planes, obtain features at any time through bilinear interpolation, and input them into a multi-layer perceptron (MLP) network for inference to predict the deformation of 3D Gaussian units (i.e., changes in position and other attributes).
[0120] The 4D Gaussian scheme sets up six two-dimensional feature planes: xt feature plane, yt feature plane, zt feature plane, xy feature plane, yz feature plane and xz feature plane, each two-dimensional feature plane is a two-dimensional grid (e.g. H×W resolution).
[0121] See Figure 1The process of obtaining the deformation parameters of a 3D Gaussian primitive at time t using the 4D Gaussian scheme is as follows: The position and attribute information of the 3D Gaussian primitive in the reference frame is obtained. Based on the position information (e.g., coordinates (x, y, z)) of the 3D Gaussian primitive in the reference frame, a query is performed on the feature vectors of the six two-dimensional feature planes. The corresponding features on the six two-dimensional feature planes are obtained through bilinear interpolation, forming a feature vector. Then, this feature vector is input into an MLP network for inference. The MLP network determines the position and shape changes of the 3D Gaussian primitive at time t relative to the reference frame (scaling matrix s and rotation quaternion q). Finally, by combining the transparency and spherical harmonic coefficients of the 3D Gaussian primitive in the reference frame, the complete deformation parameters (i.e., deformation field) of the 3D Gaussian primitive at time t are obtained, and subsequent rendering processes are then executed. The complete deformation parameters here include the position coordinates of the 3D Gaussian primitive at time t, the scaling matrix s, the rotation quaternion q, the transparency, and the spherical harmonic coefficients.
[0122] However, the deformation field representation of the aforementioned 4D Gaussian work only describes the position and shape changes of 3D Gaussian primitives (scaling matrix s and rotation quaternion q), lacking a time-varying description of the spherical harmonic coefficients and transparency of 3D Gaussian primitives. In actual rendering, these two properties are assumed to be the same as those in the reference frame (i.e., the spherical harmonic coefficients and transparency of 3D Gaussian primitives in the reference frame are reused), which makes this work only applicable to scenes with limited motion changes, resulting in limited expressive power for complex dynamic scenes (such as source movement or dynamic changes in object surface material, time-varying fading / revealing of semi-transparent objects such as smoke or flames).
[0123] Moreover, in the above-mentioned projection feature mapping, in order to distinguish the features of some neighboring 3D Gaussian elements, it is necessary to increase the density of Gaussian elements and improve the resolution of the two-dimensional feature plane, which will lead to a surge in storage overhead; in addition, the high density of Gaussian elements increases the amount of projection computation and reduces real-time performance.
[0124] Related technology 2 provides a 3DGS representation scheme for embedded deformation fields (per-Gaussian embedding-based deformation for deformable 3D Gaussian splatting, or E3DGS for short), which introduces globally shared temporal features. and E3DGS is used to describe coarse and fine deformations in dynamic scenes, respectively. By dividing the deformation field into two levels, coarse and fine deformation, E3DGS can more efficiently model complex motions in dynamic scenes.
[0125] See Figure 2The E3DGS scheme obtains the deformation parameters of 3D Gaussian elements at time t as follows: Obtain the attribute information (denoted as Y) of the 3D Gaussian elements in the reference frame, and then, based on the time characteristics at time t... and time characteristics Feature queries are performed, and the query results are combined with the spatial features of each 3D Gaussian unit to form a feature vector, which is then fed into an MLP network describing the rough deformation field (denoted as ). ) and MLP networks describing fine deformation fields (denoted as Inference is performed to obtain the coarse change (denoted as ΔY) of the properties of 3D Gaussian elements during the time interval t1 to t2 (t1≤t≤t2). c ), and the fine changes in the 3D Gaussian element properties at time t (denoted as ΔY). f Based on this, the deformation parameter Y of the 3D Gaussian element at time t t =Y+ΔY c +ΔY f Then, rendering is performed.
[0126] However, in the deformation field of E3DGS, the temporal characteristics based on global 3D Gaussian elements... It is difficult to balance the deformation and storage bandwidth consumption of each 3D Gaussian primitive with precision, and it is also impossible to show whether the properties of each 3D Gaussian primitive have changed.
[0127] Based on this, this application comprehensively considers the shortcomings of existing 3D Gaussian sputtering technology in dynamic scene reconstruction and provides an encoding and decoding method for dynamic scenes that can balance low storage overhead and high reconstruction quality. For dynamic scenes, the technical solution of this application introduces corresponding time-varying features for each rendering unit, enabling a more flexible and comprehensive description of the deformation of each rendering unit. Simultaneously, corresponding deformation indication information is introduced for each rendering unit to intuitively reflect whether the position and attribute information of each rendering unit in the current frame has changed relative to the reference frame. Thus, by using time-varying features and deformation indication information, the deformation of each rendering unit can be described in detail, and this detailed description data improves rendering quality. When storing or encoding scene representation data of dynamic scenes, the time-varying features of multiple rendering units are compressed using deformation indication information to obtain target time-varying features. Therefore, only the data of the reference frame of the dynamic scene, the target time-varying features, and the deformation indication information of multiple rendering units are stored / encoded, without needing to store the complete time-varying features of each rendering unit. This reduces the storage space requirements for storing time-varying features and the bandwidth requirements for transmitting time-varying features.
[0128] The dynamic scene encoding and decoding method provided in this application can be applied to any dynamic scene reconstruction scenario. Examples include virtual reality (VR) and augmented reality (AR), game development, robot navigation and autonomous driving, digital twins and industrial simulation, etc.
[0129] In VR / AR applications, 3D / 4D Gaussian splatter technology can provide a more immersive user experience. For example, in virtual tourism, users can "tour" famous landmarks around the world in real time using VR devices, experiencing realistic scenes and dynamic changes. In game development, game developers can use Gaussian splatter technology to achieve more efficient scene rendering and dynamic object modeling, enhancing the game's visual effects and interactivity. For example, in open-world games, dynamic scene reconstruction technology can render effects such as weather changes and day-night cycles in real time. In the fields of robot navigation and autonomous driving, 3D / 4D Gaussian splatter technology can provide robots or autonomous vehicles with more accurate environmental perception and mapping capabilities. For example, by reconstructing dynamic scenes in real time, robots can better avoid obstacles and plan optimal paths. In film and television production, Gaussian splatter technology can be used to create realistic virtual scenes and dynamic characters. For example, in special effects production, dynamic scene reconstruction technology can render complex effects such as explosions and water flow in real time, improving production efficiency and quality. In the field of digital twins, 3D / 4D Gaussian splatter technology can be used to create virtual copies of physical entities, reflecting the entity's state and behavior in real time. For example, in factory simulation, production processes can be optimized and efficiency improved by dynamically reconstructing the operating status of the production line.
[0130] As an example scenario, users can capture multiple frames of images of a dynamic scene from different perspectives using a terminal device (such as a mobile phone). These images need to include dynamic changes over time (such as object movement, lighting changes, etc.) and send the multiple frames to a cloud platform to reconstruct a dynamic 3D Gaussian model through edge-cloud collaboration. The cloud platform reconstructs the dynamic 3D Gaussian model based on the multiple frames and, using the dynamic scene encoding / decoding method provided in this application, compresses and encodes the reconstructed 3D Gaussian model data before transmitting the bitstream to the terminal device over the network. This allows the terminal device to render the 3D Gaussian model in real time using a lightweight rendering engine, supporting user interaction (such as perspective switching and dynamic object tracking).
[0131] In this embodiment, the data for reconstructing the dynamic 3D Gaussian model is the position and attribute information of multiple rendering units in the dynamic scene. That is, after determining the changes in the position and attribute information of multiple rendering units in the dynamic scene, the position and attribute information of any one rendering unit at any time (or the corresponding frame image) can be determined by combining the position and attribute information of multiple rendering units in the reference frame, thereby rendering a two-dimensional image of the dynamic scene from any perspective.
[0132] Finally, the implementation environment of the embodiments of this application will be described.
[0133] The dynamic scene encoding method provided in this application embodiment can be executed by a first computer device to combine multiple frames of images corresponding to the dynamic scene to determine the scene expression data of the dynamic scene and encode the scene expression data into the bitstream; the second computer device executes the decoding method of this application embodiment, recovers the scene expression data of the dynamic scene from the bitstream, can reconstruct the dynamic scene based on the scene expression data, and can also render a two-dimensional image corresponding to the dynamic scene based on the scene expression data.
[0134] The scene representation data for dynamic scenes includes data from the reference frame, deformation indication information from multiple rendering units, and time-varying features of the target. The process of determining the scene representation data is described in the following method embodiment, and will not be elaborated here.
[0135] As an example, the first computer device described above can be deployed on a cloud platform, that is, the encoding method for the dynamic scene provided in the embodiments of this application is executed through the cloud platform; the second computer device described above can be a user terminal, that is, the decoding method for the dynamic scene provided in the embodiments of this application is executed through the user terminal. In other words, the dynamic scene is reconstructed through end-to-cloud collaboration.
[0136] In one possible implementation, the encoding and decoding method for dynamic scenes provided in this application embodiment can also be executed independently by the same computer device, which can be a server or a terminal device.
[0137] When a computer device independently performs dynamic scene reconstruction, it does not perform the operations described in this application embodiment of encoding relevant information into a bitstream and parsing relevant information from the bitstream. Instead, it directly determines and stores the reference frame data of the dynamic scene, deformation indication information of multiple rendering units, and target time-varying features. When it is necessary to reconstruct the dynamic scene, the position and attribute information of multiple rendering units in the current frame (or at a certain moment) are recovered / determined based on the stored information.
[0138] Please refer to Figure 3 , Figure 3This is a schematic diagram of a computer device according to an embodiment of this application. The computer device can be used to execute the encoding and / or decoding methods for dynamic scenes provided in the embodiments of this application. The computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.
[0139] The processor 301 can be a general-purpose central processing unit (CPU), graphics processing unit (GPU), network processor (NP), microprocessor, or one or more integrated circuits for implementing the scheme of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0140] The communication bus 302 is used to transmit information between the aforementioned components. The communication bus 302 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0141] The memory 303 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via a communication bus 302. The memory 303 may also be integrated with the processor 301.
[0142] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 304 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.
[0143] As an example, processor 301 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in the diagram.
[0144] As an example, a computer device may include multiple processors, such as Figure 3 The processors 301 and 305 are shown. Processor 305 may include one or more GPUs, such as... Figure 3 The GPUs shown are GPU0 and GPU1. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).
[0145] In some embodiments, the computer device may further include output devices and input devices. The output device communicates with the processor 301 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 301 and can receive user input in various ways. For example, the input device may be a mouse, a keyboard, a touchscreen device, or a sensing device, etc.
[0146] In some embodiments, memory 303 is used to store program code 310 for executing the scheme of this application, and processor 301 can execute the program code 310 stored in memory 303. The program code 310 may include one or more software modules, and the computer device can implement the following by using processor 301 and the program code 310 in memory 303. Figure 4 The embodiment provides a dynamic scene encoding method, and / or implements the following: Figure 6 The example provides a method for decoding dynamic scenes.
[0147] It should be noted that the application scenarios and implementation environments described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of implementation environments, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0148] The encoding method for dynamic scenes provided in the embodiments of this application will be described in detail below.
[0149] Figure 4 This is a flowchart illustrating a dynamic scene encoding method provided in an embodiment of this application. Exemplarily, this method can be applied to a first computer device. Please refer to... Figure 4 The method includes the following steps.
[0150] Step 401: Obtain the data of the reference frame of the dynamic scene, the time-varying features and deformation indication information of multiple rendering units in the current frame. The data of the reference frame includes the position information and attribute information of multiple rendering units in the reference frame. The time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame.
[0151] In this context, a dynamic scene refers to a three-dimensional scene that changes over time. This dynamic scene can be one where the content changes with the viewpoint, meaning the information seen changes when observing a scene from different perspectives; or it can be one where the content changes over time, meaning the information viewed from a single viewpoint is constantly being updated / changing.
[0152] As an example, a dynamic scene could be a chameleon crawling from one location to another; or a dynamic scene could be a grassland where the grass sways in the breeze like rolling green waves; or a dynamic scene could be a burning flame whose color changes dynamically depending on the degree of burning.
[0153] For some dynamic scenes, only the attribute information of a rendering unit may change over time. In this case, the corresponding time-varying feature is used to describe how the attribute information of the rendering unit changes in the current frame relative to the reference frame. Similarly, for other dynamic scenes, only the position information of a rendering unit may change over time. In this case, the corresponding time-varying feature is used to describe how the position information of the rendering unit changes in the current frame relative to the reference frame. For still other dynamic scenes, both the position and attribute information of a rendering unit may change over time. In this case, the corresponding time-varying feature is used to describe how the position and attribute information of the rendering unit changes in the current frame relative to the reference frame. That is, when the content of a dynamic scene changes differently over time, the information described by the time-varying features of multiple rendering units in that dynamic scene will also be different. The embodiments of this application aim to illustrate how time-varying features are used to describe how a certain piece of information of a rendering unit in the current frame changes relative to the reference frame.
[0154] It should be noted that the current frame can be any frame in the time series. When the reference frame is the first frame in a multi-frame image, the current frame can be any frame following the first frame. Of course, from a temporal perspective, the reference frame can correspond to time t0, and the current frame can be any time after t0, such as t1, t5, t10, etc. Clearly, as time progresses, the amount of time-varying feature data for multiple rendering units in the current frame will also increase. For example, when the current frame corresponds to time t1, the time-varying features of multiple rendering units describe the positional changes and / or attribute changes of multiple rendering units within the time interval t0-t1; while when the previous frame corresponds to time t5, if the time step is 1 unit, then the time-varying features of multiple rendering units describe the positional changes and / or attribute changes of multiple rendering units within the time interval t0-t5, that is, the positional changes and / or attribute changes of the five times t0, t1, t2, t3, t4, and t5 relative to the reference frame.
[0155] The reference frame can be the first frame of a multi-frame image describing the dynamic scene, or it can be determined based on the average of the multi-frame images, or it can be determined in other ways. This application embodiment does not limit the determination method.
[0156] In the embodiments of this application, each rendering unit corresponds to a time-varying feature and deformation indication information. The time-varying features of multiple rendering units can be represented as a whole or independently; similarly, the deformation indication information of multiple rendering units can also be represented as a whole or independently. The embodiments of this application do not impose any restrictions on this.
[0157] Taking the time-varying features of multiple rendering units as an example, when the time-varying features of multiple rendering units are represented independently, the time-varying feature of each rendering unit can be represented by a multi-dimensional vector. However, when the time-varying features of multiple rendering units are represented as a whole, assuming that the time-varying feature of each rendering unit is an 8-dimensional vector, the time-varying features of M rendering units can be represented by an M×8 matrix, where each vector in the matrix is the time-varying feature of the corresponding rendering unit. Of course, the time-varying features of M rendering units can also be combined in other ways. Here, a matrix is only used as an example of one implementation method, and the specific combination method and combination result are not limited in the embodiments of this application.
[0158] In some embodiments, dynamic scenes can be represented and reconstructed using a dynamic 3D Gaussian splashing scheme, and the rendering unit in this application embodiment can specifically be a 3D Gaussian unit.
[0159] In this case, the position information of the rendering unit can be the center position μ of the 3D Gaussian unit. x ,u y ,u z The attribute information can be the scaling matrix s of the 3D Gaussian primitives, the rotation quaternion q, the opacity α, and the spherical harmonic coefficients SHs, etc.
[0160] It should be understood that, with the evolution of explicit field representation, the attribute information of 3D Gaussian primitives may include other information, and the position information may be further refined. However, as long as the deformation of 3D Gaussian primitives can be described, all such information should be included within the scope of protection of this application.
[0161] In one possible implementation, a dynamic 3D Gaussian splashing scheme based on a reference frame and deformation field can be used to determine the data of the reference frame of the dynamic scene, as well as the time-varying characteristics of multiple rendering units (i.e., 3D Gaussian units) in the dynamic scene. The time-varying characteristics of each rendering unit are used to describe the deformation field corresponding to the rendering unit in the current frame, that is, to describe the changes in the position information and / or attribute information of the rendering unit in the current frame relative to the reference frame.
[0162] As an example, suppose a dynamic scene can be described by N images. By analyzing and processing these N frames, M 3D Gaussian primitives are determined, and their deformation parameters are used to describe the changes in the dynamic scene. Since the scene content is dynamically changing, the deformation parameters of the M Gaussian primitives are different across multiple frames. In this case, each 3D Gaussian primitive corresponds to a time-varying feature, which describes the positional and / or attribute changes of that 3D Gaussian primitive in the current frame. Here, N and M are positive integers.
[0163] In some embodiments, the process of obtaining time-varying features and deformation indication information of multiple rendering units in a dynamic scene in step 401 can be as follows: obtaining time-varying features and deformation indication information based on a target neural network, wherein the target neural network is trained based on the position information of multiple rendering units in a reference frame and the image of the current frame.
[0164] In this case, the target neural network can also be encoded into a bitstream and sent to the second computer device along with the information in step 403 below, so that the second computer device can identify the target neural network with the same capabilities.
[0165] As an example, the target neural network can be a convolutional neural network (CNN), a recurrent neural network (RNN) and its variants (such as long short-term memory network (LSTM), gated recurrent unit (GRU)), graph neural network (GNN), MLP, etc., and the embodiments of this application do not limit this.
[0166] In one possible implementation, initial time-varying features and initial deformation indication information are set for multiple rendering units. These initial time-varying features and initial deformation indication information serve as network parameters of the initial neural network, and are updated during the training process of the initial neural network. After multiple rounds of training, the initial neural network at the end of training is determined as the target neural network, and the corresponding time-varying features and deformation indication information at the end of training are determined as the time-varying features and deformation indication information of multiple rendering units.
[0167] As an example, taking 3D Gaussian units as the rendering unit, in each training round, the input to the initial neural network includes the reference frame and the current frame. The output of the initial neural network is the deformation of these 3D Gaussian units in the multi-frame image, that is, their position and attribute changes relative to the reference frame. Based on the deformation predicted by the initial neural network and the reference frame, the predicted deformation can be applied to the 3D Gaussian units in the reference frame to simulate the position and attribute information of these 3D Gaussian units in the multi-frame target image. Based on the information of the deformed 3D Gaussian units, the prediction frame can be reconstructed. Then, combining the prediction frame and the current frame, the loss value of the initial neural network in the current round is determined, and based on this loss value, the backpropagation algorithm is used to update the parameters of the initial neural network, which include initial time-varying features and initial deformation indication information.
[0168] It should be understood that backpropagation determines how to adjust network parameters to reduce loss by calculating the gradient of the loss function with respect to the network parameters. Parameter updates, on the other hand, use optimization algorithms such as gradient descent to update the network parameters based on the calculated gradients.
[0169] The training process described above is an iterative one, in which the baseline frame and the current frame are input into the network multiple times, the loss is calculated, and the parameters are updated. As training progresses, the initial neural network can gradually learn how to more accurately predict the deformation of Gaussian units, thereby generating prediction frames that are closer to the current frame, thus achieving convergence.
[0170] In some embodiments, for multiple rendering units in a dynamic scene, the deformation indication information for each rendering unit includes position change indication information and / or attribute change indication information.
[0171] Among them, the position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0172] When the time-varying characteristics of a rendering unit are used to describe the change in the position information of the corresponding rendering unit in the current frame relative to the reference frame, the deformation indication information of that rendering unit includes position change indication information. When the time-varying characteristics of a rendering unit are used to describe the change in the attribute information of the corresponding rendering unit in the current frame relative to the reference frame, the deformation indication information of that rendering unit includes attribute change indication information.
[0173] However, it should be noted that when the time-varying characteristics of a rendering unit are used to describe the changes in the position and attribute information of the corresponding rendering unit in the current frame relative to the reference frame, the deformation indication information of the rendering unit may include only the position change indication information, or only the attribute change indication information, or both the position change indication information and the attribute change indication information.
[0174] That is, when the time-varying features of a rendering unit are used to describe the change of one piece of information of the corresponding rendering unit relative to the reference frame, the corresponding deformation indication information indicates whether that piece of information has changed relative to the reference frame. When the time-varying features of a rendering unit are used to describe the change of at least two pieces of information of the corresponding rendering unit relative to the reference frame, the embodiments of this application can flexibly set the corresponding deformation indication information for the at least two pieces of information, so as to compress the time-varying features of the rendering unit from at least two dimensions through the subsequent step 402.
[0175] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0176] As an example, the aforementioned position change indication information and / or attribute change indication information can be represented by masks. Specifically, position change indication information is represented by a position mask matrix, and attribute change indication information is represented by an attribute mask matrix. The position mask matrix includes multiple position mask vectors corresponding one-to-one with multiple rendering units. The position mask vector for each rendering unit indicates whether the position information of that rendering unit in the current frame has changed relative to the reference frame. Similarly, the attribute mask matrix includes multiple attribute mask vectors corresponding one-to-one with multiple rendering units. The attribute mask vector for each rendering unit indicates whether the attribute information of that rendering unit in the current frame image has changed relative to the reference frame.
[0177] Taking the position mask vector as an example, if there are P frames between the current frame and the reference frame, the position mask vector of the rendering unit can include P elements. The value of each element is used to indicate whether the position information of the rendering unit in the corresponding frame has changed relative to the reference frame.
[0178] Normally, a value of 1 in the mask indicates that the rendering unit has undergone deformation (the position information has changed relative to the reference frame, or the attribute information has changed relative to the reference frame), and a value of 0 indicates that the rendering unit has not undergone deformation (the position information has not changed relative to the reference frame, or the attribute information has not changed relative to the reference frame).
[0179] Step 402: Compress the time-varying features of multiple rendering units based on deformation indication information to obtain the target time-varying features.
[0180] Among them, the target time-varying feature is used to describe the change of target information of at least one target rendering unit. The target rendering unit refers to the rendering unit whose target information changes relative to the reference frame. The target information includes the position information and attribute information of the target rendering unit that have changed relative to the reference frame.
[0181] For ease of understanding, taking the 3D Gaussian ellipsoid (ellipsoid) as the basic unit of rendering (hereinafter referred to as the rendering unit) as introduced above as an example, we will first briefly introduce the reason why the technical solution of this application can use deformation indication information to compress time-varying features.
[0182] Each 3D Gaussian element passes through its center position μ = (u x ,u y ,u z The parameters, including the scaling matrix s, rotation quaternion q, opacity α, and spherical harmonic coefficients SHs, are defined and can be represented as G0(μ,s,q,α,SHs). For dynamic scenes, the 3D Gaussian elements at any time t can be represented by the deformation field F. t (G0) is used to describe it.
[0183] This application's embodiments divide the parameters of 3D Gaussian elements into two categories: position information (i.e., center position μ) and attribute information (parameters other than μ). By describing the position and attribute changes of 3D Gaussian elements, the deformation field of 3D Gaussian elements is fully described.
[0184] Based on this classification approach, taking the attribute information as scaling matrix and rotation quaternion as an example, such as... Figure 5 As shown, the deformation field of a 3D Gaussian element can be divided into the following four cases:
[0185] (1) The position of the 3D Gothic element remains unchanged, and the attributes of the 3D Gothic element remain unchanged;
[0186] (2) The position of the 3D Gothic element changes, but the properties of the 3D Gothic element remain unchanged;
[0187] (3) The position of the 3D Gaussian element remains unchanged, but the attributes of the 3D Gaussian element change.
[0188] (4) The position of the 3D Gaussian element changes, and the attributes of the 3D Gaussian element change.
[0189] Here, "position unchanged / changed" refers to whether the position information of the 3D Gaussian primitive at time t changes relative to the reference frame, and "attribute unchanged / changed" refers to whether the attribute information of the 3D Gaussian primitive at time t changes relative to the reference frame. In other words, based on the position information and / or attribute information of the 3D Gaussian primitive in the reference frame, we analyze whether this position information and / or attribute information changes over time.
[0190] Based on this, when the time-varying features of multiple rendering units include positional features and / or attribute features, taking the target rendering unit (any one of the multiple rendering units) as an example, the compression of the time-varying features of the target rendering unit in step 402 above includes the following three cases:
[0191] The first type: If the deformation indication information of the target rendering unit indicates that the attribute information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the attribute features of the target rendering unit.
[0192] The second type: If the deformation indication information of the target rendering unit indicates that the position information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the position feature of the target rendering unit.
[0193] The third type: If the deformation indication information of the target rendering unit indicates that the position information and attribute information of the target rendering unit in the current frame have not changed relative to the reference frame, then the time-varying feature of the target does not include the time-varying feature of the target rendering unit.
[0194] Correspondingly, when the time-varying features of the target rendering unit include positional features and / or attribute features, the technical solution of this application also includes three cases in encoding the time-varying features of multiple rendering units into the bitstream:
[0195] The first scenario is that if the position information of the target rendering unit relative to the reference frame changes, but the attribute information remains unchanged, then only the positional features of the target rendering unit are encoded into the bitstream.
[0196] The second method is as follows: if the position information of the target rendering unit relative to the reference frame remains unchanged, but the attribute information changes, then only the attribute features of the target rendering unit are encoded into the bitstream.
[0197] The third scenario is that the position information of the target rendering unit relative to the reference frame has changed, and the attribute information has also changed. In order to accurately describe the deformation of the target rendering unit in the current frame and ensure rendering quality, the time-varying features of the target rendering unit are not compressed. In other words, the complete time-varying features of the target rendering unit (including position features and / or attribute features, depending on the actual time-varying information of the target rendering unit) are encoded into the bitstream.
[0198] Step 403: Encode the data of the reference frame, deformation indication information, and target time-varying features into the bitstream.
[0199] In one possible implementation, encoding the deformation indication information of multiple rendering units into the bitstream may include the following steps 11-12:
[0200] Step 11: Based on the deformation indication information of multiple rendering units, obtain the first feature and / or the second feature through the first neural network.
[0201] The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame.
[0202] As an example, the first neural network can be a trained MLP network that learns the complex relationships between features through its internal nonlinear transformations, and directly extracts a more compact and representative feature representation (i.e., the first feature and the second feature) from the deformation indication information of multiple rendering units.
[0203] As another example, the first neural network may also include a trained first MLP network and a second MLP network. Deformation indication information from multiple rendering units is input into the first MLP network, and the first MLP network extracts a first feature that reflects positional changes. Similarly, deformation indication information from multiple rendering units is input into the second MLP network, and the second MLP network extracts a second feature that reflects attribute changes.
[0204] During the training of the first MLP network, deformation indication information is coupled with the time-varying features of the corresponding rendering units. This coupling can be achieved through feature multiplication or concatenation, aiming to enable the MLP network to simultaneously learn the changes in position and attributes. The coupled features are then fed into both the MLP network describing attribute changes and the MLP network describing position changes for training. Through optimization algorithms such as backpropagation and gradient descent, the MLP network gradually adjusts its internal parameters to minimize prediction errors, thereby learning a model that accurately describes changes in the position and attributes of rendering units.
[0205] In some embodiments, the first neural network may also be encoded into a bitstream and sent to a second computer device along with the information from step 403 below, so that the second computer device can determine the first neural network having the same capabilities.
[0206] In this way, the first neural network can efficiently represent and process the deformation indication information of multiple rendering units in a dynamic scene. It can not only retain the information related to dynamic scene reconstruction and rendering, but also reduce the amount of raw data for subsequent encoding by extracting key features.
[0207] Step 12: Encode the first feature and / or the second feature into the bitstream.
[0208] Since the first feature and / or the second feature are compressed, the amount of data encoded into the bitstream is further reduced compared to directly encoding the deformation indication information of multiple rendering units into the bitstream.
[0209] In some embodiments, the data of the reference frame, deformation indication information and target time-varying features can be encoded into a single bitstream; alternatively, they can be encoded into separate bitstreams, resulting in multiple bitstreams. In this case, it is necessary to establish a relationship between the multiple bitstreams to indicate that the information encoded in the multiple bitstreams is for the same dynamic scene. In this embodiment, the number of encoded bitstreams generated in step 403 is not limited.
[0210] In some embodiments, the information to be encoded (data of the reference frame, deformation indication information, and target time-varying features) can be coupled with various coding schemes to further compress the data volume and reduce bit rate and bandwidth pressure.
[0211] As an example, the data and deformation indication information of the reference frame can be encoded into the bitstream through quantization and image compression; the time-varying features of the target can be encoded into the bitstream through video encoding.
[0212] In summary, for dynamic scenes, this application introduces corresponding time-varying features for each rendering unit, enabling a more flexible and comprehensive description of the deformation of each rendering unit. Simultaneously, it introduces corresponding deformation indication information for each rendering unit to intuitively reflect whether the position and attribute information of each rendering unit in the current frame has changed relative to the reference frame. Thus, by using time-varying features and deformation indication information, the deformation of each rendering unit can be described in detail, improving rendering quality. When storing or encoding scene representation data for dynamic scenes, deformation indication information is used to compress the time-varying features of multiple rendering units to obtain target time-varying features. Therefore, only the data of the reference frame of the dynamic scene, the target time-varying features, and the deformation indication information of multiple rendering units are stored / encoded, without needing to store the complete time-varying features of each rendering unit. This reduces the storage space requirements for storing time-varying features and the bandwidth requirements for transmitting them. Thus, based on reference frame data, deformation indication information, and target time-varying characteristics, the embodiments of this application can not only accurately describe the details of the dynamic scene to ensure the quality of reconstructing / rendering the dynamic scene, but also reduce the requirements for storage space and transmission bandwidth, achieving the effect of low storage overhead and high reconstruction quality at the same time.
[0213] The decoding method for dynamic scenes provided in the embodiments of this application will be described in detail below.
[0214] Figure 6 This is a flowchart illustrating a dynamic scene encoding method provided in an embodiment of this application. Exemplarily, this method can be applied to a second computer device. Please refer to... Figure 6 The method includes the following steps.
[0215] Step 601: Parse the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features from the bitstream. The reference frame data includes the position information and attribute information of multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame.
[0216] Among them, dynamic scenes refer to three-dimensional scenes that change over time.
[0217] Taking the dynamic 3D Gaussian splashing scheme as an example, the rendering unit in this embodiment can specifically be a 3D Gaussian element. In this case, the position information of the rendering unit can be the center position μ of the 3D Gaussian element μ = (u x ,u y ,u z The attribute information can be the scaling matrix s of the 3D Gaussian primitives, the rotation quaternion q, the opacity α, and the spherical harmonic coefficients SHs, etc.
[0218] In one possible implementation, the process of parsing deformation indication information of multiple rendering units in a dynamic scene from the bitstream can be as follows: parsing a first feature and / or a second feature from the bitstream, wherein the first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame; and obtaining the deformation indication information of multiple rendering units through a first neural network based on the first feature and / or the second feature.
[0219] The data of the first neural network can also be sent from the first computer device to the second computer device via an encoded bitstream, so that the second computer device can reconstruct / determine the first neural network with the same function based on the data of the first neural network.
[0220] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0221] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0222] As an example, position change indication information and attribute change indication information can be represented by masks. Specifically, position change indication information is represented by a position mask matrix, and attribute change indication information is represented by an attribute mask matrix. The position mask matrix includes multiple position mask vectors corresponding one-to-one with multiple rendering units. The position mask vector for each rendering unit indicates whether the position information of that rendering unit in the current frame has changed relative to the reference frame. Similarly, the attribute mask matrix includes multiple attribute mask vectors corresponding one-to-one with multiple rendering units. The attribute mask vector for each rendering unit indicates whether the attribute information of that rendering unit in the current frame has changed relative to the reference frame.
[0223] Step 602: Based on deformation indication information and target time-varying features, determine the time-varying features of multiple rendering units. The time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering units relative to the reference frame.
[0224] As explained earlier, if the time-varying features of the target rendering unit are not compressed during encoding, the time-varying features of the target rendering unit can be directly obtained after parsing, without needing to combine them with deformation indication information for recovery. For rendering units whose time-varying features are compressed during encoding, since the target time-varying features included in the bitstream only include features related to position information or attribute information, i.e., the time-varying features of the target rendering unit are incomplete, the time-varying features not included in the bitstream can be recovered after decoding by combining them with deformation indication information.
[0225] If the position and attribute information of the rendering unit in the current frame does not change relative to the reference frame, then the time-varying features of these rendering units are not encoded in the target time-varying features. In this case, it is necessary to combine the deformation indication information to determine the time-varying features of these rendering units.
[0226] It should be noted that when the position and attribute information of a rendering unit remains unchanged relative to the reference frame, the parameter values in its corresponding time-varying features can be set to 0, or represented by 0, or in other words, its time-varying features are empty. In other words, the second computer device needs to combine deformation indication information and target time-varying features to determine the rendering units with empty time-varying features among multiple rendering units.
[0227] Step 603: Based on the data of the reference frame and the time-varying characteristics of multiple rendering units, determine the position information and attribute information of multiple rendering units in the current frame.
[0228] In one possible implementation, step 603 may include the following steps 21-22:
[0229] Step 21: Input the data of the reference frame and the time-varying features of multiple rendering units into the second neural network to obtain the position change and / or attribute change of multiple rendering units in the current frame, respectively, as output by the second neural network.
[0230] It should be understood that the output of the second neural network is the change in position and / or the change in attribute, which corresponds to the time-varying features of multiple rendering units, including position features and / or attribute features.
[0231] Among them, the position change indicates the amount of change in the position information of the rendering unit in the current frame relative to the reference frame, and the attribute change indicates the amount of change in the attribute information of the rendering unit in the current frame relative to the reference frame.
[0232] As an example, the second neural network is a pre-trained MLP network that learns and analyzes the position information, attribute information and time-varying features of each rendering unit in the reference frame through its internal nonlinear transformation, and then determines the change in the position information and / or attribute information of each rendering unit in the current frame.
[0233] In some embodiments, the first computer device may also encode the network parameters of the second neural network into a bitstream and send it to the second computer device, which can then parse the corresponding second neural network from the bitstream. This application does not impose any limitations on this.
[0234] In some embodiments, the time-varying features of the multiple rendering units include position features and attribute features, and the second neural network mentioned above includes a position analysis subnetwork and an attribute analysis subnetwork.
[0235] In this case, the implementation process of step 21 can be as follows: inputting the position information of multiple rendering units in the reference frame and the position features of multiple rendering units into the position analysis sub-network to obtain the position change amount of multiple rendering units in multiple frames of images output by the position analysis sub-network; and / or, inputting the attribute information of multiple rendering units in the reference frame and the attribute features of multiple rendering units into the attribute analysis sub-network to obtain the attribute change amount of multiple rendering units in multiple frames of images output by the attribute analysis sub-network.
[0236] As an example, the location analysis subnetwork and the attribute analysis subnetwork can also be implemented using an MLP network, and this application embodiment does not limit this.
[0237] In some embodiments, considering that the scaling matrix s and rotation quaternion q in the attribute information also affect the position information of 3D Gaussian primitives, when inferring the position changes of multiple rendering units in multiple frames of images through the position analysis subnetwork, the position information and attribute information of multiple rendering units in the reference frame, as well as the position features of multiple rendering units, can be simultaneously input into the position analysis subnetwork to obtain the position changes of multiple rendering units in multiple frames of images output by the position analysis subnetwork. This improves the accuracy of the position changes output by the position analysis subnetwork.
[0238] Step 22: Based on the data of the reference frame, and the position and attribute changes of multiple rendering units in the current frame, determine the position information and / or attribute information of multiple rendering units in the current frame.
[0239] In one possible implementation, the position information of the rendering unit in the reference frame is adjusted based on the change in its position in the current frame to obtain its position information in the current frame. Similarly, the attribute information of the rendering unit in the reference frame is adjusted based on the change in its attribute in the current frame to obtain its attribute information in the current frame.
[0240] In some embodiments, considering that the scaling matrix s and rotation quaternion q in the attribute information also affect the position information of the 3D Gaussian unit, the position information of the rendering unit in multiple frames can be comprehensively determined by combining the data of the reference frame, the position change amount and attribute change amount of the rendering unit in the current frame, so as to improve the accuracy of the position information.
[0241] In summary, by encoding and transmitting the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features, the bandwidth requirements for transmitting the time-varying features of multiple rendering units in the dynamic scene are greatly reduced. After parsing the deformation indication information and target time-varying features of multiple rendering units from the bitstream, the time-varying features of each rendering unit can be recovered based on the deformation indication information. Then, combined with the data of the reference frame of the dynamic scene, the position information and attribute information of each rendering unit in the current frame can be accurately determined. Thus, the embodiments of this application, based on the data of the reference frame, deformation indication information, and target time-varying features, can not only accurately describe the details of the dynamic scene to ensure the quality of reconstructing / rendering the dynamic scene, but also reduce the requirements for transmission bandwidth, achieving the effect of low storage overhead and high reconstruction quality simultaneously.
[0242] Regarding the embodiments of the encoding and decoding methods for the aforementioned dynamic scenes, it should be noted that... Figure 6 The decoding method for the dynamic scene shown is similar to... Figure 4 The encoding method for the dynamic scene shown is corresponding, and the meanings of the related technical features are similar. Their specific meanings and related implementation logic can also be found in the explanation of the encoding method, which will not be elaborated here.
[0243] The following section describes the encoding and decoding apparatus for dynamic scenes provided in the embodiments of this application.
[0244] Figure 7 This is a schematic diagram of the structure of a dynamic scene encoding device provided in an embodiment of this application. The dynamic scene encoding device can be implemented as part or all of a first computer device by software, hardware, or a combination of both. See also... Figure 7 The encoding device for this dynamic scene includes: an acquisition module 701, a compression module 702, and an encoding module 703.
[0245] The acquisition module 701 is used to acquire data from the reference frame of the dynamic scene, time-varying features of multiple rendering units in the current frame, and deformation indication information. The reference frame data includes the position and attribute information of multiple rendering units within the reference frame. The time-varying features describe the changes in the position and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. The deformation indication information indicates whether the position and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. For detailed implementation details, please refer to [link to implementation details]. Figure 4 The description of step 401 in the illustrated embodiment will not be repeated here.
[0246] Compression module 702 is used to compress the time-varying features of multiple rendering units based on deformation indication information to obtain the target time-varying features. For detailed implementation details, please refer to [reference needed]. Figure 4 The description of step 402 in the illustrated embodiment will not be repeated here.
[0247] Encoding module 703 is used to encode the data from the reference frame, deformation indication information, and target time-varying features into the bitstream. For detailed implementation details, please refer to [reference needed]. Figure 4 The relevant description of step 403 in the illustrated embodiment will not be repeated here.
[0248] In one possible implementation, the acquisition module 701 is specifically used for:
[0249] The target neural network is used to obtain time-varying features and deformation indication information. The target neural network is trained based on the position information of multiple rendering units in the reference frame and the image of the current frame.
[0250] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0251] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0252] In one possible implementation, the encoding module 703 is specifically used for:
[0253] Based on the deformation indication information of multiple rendering units, a first feature and / or a second feature are obtained through a first neural network. The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame.
[0254] The first feature and / or the second feature are encoded into the bitstream.
[0255] In one possible implementation, the time-varying features of multiple rendering units include positional features and / or attribute features; the compression module 702 is specifically used for:
[0256] If the deformation indication information of the target rendering unit indicates that the attribute information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the attribute feature of the target rendering unit, and the target rendering unit is any one of the multiple rendering units.
[0257] If the deformation indication information of the target rendering unit indicates that the position information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the position feature of the target rendering unit.
[0258] If the deformation indication information of the target rendering unit indicates that the position and attribute information of the target rendering unit in the current frame have not changed relative to the reference frame, then the time-varying features of the target do not include the time-varying features of the target rendering unit.
[0259] In one possible implementation, the rendering unit is a 3D Gothic unit.
[0260] In one possible implementation, a dynamic scene refers to a three-dimensional scene that changes over time.
[0261] In this embodiment, for dynamic scenes, a corresponding time-varying feature is introduced for each rendering unit, enabling a more flexible and comprehensive description of the deformation of each rendering unit. Simultaneously, corresponding deformation indication information is introduced for each rendering unit to intuitively reflect whether the position and attribute information of each rendering unit in the current frame has changed relative to the reference frame. Thus, by using time-varying features and deformation indication information, the deformation of each rendering unit can be described in detail, improving rendering quality. When storing or encoding scene representation data for dynamic scenes, the time-varying features of multiple rendering units are compressed using deformation indication information to obtain target time-varying features. Therefore, only the data of the reference frame of the dynamic scene, the target time-varying features, and the deformation indication information of multiple rendering units are stored / encoded, without needing to store the complete time-varying features of each rendering unit. This reduces the storage space requirements for storing time-varying features and the bandwidth requirements for transmitting them. Thus, based on reference frame data, deformation indication information, and target time-varying characteristics, the embodiments of this application can not only accurately describe the details of the dynamic scene to ensure the quality of reconstructing / rendering the dynamic scene, but also reduce the requirements for storage space and transmission bandwidth, achieving the effect of low storage overhead and high reconstruction quality at the same time.
[0262] It should be noted that the encoding device for dynamic scenes provided in the above embodiments is only illustrated by the division of the functional modules described above when encoding the data for reconstructing the dynamic scene. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the encoding device for dynamic scenes provided in the above embodiments and... Figure 4 The encoding method embodiments for the dynamic scenes shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0263] Figure 8 This is a schematic diagram of the structure of a dynamic scene decoding device provided in an embodiment of this application. This dynamic scene decoding device can be implemented as part or all of a second computer device by software, hardware, or a combination of both. See also... Figure 8 The decoding device for the dynamic scene includes: a decoding module 801, a first determining module 802, and a second determining module 803.
[0264] Decoding module 801 is used to parse the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the time-varying features of the target from the bitstream. The data of the reference frame includes the position information and attribute information of multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. For detailed implementation process, please refer to [reference needed]. Figure 6 The relevant description of step 601 in the illustrated embodiment will not be repeated here.
[0265] The first determining module 802 is used to determine the time-varying characteristics of multiple rendering units based on deformation indication information and target time-varying characteristics. The time-varying characteristics describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. For detailed implementation details, please refer to [link to implementation details]. Figure 6 The relevant description of step 602 in the illustrated embodiment will not be repeated here.
[0266] The second determining module 803 is used to determine the position information and / or attribute information of multiple rendering units in the current frame based on the data of the reference frame and the time-varying characteristics of multiple rendering units. For detailed implementation details, please refer to [link / reference needed]. Figure 6 The relevant description of step 603 in the illustrated embodiment will not be repeated here.
[0267] In one possible implementation, the deformation indication information of each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0268] In one possible implementation, the deformation indication information of each rendering unit is represented by a deformation mask, and the value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
[0269] In one possible implementation, the decoding module 801 is specifically used for:
[0270] The first feature and / or the second feature are parsed from the bitstream. The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame.
[0271] Deformation indication information is obtained through a first neural network based on the first feature and / or the second feature.
[0272] In one possible implementation, the second determining module 803 includes:
[0273] The first determining unit is used to input the data of the reference frame and the time-varying features of multiple rendering units into the second neural network to obtain the position change and / or attribute change of multiple rendering units in the current frame, respectively, as output by the second neural network. The position change indicates the change of the position information of the rendering unit in the current frame relative to the reference frame, and the attribute change indicates the change of the attribute information of the rendering unit in the current frame relative to the reference frame.
[0274] The second determining unit is used to determine the position information and / or attribute information of multiple rendering units in the current frame based on the data of the reference frame and the position change and / or attribute change of multiple rendering units in the current frame.
[0275] In one possible implementation, the time-varying features of multiple rendering units include positional features and / or attribute features, and the second neural network includes a positional analysis subnetwork and / or an attribute analysis subnetwork; the first determining unit is specifically used for:
[0276] The position information or position information and attribute information of multiple rendering units in the reference frame, as well as the position features of multiple rendering units, are input into the position analysis sub-network to obtain the position change of each rendering unit in the current frame relative to the reference frame, as output by the position analysis sub-network; and / or
[0277] The attribute information of multiple rendering units in the reference frame and the attribute features of multiple rendering units are input into the attribute analysis sub-network to obtain the attribute change of multiple rendering units in the current frame relative to the reference frame.
[0278] In one possible implementation, the rendering unit is a 3D Gothic unit.
[0279] In one possible implementation, a dynamic scene refers to a three-dimensional scene that changes over time.
[0280] In this embodiment, by encoding and transmitting the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features, the bandwidth requirement for transmitting the time-varying features of multiple rendering units in the dynamic scene is greatly reduced. After parsing the deformation indication information and target time-varying features of multiple rendering units from the bitstream, the time-varying features of each rendering unit can be recovered based on the deformation indication information. Then, combined with the data of the reference frame of the dynamic scene, the position information and attribute information of each rendering unit in the current frame can be accurately determined. Thus, this embodiment, based on the data of the reference frame, deformation indication information, and target time-varying features, can not only accurately describe the details of the dynamic scene to ensure the quality of reconstructing / rendering the dynamic scene, but also reduce the requirement for transmission bandwidth, achieving the effect of low storage overhead and high reconstruction quality simultaneously.
[0281] It should be noted that the dynamic scene decoding device provided in the above embodiments, when parsing the bitstream to determine the data for reconstructing the dynamic scene, is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the dynamic scene decoding device provided in the above embodiments and... Figure 6 The decoding method embodiments of the dynamic scenes shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0282] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to execute the steps of the dynamic scene encoding method shown in the above embodiments, or to execute the steps of the dynamic scene decoding method shown in the above embodiments.
[0283] This application also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the steps of the dynamic scene encoding method shown in the above embodiments, or the steps of the dynamic scene decoding method shown in the above embodiments.
[0284] This application also provides a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the dynamic scene encoding method shown in the above embodiments, or to perform the steps of the dynamic scene decoding method shown in the above embodiments.
[0285] This application also provides an encoding and decoding system, which includes an encoding device and a decoding device. The encoding device is used to implement the steps of the encoding method for the dynamic scene shown in the above embodiments, and the decoding device is used to implement the steps of the decoding method for the dynamic scene shown in the above embodiments.
[0286] This application embodiment also provides an encoded bitstream, which is generated according to the encoding method for the dynamic scene shown in the above embodiments.
[0287] This application embodiment also provides an encoded bitstream, which includes reference frame data of a dynamic scene, deformation indication information of multiple rendering units in the dynamic scene, and target time-varying features. The reference frame data includes position information and attribute information of multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the multi-frame images has changed relative to the reference frame. The multi-frame images are used to describe the dynamic scene. The target time-varying features are obtained by compressing the time-varying features of multiple rendering units based on the deformation indication information. The time-varying features are used to describe the changes in the position information and attribute information of the corresponding rendering unit in the multi-frame images relative to the reference frame.
[0288] This application also provides a computer-readable storage medium storing a bitstream generated according to the encoding method for the dynamic scene shown in the above embodiments.
[0289] This application also provides an apparatus for storing a bitstream, the apparatus including a receiver and at least one storage medium; the receiver is used to receive a bitstream generated according to the encoding method of the dynamic scene shown in the above embodiments; at least one storage medium is used to store the bitstream.
[0290] This application also provides an apparatus for transmitting a bitstream, which includes a transmitter and a receiver; the receiver is used to receive a bitstream generated according to the encoding method of the dynamic scene shown in the above embodiments; the transmitter is used to send the bitstream to an end-side device through a transmission medium.
[0291] This application also provides an apparatus for transmitting a bitstream, the apparatus including a transmitter and at least one storage medium; the at least one storage medium is used to store a bitstream generated by the encoding method of the dynamic scene shown in the above embodiments; the transmitter is used to obtain the bitstream from the storage medium and transmit the bitstream to the end-side device through the transmission medium.
[0292] This application also provides a system for distributing bitstreams, the system comprising:
[0293] At least one storage medium is used to store the bitstream generated by the encoding method for the dynamic scene shown in the above embodiments;
[0294] A streaming media device for acquiring a target bitstream from the at least one storage medium and sending the target bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server.
[0295] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.
[0296] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.
[0297] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0298] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for encoding dynamic scenes, characterized in that, The method includes: The data of a reference frame in a dynamic scene, time-varying features of multiple rendering units in the current frame, and deformation indication information are obtained. The data of the reference frame includes the position information and attribute information of the multiple rendering units in the reference frame. The time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. Based on the deformation indication information, the time-varying features of the multiple rendering units are compressed to obtain the target time-varying features; The data of the reference frame, the deformation indication information, and the target time-varying features are encoded into the bitstream.
2. The method as described in claim 1, characterized in that, The step of obtaining time-varying features and deformation indication information of multiple rendering units in the dynamic scene includes: The time-varying features and deformation indication information are obtained based on the target neural network, which is trained based on the position information of the multiple rendering units in the reference frame and the image of the current frame.
3. The method as described in claim 1 or 2, characterized in that, The deformation indication information for each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
4. The method according to any one of claims 1-3, characterized in that, The deformation indication information of each rendering unit is represented by a deformation mask. The value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
5. The method according to any one of claims 1-4, characterized in that, The step of encoding the deformation indication information into the bitstream includes: Based on the deformation indication information of the plurality of rendering units, a first feature and / or a second feature are obtained through a first neural network. The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame. The first feature and / or the second feature are encoded into the bitstream.
6. The method according to any one of claims 1-5, characterized in that, The time-varying features of the multiple rendering units include position features and / or attribute features; The step of compressing the time-varying features of the multiple rendering units based on the deformation indication information to obtain the target time-varying features includes: If the deformation indication information of the target rendering unit indicates that the attribute information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the attribute feature of the target rendering unit, and the target rendering unit is any one of the plurality of rendering units. If the deformation indication information of the target rendering unit indicates that the position information of the target rendering unit in the current frame has not changed relative to the reference frame, then the target time-varying feature does not include the position feature of the target rendering unit; If the deformation indication information of the target rendering unit indicates that the position information and attribute information of the target rendering unit in the current frame have not changed relative to the reference frame, then the target time-varying feature does not include the time-varying feature of the target rendering unit.
7. The method according to any one of claims 1-6, characterized in that, The rendering unit is a 3D Gaussian unit.
8. The method according to any one of claims 1-7, characterized in that, The dynamic scene refers to a three-dimensional scene that changes over time.
9. A decoding method for dynamic scenes, characterized in that, The method includes: The data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features are parsed from the bitstream. The data of the reference frame includes the position information and attribute information of the multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. Based on the deformation indication information and the target time-varying features, the time-varying features of the plurality of rendering units are determined. The time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. Based on the data of the reference frame and the time-varying characteristics of the plurality of rendering units, the position information and / or attribute information of the plurality of rendering units in the current frame are determined respectively.
10. The method as described in claim 9, characterized in that, The deformation indication information for each rendering unit includes position change indication information and / or attribute change indication information. The position change indication information is used to indicate whether the position information of the rendering unit in the current frame has changed relative to the reference frame, and the attribute change indication information is used to indicate whether the attribute information of the rendering unit in the current frame has changed relative to the reference frame.
11. The method as described in claim 9 or 10, characterized in that, The deformation indication information of each rendering unit is represented by a deformation mask. The value of each deformation mask is used to indicate whether the position information and / or attribute information of the rendering unit in the current frame has changed relative to the reference frame.
12. The method according to any one of claims 9-11, characterized in that, The step of parsing deformation indication information of multiple rendering units in the dynamic scene from the bitstream includes: The first feature and / or the second feature are parsed from the bitstream. The first feature is used to characterize whether the position information of each rendering unit in the current frame has changed relative to the reference frame, and the second feature is used to characterize whether the attribute information of each rendering unit in the current frame has changed relative to the reference frame. Based on the first feature and / or the second feature, the deformation indication information is obtained through a first neural network.
13. The method according to any one of claims 9-12, characterized in that, The determination of the position information and / or attribute information of the multiple rendering units in the current frame based on the data of the reference frame and the time-varying characteristics of the multiple rendering units includes: The data of the reference frame and the time-varying features of the plurality of rendering units are input into the second neural network to obtain the position change and / or attribute change of the plurality of rendering units in the current frame, respectively, as output by the second neural network. The position change indicates the change of the position information of the rendering unit in the current frame relative to the reference frame, and the attribute change indicates the change of the attribute information of the rendering unit in the current frame relative to the reference frame. Based on the data of the reference frame, and the position change and / or attribute change of the plurality of rendering units in the current frame respectively, the position information and / or attribute information of the plurality of rendering units in the current frame are determined.
14. The method as described in claim 13, characterized in that, The time-varying features of the plurality of rendering units include position features and / or attribute features, and the second neural network includes a position analysis subnetwork and / or an attribute analysis subnetwork. The step of inputting the data of the reference frame and the time-varying features of the plurality of rendering units into the second neural network to obtain the position change and / or attribute change of the plurality of rendering units in the current frame, respectively, as output by the second neural network, includes: The position information or position information and attribute information of the plurality of rendering units in the reference frame, as well as the position features of the plurality of rendering units, are input into the position analysis sub-network to obtain the position change of the plurality of rendering units in the current frame relative to the reference frame, respectively, output by the position analysis sub-network; and / or The attribute information and attribute features of the multiple rendering units in the reference frame are input into the attribute analysis sub-network to obtain the attribute change amount of the current frame relative to the reference frame for each of the multiple rendering units, as output by the attribute analysis sub-network.
15. The method according to any one of claims 9-14, characterized in that, The rendering unit is a 3D Gaussian unit.
16. The method according to any one of claims 9-15, characterized in that, The dynamic scene refers to a three-dimensional scene that changes over time.
17. An encoding device for dynamic scenes, characterized in that, The device includes: The acquisition module is used to acquire data of a reference frame of a dynamic scene, time-varying features and deformation indication information of multiple rendering units in the current frame. The data of the reference frame includes the position information and attribute information of the multiple rendering units in the reference frame. The time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. A compression module is used to compress the time-varying features of the multiple rendering units based on the deformation indication information to obtain the target time-varying features; The encoding module is used to encode the data of the reference frame, the deformation indication information, and the target time-varying features into the bitstream.
18. A decoding device for dynamic scenes, characterized in that, The device includes: The decoding module is used to parse the data of the reference frame of the dynamic scene, the deformation indication information of multiple rendering units in the dynamic scene, and the target time-varying features from the bitstream. The data of the reference frame includes the position information and attribute information of the multiple rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the current frame has changed relative to the reference frame. The first determining module is used to determine the time-varying features of the plurality of rendering units based on the deformation indication information and the target time-varying features, wherein the time-varying features are used to describe the changes in the position information and / or attribute information of the corresponding rendering unit in the current frame relative to the reference frame; The second determining module is used to determine the position information and / or attribute information of the multiple rendering units in the current frame based on the data of the reference frame and the time-varying characteristics of the multiple rendering units.
19. A computer device, characterized in that, The computer device includes a processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method of any one of claims 1-8, or to implement the method of any one of claims 9-16.
20. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-8, or the method described in any one of claims 9-16.
21. A computer program product, characterized in that, The computer program product stores computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-8, or the method described in any one of claims 9-16.
22. An encoded bitstream, characterized in that, The bitstream includes reference frame data of the dynamic scene, deformation indication information of multiple rendering units in the dynamic scene, and time-varying features of the target. The reference frame data includes the position information and attribute information of the plurality of rendering units in the reference frame. The deformation indication information is used to indicate whether the position information and / or attribute information of the corresponding rendering unit in the multi-frame images has changed relative to the reference frame. The multi-frame images are used to describe the dynamic scene. The target time-varying feature is obtained by compressing the time-varying features of the plurality of rendering units based on the deformation indication information. The time-varying feature is used to describe the changes in the position information and attribute information of the corresponding rendering unit in the multi-frame images relative to the reference frame.
