Dynamic scene volume video streaming method, apparatus, and storage medium

Abstract

The application relates to a dynamic scene volume video streaming method, device and storage medium. The dynamic scene volume video streaming method comprises the following steps: acquiring a dynamic scene video sequence, separating a starting frame of the dynamic scene video sequence into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged in the whole video sequence; for each video frame after the starting frame in the dynamic scene video sequence, the dynamic Gaussian set of each video frame is generated frame by frame based on the motion transformation of the dynamic Gaussian set of the previous video frame, and each video frame dynamic Gaussian set is compensated and optimized to obtain a complete dynamic scene of the dynamic scene video sequence; and the complete dynamic scene is compressed to generate a bit stream for transmission. By decoupling the dynamic scene into a static background and a dynamic foreground, storing the static background, and performing inter-frame motion prediction and compensation on the dynamic foreground, the coding data amount of the volume video is reduced, and the dynamic scene volume video streaming efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of dynamic volume video stream compression and transmission processing technology, and in particular to dynamic scene volume video stream transmission methods, apparatus and storage media. Background Technology

[0002] Dynamic scene volumetric video streaming refers to encoding and transmitting the motion information of people and objects in a real scene in three-dimensional space as a video stream to a remote location, allowing users to view and interact from any perspective. Volumetric video streaming has broad application prospects in scenarios such as immersive telepresence, virtual reality, digital cultural heritage preservation, and free-viewpoint replay of sports events.

[0003] Currently, the mainstream technical approach for dynamic scene volumetric video is based on neural radiation fields and their variants for scene modeling and encoding. Neural radiation field methods can achieve high-fidelity synthesis of new perspectives through implicit or explicit 3D representations. However, existing volumetric video coding methods typically model and encode the entire dynamic scene, resulting in a large amount of information being repeatedly encoded and stored in each frame. This leads to significant redundancy in the overall encoded data, low compression efficiency, and low transmission efficiency for dynamic scene volumetric video streams.

[0004] There is currently no effective solution to the problem of low transmission efficiency of dynamic scene volume video streams in related technologies. Summary of the Invention

[0005] This embodiment provides a method, apparatus, and storage medium for transmitting dynamic scene volume video streams to solve the problem of low transmission efficiency of dynamic scene volume video streams in related technologies.

[0006] Firstly, this embodiment provides a method for transmitting dynamic scene volume video streams, including:

[0007] A dynamic scene video sequence is obtained, and the starting frame of the dynamic scene video sequence is separated into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the entire video sequence.

[0008] For each video frame after the starting frame in the dynamic scene video sequence, the dynamic Gaussian set of each video frame is generated frame by frame based on the motion transformation of the dynamic Gaussian set of the previous video frame, and the dynamic Gaussian set of each video frame is compensated and optimized to obtain the complete dynamic scene of the dynamic scene video sequence.

[0009] The complete dynamic scene is compressed to generate a bitstream for transmission.

[0010] In some embodiments, generating a dynamic Gaussian set for each video frame after the starting frame in the dynamic scene video sequence, based on the motion transform of the dynamic Gaussian set of the previous video frame, frame by frame includes:

[0011] Each video frame after the starting frame in the dynamic scene video sequence is taken as the target frame.

[0012] The dynamic Gaussian centers of the preceding video frame of the target frame are position-encoded. Based on the position encoding results, interpolation sampling is performed on the multi-resolution motion grid corresponding to the target frame. The motion description of the target frame is obtained based on the interpolation sampling results. When the target frame is the second video frame, the preceding video frame of the target frame is the starting frame. The multi-resolution motion grid is used to represent the motion information of Gaussian points at different spatial resolutions.

[0013] The motion description is input into a multilayer perceptron network, which outputs the translation and rotation increments of each dynamic Gaussian from the previous video frame to the target frame.

[0014] The translation and rotation increments are applied to the dynamic Gaussian set of the previous video frame of the target frame to generate the dynamic Gaussian set of the target frame.

[0015] In some embodiments, the step of position-encoding the dynamic Gaussian centers of the preceding video frame of the target frame, performing interpolation sampling based on the position-encoding results in the multi-resolution motion grid corresponding to the target frame, and obtaining the motion description of the target frame based on the interpolation sampling results includes:

[0016] Obtain the dynamic Gaussian set of the previous video frame of the target frame, and extract the center position of each dynamic Gaussian in the dynamic Gaussian set;

[0017] Multi-scale position encoding is performed on each dynamic Gaussian center location to generate multi-scale position embeddings of each dynamic Gaussian center at different spatial frequencies.

[0018] A multi-resolution motion mesh is constructed for the target frame; wherein the multi-resolution motion mesh contains multiple resolution layers, and each resolution layer stores motion features at different spatial resolutions;

[0019] Trilinear interpolation is performed on the motion mesh of the multi-scale position embedded in each resolution layer to extract the motion features of the dynamic Gaussian center position at each resolution layer.

[0020] The motion features extracted by the dynamic Gaussian at each resolution layer are spliced ​​together to form the motion description of the dynamic Gaussian.

[0021] Summarize the motion descriptions of all dynamic Gaussians to obtain the motion description of the target frame.

[0022] In some embodiments, the compensation optimization of the dynamic Gaussian set for each video frame includes:

[0023] Obtain the dynamic Gaussian set of the current target frame, perform compensation detection on the dynamic Gaussian set, and obtain the compensated Gaussian set;

[0024] The compensated Gaussian set is merged with the dynamic Gaussian set to generate the complete dynamic Gaussian set of the target frame.

[0025] In some embodiments, obtaining the dynamic Gaussian set of the current target frame and performing compensation detection on the dynamic Gaussian set to obtain a compensated Gaussian set includes:

[0026] Detect the gradient values ​​of each dynamic Gaussian in the dynamic Gaussian set at its projection position in the image space;

[0027] The dynamic Gaussian corresponding to the pixel whose gradient value exceeds the preset gradient threshold is used as the appearance compensation Gaussian and copied to the compensation Gaussian set.

[0028] In some embodiments, obtaining the dynamic Gaussian set of the current target frame and performing compensation detection on the dynamic Gaussian set to obtain a compensated Gaussian set includes:

[0029] Extract the translation and rotation increments of each dynamic Gaussian in the dynamic Gaussian set;

[0030] When the absolute value of the translation increment of the dynamic Gaussian exceeds the first preset threshold, and / or the absolute value of the rotation increment exceeds the second preset threshold, it is determined that the dynamic Gaussian has undergone significant motion.

[0031] For a dynamic Gaussian that undergoes significant motion, a Gaussian splitting operation is performed in the local neighborhood of the dynamic Gaussian to obtain a reduced sub-Gaussian. The sub-Gaussian is then copied as a motion-compensated Gaussian to the compensation Gaussian set.

[0032] In some embodiments, the dynamic scene volume video stream transmission method further includes:

[0033] Obtain the compensated Gaussian set and the dynamic Gaussian set of the current target frame, and calculate the ratio of the number of the compensated Gaussian set to the number of the dynamic Gaussian set;

[0034] When the quantity ratio exceeds a preset quantity ratio threshold, it is determined that the scene under the target frame has changed significantly, and a new image group is started; the new image group uses the static Gaussian set of the previous video frame of the target frame to initialize the static background of the new image group, and retains the dynamic Gaussian set of the previous image group.

[0035] In some embodiments, compressing the complete dynamic scene to generate a bitstream for transmission includes:

[0036] Entropy coding is performed on the multi-resolution motion mesh, and the data distribution of the multi-resolution motion mesh and the entropy-coded data are transmitted.

[0037] The static Gaussian set, the dynamic Gaussian set of the starting frame, and the compensated Gaussian set are compressed using a point cloud compression framework. The spatial position, scale, rotation, opacity, and spherical harmonic coefficient of each Gaussian are quantized and entropy encoded as independent attribute channels to generate a Gaussian compressed bitstream.

[0038] Secondly, this embodiment provides a dynamic scene volume video stream transmission device, including: a separation description module, a prediction generation module, and a generation transmission module; wherein:

[0039] The separation description module is used to acquire a dynamic scene video sequence and separate the starting frame of the dynamic scene video sequence into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the entire video sequence.

[0040] The prediction generation module is used to generate a dynamic Gaussian set for each video frame after the starting frame in the dynamic scene video sequence, based on the motion transformation of the dynamic Gaussian set of the previous video frame, and to compensate and optimize the dynamic Gaussian set of each video frame to obtain the complete dynamic scene of the dynamic scene video sequence.

[0041] The generation and transmission module is used to compress the complete dynamic scene and generate a bit stream for transmission.

[0042] Thirdly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the steps of the dynamic scene volume video stream transmission method described in the first aspect.

[0043] Compared with related technologies, the dynamic scene volumetric video stream transmission method, apparatus, and storage medium provided in this embodiment are as follows: First, a dynamic scene video sequence is acquired, and the starting frame of the dynamic scene video sequence is separated into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the video sequence. Second, for each video frame after the starting frame in the dynamic scene video sequence, a dynamic Gaussian set for each video frame is generated frame-by-frame based on the motion transformation of the dynamic Gaussian set of the previous video frame, and the dynamic Gaussian set of each video frame is compensated and optimized to obtain the complete dynamic scene of the dynamic scene video sequence. Finally, the complete dynamic scene is compressed to generate a bitstream for transmission. By decoupling the dynamic scene into a static background and a dynamic foreground, storing the static background, and using a multi-resolution motion mesh and a multilayer perceptron network for inter-frame motion prediction and compensation of the dynamic foreground, the static background information shared across frames can be avoided from being repeatedly encoded and stored in each frame, thereby reducing the amount of encoded data for the volumetric video and improving the transmission efficiency of the dynamic scene volumetric video stream.

[0044] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a hardware structure block diagram of a terminal for a dynamic scene volume video stream transmission method according to an embodiment of this application;

[0047] Figure 2 This is a flowchart of a dynamic scene volume video stream transmission method according to an embodiment of this application;

[0048] Figure 3 This is a flowchart of a dynamic scene volume video stream transmission method according to another embodiment of this application;

[0049] Figure 4 This is a structural block diagram of a dynamic scene video stream transmission device according to an embodiment of this application. Detailed Implementation

[0050] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0051] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0052] The method embodiments provided in this example can be executed in a terminal, computer, or similar electronic device with a certain computing power. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of a terminal for a dynamic scene volume video stream transmission method according to an embodiment of this application. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0053] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the dynamic scene volume video stream transmission method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0055] This embodiment provides a method for transmitting dynamic scene volume video streams. Figure 2 This is a flowchart of a dynamic scene volume video stream transmission method according to an embodiment of this application, as follows: Figure 2 As shown, the process includes the following steps:

[0056] Step S210: Obtain the dynamic scene video sequence and separate the starting frame of the dynamic scene video sequence into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the entire video sequence.

[0057] Specifically, the dynamic scene video sequence to be processed is acquired, with the starting frame of the sequence serving as the first keyframe. For the starting frame, a 3D Gaussian splashing technique is used to model its scene content, explicitly separating the 3D Gaussian primitives in the scene into two independent sets: a static Gaussian set and a dynamic Gaussian set. The static Gaussian set corresponds to spatial regions in the scene that remain static in the time dimension, including static structures such as background walls, the ground, and fixed facilities. The dynamic Gaussian set corresponds to spatial regions in the scene that change over time, including dynamic entities such as moving people and objects, and includes multiple dynamic Gaussian points. Once the static Gaussian set is determined, it remains unchanged in subsequent frames within the current image group, without any updates or retransmissions, thus avoiding redundant storage and transmission of static information.

[0058] Furthermore, for each video frame after the initial frame, the dynamic foreground Gaussian set is generated by a shared multilayer perceptron network based on rigid transformation parameters predicted by multi-resolution motion field estimation. Simultaneously, a sparse compensated Gaussian set is added to each frame to handle bursty content and high-frequency details. Therefore, the entire dynamic scene video sequence is represented as:

[0059] ;

[0060] in, The static Gaussian set of the starting frame. M is the dynamic Gaussian set of the starting frame. t Let t be the multi-resolution motion mesh for the t-th frame. Let t be the compensated Gaussian set of the t-th frame. These are the parameters of a multilayer perceptron network shared across all frames. In the representation of dynamic scene video sequences, the static Gaussian set remains unchanged, while the dynamic content evolves over time through motion transformation and adaptive compensation.

[0061] Step S220: For each video frame after the starting frame in the dynamic scene video sequence, the dynamic Gaussian set of each video frame is generated frame by frame based on the motion transformation of the dynamic Gaussian set of the previous video frame, and the dynamic Gaussian set of each video frame is compensated and optimized to obtain the complete dynamic scene of the dynamic scene video sequence.

[0062] First, each video frame after the starting frame in the dynamic scene video sequence is designated as a target frame. For each target frame after the starting frame, the dynamic Gaussian set of the preceding keyframe (or the starting frame if the target frame is the second frame) is obtained. Simultaneously, a corresponding multi-resolution motion mesh is constructed for each target frame. This multi-resolution motion mesh contains multiple resolution layers, each storing motion features at a different spatial resolution. Motion features of each dynamic Gaussian point at each resolution layer can be extracted by performing trilinear interpolation or nearest-neighbor interpolation within the motion mesh at each resolution layer. Then, the motion features extracted from the same dynamic Gaussian point at each resolution layer are concatenated to form the motion description of that dynamic Gaussian point. By summarizing the motion descriptions of all dynamic Gaussian points, the motion description of the target frame can be obtained.

[0063] The motion description of the target frame is input into a shared multilayer perceptron network, which outputs the translation and rotation increments of each dynamic Gaussian point from the previous keyframe to the target frame. Applying these translation and rotation increments to the corresponding dynamic Gaussian point in the previous keyframe yields the dynamic Gaussian set of the target frame.

[0064] Since motion prediction cannot fully cover complex situations such as sudden new objects or drastic deformations, compensation optimization is performed on the dynamic Gaussian set. Compensation detection can be performed based on appearance, copying the Gaussian corresponding to pixels with image spatial gradients exceeding a threshold into the compensation set. Alternatively, compensation detection can be performed based on motion, splitting Gaussian pixels with motion increments exceeding a threshold within their local neighborhoods and incorporating the resulting sub-Gaussian sets into the compensation set. The compensation Gaussian sets obtained through these two methods are merged to obtain a complete compensation Gaussian set. This compensation Gaussian set is then merged with the dynamic Gaussian set to generate the complete dynamic Gaussian set for the target frame. Finally, the complete dynamic Gaussian set and the static Gaussian set are merged to obtain the complete dynamic scene of the dynamic scene video sequence.

[0065] Step S230: Compress the complete dynamic scene to generate a bit stream for transmission.

[0066] First, the multi-resolution motion mesh needs to be quantized and entropy-encoded using range coding to generate a motion mesh bitstream. Second, the static Gaussian set, dynamic Gaussian set, and compensated Gaussian set need to be quantized and entropy-encoded separately for each Gaussian's spatial position, scale, rotation, opacity, and spherical harmonic coefficients as independent attribute channels to generate Gaussian bitstreams. Finally, the motion mesh bitstream and the Gaussian bitstream are merged to generate a bitstream for transmission.

[0067] The mainstream technical approach for dynamic scene volumetric video is based on Neural Radiation Field (NeRF) and its variants for scene modeling and encoding. NeRF implicitly represents the geometric and appearance information of a 3D scene through neural networks, or through explicit representations such as 3D Gaussian splashing, enabling the synthesis of high-fidelity new perspective images from sparse input viewpoints. In contrast, existing volumetric video coding methods typically model and encode the dynamic scene as a whole end-to-end, resulting in the repeated encoding and storage of static background information in each frame. This leads to significant redundancy in the overall encoded data of the dynamic scene volumetric video, low compression efficiency, and low transmission efficiency for dynamic scene volumetric video streams.

[0068] Steps S210 to S230 above involve: First, acquiring a dynamic scene video sequence and separating the starting frame of the dynamic scene video sequence into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the video sequence. Second, for each video frame after the starting frame in the dynamic scene video sequence, based on the motion transformation of the dynamic Gaussian set of the previous video frame, generating the dynamic Gaussian set for each video frame frame by frame, and compensating and optimizing the dynamic Gaussian set of each video frame to obtain the complete dynamic scene of the dynamic scene video sequence. Finally, compressing the complete dynamic scene to generate a bitstream for transmission. By decoupling the dynamic scene into a static background and a dynamic foreground, storing the static background first, and using a multi-resolution motion mesh and a multilayer perceptron network for inter-frame motion prediction and compensation for the dynamic foreground, the static background information shared across frames can be avoided from being repeatedly encoded and stored in each frame, thereby reducing the amount of encoded data in the volumetric video and improving the transmission efficiency of the dynamic scene volumetric video stream.

[0069] In one embodiment, a 3D Gaussian model of the starting frame is trained using multi-view images, optimizing all Gaussian properties (including position, scale, rotation, appearance, and opacity) to minimize the color loss function. The color loss function is a weighted sum of the L1 norm loss and the DisSimilarity Structural Similarity (D-SSIM) loss, expressed as:

[0070] ;

[0071] Where I represents the real image, Indicates the rendered image. L1 loss is used to measure color differences at the pixel level; Structural similarity loss is used to measure the structural similarity of images; The weighting coefficients are used to balance the contributions of the two losses. Then, an intermediate frame (e.g., frame 30) after the starting frame is selected for fine-tuning. The Gaussian number, rotation, appearance, and opacity are fixed, and only the position and scale of the Gaussians are optimized to minimize the color loss in that intermediate frame. Through this fine-tuning, the Gaussians in the moving region can capture dynamic features through changes in position and scale, providing signals for subsequent motion-static decoupling.

[0072] Next, for each Gaussian, its offset in the image space is calculated, and normalized using its depth:

[0073] ;

[0074] Where, d i Let represent the offset vector of the i-th Gaussian in the image space. Let z represent the L2 norm of the offset vector. i This represents the depth value of the Gaussian. Normalized offset δ i This reflects the degree of motion of the Gaussian relative to its depth in the image space. The larger the normalized offset, the more likely the Gaussian is to be a dynamic object. All Gaussians are sorted from largest to smallest according to their normalized offset, and the top η% of Gaussians are marked as dynamic Gaussians, while the rest are marked as static Gaussians, thus obtaining the dynamic pre-set.

[0075] Based on this, the contribution of each Gaussian in the dynamic pre-set to the rendering L1 error is calculated:

[0076] ;

[0077] Among them, C i,k q represents the contribution weight of the i-th Gaussian on the k-th pixel. k This represents the rendering error of the k-th pixel. This error contribution reflects the degree of influence of each Gaussian on the overall rendering error. If the error contribution of a Gaussian in the dynamic pre-set exceeds a preset threshold, that Gaussian is reclassified as a static Gaussian because its higher error contribution is considered to be caused by fitting artifacts rather than real motion.

[0078] Finally, the classification results are optimized based on the K nearest neighbor majority voting in three-dimensional space: for each Gaussian, the categories of its K nearest neighbor Gaussians in three-dimensional space are counted, and the final category of the Gaussian is determined by majority voting, thereby maintaining local consistency and smoothing the classification boundary between static and dynamic, and finally obtaining the separated static Gaussian set and dynamic Gaussian set.

[0079] Optionally, in one embodiment, for each video frame after the starting frame in the dynamic scene video sequence, a dynamic Gaussian set for each video frame is generated frame by frame based on the motion transform of the dynamic Gaussian set of the previous video frame, including:

[0080] Each video frame following the starting frame in the dynamic scene video sequence is sequentially designated as a target frame. The dynamic Gaussian centers of the preceding video frame are position-encoded, and interpolation sampling is performed on the multi-resolution motion grid corresponding to the target frame based on the position encoding results. The motion description of the target frame is obtained based on the interpolation sampling results. When the target frame is the second video frame, the preceding video frame is the starting frame. The multi-resolution motion grid is used to represent the motion information of Gaussian points at different spatial resolutions. The motion description is input into a multilayer perceptron network, which outputs the translation and rotation increments of each dynamic Gaussian point from the preceding video frame to the target frame. The translation and rotation increments are applied to the dynamic Gaussian set of the preceding video frame to generate the dynamic Gaussian set of the target frame.

[0081] First, each video frame after the starting frame in the dynamic scene video sequence needs to be used as the target frame. The dynamic foreground Gaussian set of the video frame preceding the target frame (or the starting frame when the target frame is the second video frame) is used as the basis for motion prediction. Then, a set of multi-resolution motion meshes is constructed for the target frames. Each resolution layer l captures motion features at different spatial resolutions. The dynamic Gaussian centers of each element in the dynamic Gaussian set of the previous video frame are extracted. Three-dimensional spatial position And perform multi-scale location coding. Generate multi-scale position embeddings of each dynamic Gaussian center at different spatial frequencies. Next, the multi-scale positions of each dynamic Gaussian center are embedded into the motion mesh of each resolution layer, and trilinear interpolation is performed to extract the motion features of each dynamic Gaussian center at each resolution layer. Then, the motion features extracted from all resolution layers are concatenated to obtain a unified motion description.

[0082] ;

[0083] Input the motion description into a shared multilayer perceptron network. The shared multilayer perceptron network outputs the translation increment of each dynamic Gaussian from the previous video frame to the target frame. and rotation increment :

[0084] ;

[0085] Applying the translation and rotation increments to the dynamic Gaussian of the previous video frame yields the Gaussian parameters of the target frame:

[0086] ;

[0087] in, Represents the dynamic Gaussian set of the current target frame. This represents the complete dynamic Gaussian set of the previous frame after compensation and optimization. This represents the translation increment of the current target frame. This represents the rotation increment of the current target frame. The three-dimensional spatial position of each dynamic Gaussian point This represents the dynamic Gaussian points from the previous frame's complete dynamic Gaussian set after compensation and optimization. C represents the inherent properties of the dynamic Gaussian, including scale, appearance coefficient, and opacity, which remain unchanged during motion transformation.

[0088] Using the above method, motion features are extracted using a multi-resolution motion mesh, and inter-frame motion transformations are predicted by combining a shared multilayer perceptron. These transformations are then applied frame by frame to the dynamic Gaussian set of the previous video frame, thereby generating the dynamic Gaussian set for each video frame after the starting frame.

[0089] In one embodiment, motion prediction for multi-resolution motion meshes can also be optimized using a rate-distortion joint optimization strategy. During compression, while quantization effectively reduces the bit rate, it leads to information loss and hinders gradient propagation. To address this issue, a differentiable quantization method is introduced, utilizing simulated quantization noise. To simulate the quantization effect and thus maintain gradient continuity, a lightweight trainable model is used to estimate the probability mass function of the quantized motion features, and the probability mass function is calculated using the cumulative distribution function.

[0090] During training, the Gaussian appearance parameters, opacity, and scale are kept constant, and the multi-resolution motion mesh, motion prediction multilayer perceptron, and entropy model are co-optimized. The total loss function consists of a photometric term. and compression ratio term Composition, represented as:

[0091] ;

[0092] The photometric term measures the difference between the reconstructed image and the real image, and is expressed as a weighted sum of L1 loss and structural similarity loss, specifically:

[0093] ;

[0094] in, L1 loss is used to measure color differences at the pixel level; Structural similarity loss is used to measure the structural similarity of images. These are weighting coefficients. Compression ratio term. The bitrate used to estimate the quantized motion features is calculated by using the probability mass function of the quantized motion features to calculate their information entropy, specifically:

[0095] ;

[0096] in, This represents the probability mass function that quantifies the motion feature y. This loss function uses weighting coefficients... Balancing reconstruction quality and compression efficiency allows the network to maintain high-fidelity rendering quality while outputting compact motion representations with low entropy, thereby effectively reducing the transmission bit rate.

[0097] Furthermore, in one embodiment, the dynamic Gaussian centers of the preceding video frame of the target frame are position-encoded, and interpolation sampling is performed in the multi-resolution motion grid corresponding to the target frame based on the position-encoding results. A motion description of the target frame is obtained based on the interpolation sampling results, including:

[0098] The process involves: acquiring the dynamic Gaussian set of the previous video frame of the target frame; extracting the center position of each dynamic Gaussian in the set; performing multi-scale position encoding on each dynamic Gaussian center to generate multi-scale position embeddings of each dynamic Gaussian center at different spatial frequencies; constructing a multi-resolution motion mesh for the target frame; wherein the multi-resolution motion mesh contains multiple resolution layers, each storing motion features at different spatial resolutions; performing trilinear interpolation on the multi-scale position embeddings in the motion meshes of each resolution layer to extract the motion features of the dynamic Gaussian center at each resolution layer; concatenating the motion features extracted from the dynamic Gaussian at each resolution layer to form a motion description of the dynamic Gaussian; and summarizing all the motion descriptions of the dynamic Gaussian to obtain the motion description of the target frame.

[0099] First, it is necessary to obtain the dynamic Gaussian set of the previous video frame of the target frame and extract the 3D spatial position of each dynamic Gaussian center in the dynamic Gaussian set. Multi-scale position encoding is then performed on each dynamic Gaussian center position. By mapping the 3D spatial coordinates to a high-dimensional feature space, multi-scale position embeddings of each dynamic Gaussian center at different spatial frequencies are generated, enabling the network to perceive spatial position information at different granularities.

[0100] Secondly, a multi-resolution motion mesh is constructed for the target frame. The multi-resolution motion mesh contains multiple resolution layers, each storing motion features at a different spatial resolution. The lower resolution layers are used to capture the overall motion trend over a large area, while the higher resolution layers are used to preserve the fine local motion details, thus achieving motion feature extraction from coarse to fine.

[0101] Next, for each dynamic Gaussian center, a multi-scale location embedding is performed, and trilinear interpolation is conducted in the motion mesh of each resolution layer. Specifically, for the spatial location of each dynamic Gaussian center, the eight surrounding grid vertices are located in the motion mesh of its respective resolution layer, and weighted interpolation is performed based on the distance from the location to each vertex to extract the motion features of the location at each resolution layer.

[0102] Then, the motion features extracted from the same dynamic Gaussian at different resolution levels are concatenated to form the motion description of the dynamic Gaussian. This motion description integrates motion information at different spatial resolutions, including both large-scale overall motion trends and small-scale local motion changes.

[0103] Finally, the motion descriptions of all dynamic Gaussians are summarized to obtain the motion description of the target frame. The motion description of the target frame is used to characterize the motion trend of each dynamic Gaussian in the previous video frame at the current time of the target frame, providing a unified input representation for motion prediction of the subsequent multilayer perceptron network.

[0104] In one embodiment, the compensation optimization of the dynamic Gaussian set for each video frame includes: obtaining the dynamic Gaussian set of the current target frame, performing compensation detection on the dynamic Gaussian set to obtain the compensated Gaussian set; and merging the compensated Gaussian set with the dynamic Gaussian set to generate the complete dynamic Gaussian set of the target frame.

[0105] While obtaining a dynamic Gaussian set for the target frame can effectively model the continuous motion transformation of known objects between frames, it can lead to local distortion or loss of detail in the generated dynamic Gaussian set for complex situations such as suddenly appearing new objects, objects that reappear after being occluded, and objects with drastic deformation or high-speed motion. To alleviate these problems, this embodiment introduces a sparse compensated Gaussian set to enhance the expressiveness of the dynamic Gaussian set.

[0106] Specifically, the compensation optimization process involves generating a dynamic Gaussian set for the current target frame through motion prediction, followed by compensation detection on the dynamic Gaussian set. Compensation detection identifies Gaussian regions within the dynamic Gaussian set with insufficient prediction accuracy, i.e., spatial locations requiring additional compensation. Based on the compensation detection results, the corresponding compensated Gaussian set is obtained. Then, the compensated Gaussian set is merged with the dynamic Gaussian set to generate the complete dynamic Gaussian set for the current target frame.

[0107] In one embodiment, the appearance properties and opacity of the compensated Gaussian set can also be optimized to capture details missed in the motion estimation stage, newly emerging content, or complex deformations, maximizing reconstruction quality. The compensated Gaussian set for the current target frame is generated using the trained multi-resolution motion mesh. In this stage, the motion mesh and multilayer perceptron network parameters are kept fixed; only the spherical harmonic coefficients (SH coefficients) and opacity of the compensated Gaussian set are optimized, without introducing explicit bitrate loss. The total loss function consists of a photometric term and a sparsity constraint term, expressed as:

[0108] ;

[0109] in, Used to measure the difference between a reconstructed image and a real image. To compensate for the zero norm of the Gaussian set, the number of compensated Gaussians is constrained. where represents the weighting coefficient. Through the above optimization, while introducing a small number of additional Gaussian elements, the reconstruction quality of complex regions that are difficult to cover by motion prediction in dynamic scenes is effectively improved.

[0110] By introducing a compensated Gaussian set, dynamic change details that cannot be covered in the prediction stage can be supplemented in a targeted manner on the basis of motion prediction, which effectively improves the reconstruction accuracy and rendering quality of sudden content, drastic deformation and high frequency details in dynamic scenes.

[0111] In one embodiment, obtaining the dynamic Gaussian set of the current target frame and performing compensation detection on the dynamic Gaussian set to obtain a compensated Gaussian set includes: detecting the gradient value of each dynamic Gaussian in the dynamic Gaussian set at the projection position in the image space; and copying the dynamic Gaussian corresponding to the pixel whose gradient value exceeds a preset gradient threshold as the appearance compensation Gaussian to the compensated Gaussian set.

[0112] To obtain the dynamic Gaussian set of the current target frame, firstly, each dynamic Gaussian in the set needs to be projected into the image space to determine its projection position. Secondly, the image gradient value at each projection position is calculated and compared with a preset gradient threshold. Then, when the image gradient value exceeds the preset threshold, it is determined that a newly appearing object or an object that has reappeared after being occluded exists in the pixel region corresponding to that projection position, indicating that the motion prediction module has failed to accurately model the current dynamic content in that region. Finally, the dynamic Gaussian corresponding to the pixels with gradient values ​​exceeding the preset threshold is copied to the compensated Gaussian set. In this context, the Gaussian gradient is used for appearance compensation. The magnitude of the image gradient reflects the severity of color or texture changes in the image. Regions with gradient values ​​exceeding a threshold typically correspond to areas with high prediction uncertainty, such as new object edges, object boundaries, or occlusion boundaries. The compensated Gaussian set can be represented as:

[0113] ;

[0114] in, This represents the compensated Gaussian set obtained based on appearance inspection. Represents the dynamic Gaussian set of the current target frame. Indicates the position of the projection at the center of the dynamic Gaussian. This represents a mapping function that projects points in three-dimensional space onto image space. This indicates the preset gradient threshold.

[0115] In one embodiment, the dynamic Gaussian set of the current target frame is obtained, and compensation detection is performed on the dynamic Gaussian set to obtain a compensated Gaussian set, including: extracting the translation increment and rotation increment of each dynamic Gaussian in the dynamic Gaussian set; when the absolute value of the translation increment of the dynamic Gaussian exceeds a first preset threshold, and / or the absolute value of the rotation increment exceeds a second preset threshold, it is determined that the dynamic Gaussian has undergone significant motion; for the dynamic Gaussian that has undergone significant motion, a Gaussian splitting operation is performed in the local neighborhood of the dynamic Gaussian to obtain a reduced sub-Gaussian, and the sub-Gaussian is copied as a motion compensation Gaussian to the compensated Gaussian set.

[0116] Obtain the dynamic Gaussian set of the current target frame, and extract the translation increment of each dynamic Gaussian in the set from the previous video frame to the current target frame. and rotation increment The absolute value of the translation increment of each dynamic Gaussian is compared with the first preset threshold. The values ​​of the rotation increments of each dynamic Gaussian are compared, and the absolute values ​​of these increments are compared with a second preset threshold. A comparison is made. When the absolute value of the translation increment of any dynamic Gaussian exceeds the first preset threshold, and / or the absolute value of the rotation increment exceeds the second preset threshold, it is determined that the dynamic Gaussian has undergone significant motion, indicating that there is drastic local motion or deformation in the region.

[0117] For a dynamic Gaussian undergoing significant motion, in the local region The covariance of each Gaussian is reduced by a factor of 100, enabling the split sub-Gaussians to capture the motion details of local regions with higher spatial precision. These sub-Gaussians are used as motion-compensating Gaussians. The covariance of each sub-Gaussian is smaller than the original covariance of the dynamic Gaussian, and there are at least two sub-Gaussians. Therefore, complete Gaussian compensation can be expressed as... The new dynamic Gaussian can be represented as:

[0118] ;

[0119] in, Represents the dynamic Gaussian set of the current target frame. Represents the complete compensated Gaussian set. Represents the complete dynamic Gaussian set of the current target frame.

[0120] In addition, in one embodiment, the dynamic scene volume video stream transmission method further includes: obtaining the compensated Gaussian set and the dynamic Gaussian set of the current target frame, and calculating the ratio of the number of the compensated Gaussian set to the number of the dynamic Gaussian set; when the ratio exceeds a preset ratio threshold, determining that the scene under the target frame has changed significantly, and starting a new image group; the new image group uses the static Gaussian set of the previous video frame of the target frame to initialize the static background of the new image group, and retains the dynamic Gaussian set of the previous image group.

[0121] Fixed-image-group-size encoding strategies cannot adapt to the uncertainty of scene content changes in dynamic scenes. When the scene background does not change significantly, frequent keyframe insertion leads to a large amount of redundant encoded data, increasing transmission overhead. Furthermore, when the scene undergoes abrupt changes (such as camera cuts, drastic lighting changes, or changes in background content), the fixed-interval update method may cause the background model to lag behind the actual scene changes, affecting the separation accuracy between dynamic foreground and static background. To address these issues, this embodiment also proposes an adaptive image group partitioning strategy.

[0122] Specifically, the compensated Gaussian set and the dynamic Gaussian set of the current target frame are obtained. The compensated Gaussian set is an additional set of Gaussians introduced to compensate for dynamic change details that cannot be covered in the motion prediction stage. The quantity of the compensated Gaussian set reflects the size of the region in the current target frame that motion prediction failed to accurately model. The ratio of the quantity of the compensated Gaussian set to the dynamic Gaussian set is calculated, and this ratio is used to measure the significance of scene changes in the current target frame. When the ratio exceeds a preset threshold, it is determined that the scene in the current target frame has changed significantly, and a new image group is started.

[0123] When constructing a new image set, the static background of the new image set is initialized using the static Gaussian set of the previous video frame of the target frame, while the dynamic Gaussian set of the previous image set is retained, thus avoiding unnecessary background reconstruction. During the training of the new image set, only the static Gaussian set is updated. Gaussian splitting and pruning operations are used to optimize the static Gaussian set to adapt to changes in static content under the new scene. At the same time, the dynamic Gaussian set is adjusted through a motion transform network to achieve accurate separation between dynamic content and static background. Within the new image set, the static Gaussian set remains unchanged after the update, while the dynamic Gaussian set continues to be generated frame by frame through inter-frame motion prediction.

[0124] By using an adaptive image group partitioning strategy, the timing of keyframe insertion can be dynamically adjusted according to the scene content. When the background is stable, redundant coding caused by frequent keyframe insertion can be avoided. When the scene changes abruptly, the static background model can be updated in a timely manner to ensure the accuracy of dynamic and static separation, thereby improving the overall coding efficiency and reconstruction quality.

[0125] In one embodiment, the complete dynamic scene is compressed to generate a bitstream for transmission, including: entropy encoding of the multi-resolution motion mesh, transmitting the data distribution of the multi-resolution motion mesh and the entropy-encoded data; compressing the static Gaussian set, the dynamic Gaussian set of the starting frame, and the compensated Gaussian set using a point cloud compression framework, quantizing and entropy encoding the spatial position, scale, rotation, opacity, and spherical harmonic coefficient of each Gaussian as independent attribute channels to generate a Gaussian compressed bitstream.

[0126] On one hand, the multi-resolution motion mesh is compressed. Each frame of the multi-resolution motion mesh is quantized, converting floating-point mesh data into discrete integer data; subsequently, a range encoder is used to entropy encode the quantized motion mesh. Since the entropy model itself occupies a relatively large amount of space (approximately 100KB / frame), to reduce its impact, the data distribution of the multi-resolution motion mesh is analyzed before entropy encoding. Instead of transmitting the complete entropy model, the data distribution is transmitted, thus effectively reducing the transmission overhead per frame. The entropy coding process is as follows:

[0127] ;

[0128] ;

[0129] Where x represents the multi-resolution motion mesh to be compressed, q is the quantization parameter used to balance compression accuracy and data range, Q(x) represents the quantized integer data, and Q(min(x)) represents the minimum value after quantization. This represents the bitstream generated after entropy encoding, and E represents the range encoder. For data compressed to int8 format, it is first converted to a non-negative value, and then restored to the original range during decoding. At the decoding end, the data is recovered through the range decoder D and the dequantization operation, where the formula for the dequantization operation is:

[0130] ;

[0131] Each frame needs to transmit only the bit stream. and its corresponding data distribution This eliminates the need to transmit the complete entropy model, further reducing the amount of data transmitted per frame.

[0132] On the other hand, to achieve efficient compression of static backgrounds, dynamic foregrounds, and compensated Gaussians, a static Gaussian set, a dynamic Gaussian set for the starting frame, and compensated Gaussian sets for each target frame are acquired. A point cloud compression framework based on a KD-tree provided by the data compression algorithm (Draco) is used to compress each Gaussian set. The native geometry and attribute encoder of the data compression algorithm is extended to support the compression of 3D Gaussian primitives for representing static volume content in the system. Each Gaussian has attributes such as spatial position (x, y, z), anisotropic scale (s0, s1, s2), rotation quaternion, opacity, and spherical harmonic coefficients. During encoding, these attributes of each Gaussian are processed as independent attribute channels: spatial position, scale, and rotation are encoded as multi-component floating-point attributes, and spherical harmonic coefficients are divided into DC components and higher-order components for separate processing to further improve compression efficiency. Each attribute channel is quantized independently, and a KD-tree spatial partitioning structure is used to organize the Gaussian centers in 3D space, ensuring locality preservation encoding and improving entropy coding efficiency. The extended encoder and decoder are compatible with Draco's modular design, allowing seamless integration with standard Draco tools. The compressed stream can be efficiently decoded for static scene reconstruction, supporting storage and transmission in volumetric video streaming systems. The quantization results of each attribute channel are entropy-encoded using a range-encoding backend to generate a Gaussian compressed bitstream.

[0133] By jointly compressing the spatial and appearance features of static Gaussians, a high compression ratio is achieved while maintaining the visual fidelity required for downstream rendering. The compressed bitstream can be efficiently decoded for static scene reconstruction and supports storage and transmission in volumetric video streaming systems.

[0134] Figure 3 This is a flowchart of a dynamic scene volume video stream transmission method according to another embodiment of this application. Figure 3 As shown, the dynamic scene volume video stream transmission method includes the following steps:

[0135] Step S301: Obtain the dynamic scene video sequence and separate the starting frame into a static Gaussian set and a dynamic Gaussian set.

[0136] Step S302: For each target frame after the starting frame, generate the dynamic Gaussian set of each target frame frame by frame based on the motion transformation of the dynamic Gaussian set of the previous video frame.

[0137] Step S303: Compensate and optimize the dynamic Gaussian set of each target frame, and merge the compensated Gaussian set with the dynamic Gaussian set to generate the complete dynamic Gaussian set of the target frame.

[0138] Step S304: Based on the ratio of the amount of the compensated Gaussian set to the dynamic Gaussian set, an adaptive new image group is started, and the static background of the new image group is initialized with the static Gaussian set of the previous video frame of the target frame.

[0139] Step S305: Compress the multi-resolution motion mesh and Gaussian data in the complete dynamic scene to generate a bitstream for transmission.

[0140] Steps S301 to S305 above reduce the amount of encoded data and transmission bitrate of dynamic scene volume video by decoupling Gaussian representation and compression strategies of dynamic Gaussian and static Gaussian, achieving efficient streaming transmission and flexible editing while ensuring high-fidelity rendering quality.

[0141] Figure 4 This is a structural block diagram of a dynamic scene volume video stream transmission device 40 according to an embodiment of this application, as follows: Figure 4 As shown, the dynamic scene video stream transmission device 40 includes: a separation module 42, a prediction module 44, and a transmission module 46; wherein: the separation module 42 is used to acquire the dynamic scene video sequence and separate the starting frame of the dynamic scene video sequence into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the entire video sequence; the prediction module 44 is used to generate the dynamic Gaussian set of each video frame after the starting frame in the dynamic scene video sequence, based on the motion transformation of the dynamic Gaussian set of the previous video frame, and to compensate and optimize the dynamic Gaussian set of each video frame to obtain the complete dynamic scene of the dynamic scene video sequence; the transmission module 46 is used to compress the complete dynamic scene to generate a bit stream for transmission.

[0142] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0143] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0144] Furthermore, in conjunction with the dynamic scene volume video stream transmission method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the dynamic scene volume video stream transmission methods in the above embodiments.

[0145] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0147] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0148] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0149] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for transmitting dynamic scene volumetric video streams, characterized in that, include: A dynamic scene video sequence is obtained, and the starting frame of the dynamic scene video sequence is separated into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the entire video sequence. For each video frame after the starting frame in the dynamic scene video sequence, the dynamic Gaussian set of each video frame is generated frame by frame based on the motion transformation of the dynamic Gaussian set of the previous video frame, and the dynamic Gaussian set of each video frame is compensated and optimized to obtain the complete dynamic scene of the dynamic scene video sequence. The complete dynamic scene is compressed to generate a bitstream for transmission.

2. The dynamic scene volumetric video stream transmission method according to claim 1, characterized in that, The step of generating a dynamic Gaussian set for each video frame after the starting frame in the dynamic scene video sequence, based on the motion transformation of the dynamic Gaussian set of the previous video frame, frame by frame, includes: Each video frame after the starting frame in the dynamic scene video sequence is taken as the target frame. The dynamic Gaussian centers of the preceding video frame of the target frame are position-encoded. Based on the position encoding results, interpolation sampling is performed on the multi-resolution motion grid corresponding to the target frame. The motion description of the target frame is obtained based on the interpolation sampling results. When the target frame is the second video frame, the preceding video frame of the target frame is the starting frame. The multi-resolution motion grid is used to represent the motion information of Gaussian points at different spatial resolutions. The motion description is input into a multilayer perceptron network, which outputs the translation and rotation increments of each dynamic Gaussian from the previous video frame to the target frame. The translation and rotation increments are applied to the dynamic Gaussian set of the previous video frame of the target frame to generate the dynamic Gaussian set of the target frame.

3. The dynamic scene volume video stream transmission method according to claim 2, characterized in that, The step involves encoding the positions of the dynamic Gaussian centers of the preceding video frame of the target frame, performing interpolation sampling based on the position encoding results in the multi-resolution motion grid corresponding to the target frame, and obtaining the motion description of the target frame based on the interpolation sampling results, including: Obtain the dynamic Gaussian set of the previous video frame of the target frame, and extract the center position of each dynamic Gaussian in the dynamic Gaussian set; Multi-scale position encoding is performed on each dynamic Gaussian center location to generate multi-scale position embeddings of each dynamic Gaussian center at different spatial frequencies. A multi-resolution motion mesh is constructed for the target frame; wherein the multi-resolution motion mesh contains multiple resolution layers, and each resolution layer stores motion features at different spatial resolutions; Trilinear interpolation is performed on the motion mesh of the multi-scale position embedded in each resolution layer to extract the motion features of the dynamic Gaussian center position at each resolution layer. The motion features extracted by the dynamic Gaussian at each resolution layer are spliced ​​together to form the motion description of the dynamic Gaussian. Summarize the motion descriptions of all dynamic Gaussians to obtain the motion description of the target frame.

4. The dynamic scene volume video stream transmission method according to claim 1, characterized in that, The compensation and optimization of the dynamic Gaussian set for each video frame includes: Obtain the dynamic Gaussian set of the current target frame, perform compensation detection on the dynamic Gaussian set, and obtain the compensated Gaussian set; The compensated Gaussian set is merged with the dynamic Gaussian set to generate the complete dynamic Gaussian set of the target frame.

5. The dynamic scene volumetric video stream transmission method according to claim 4, characterized in that, The step of obtaining the dynamic Gaussian set of the current target frame and performing compensation detection on the dynamic Gaussian set to obtain the compensated Gaussian set includes: Detect the gradient values ​​of each dynamic Gaussian in the dynamic Gaussian set at its projection position in the image space; The dynamic Gaussian corresponding to the pixel whose gradient value exceeds the preset gradient threshold is used as the appearance compensation Gaussian and copied to the compensation Gaussian set.

6. The dynamic scene volumetric video stream transmission method according to claim 4, characterized in that, The step of obtaining the dynamic Gaussian set of the current target frame and performing compensation detection on the dynamic Gaussian set to obtain the compensated Gaussian set includes: Extract the translation and rotation increments of each dynamic Gaussian in the dynamic Gaussian set; When the absolute value of the translation increment of the dynamic Gaussian exceeds the first preset threshold, and / or the absolute value of the rotation increment exceeds the second preset threshold, it is determined that the dynamic Gaussian has undergone significant motion. For a dynamic Gaussian that undergoes significant motion, a Gaussian splitting operation is performed in the local neighborhood of the dynamic Gaussian to obtain a reduced sub-Gaussian. The sub-Gaussian is then copied as a motion-compensated Gaussian to the compensation Gaussian set.

7. The dynamic scene volume video stream transmission method according to claim 4, characterized in that, The dynamic scene volume video stream transmission method further includes: Obtain the compensated Gaussian set and the dynamic Gaussian set of the current target frame, and calculate the ratio of the number of the compensated Gaussian set to the number of the dynamic Gaussian set; When the quantity ratio exceeds a preset quantity ratio threshold, it is determined that the scene under the target frame has changed significantly, and a new image group is started; the new image group uses the static Gaussian set of the previous video frame of the target frame to initialize the static background of the new image group, and retains the dynamic Gaussian set of the previous image group.

8. The dynamic scene volume video stream transmission method according to claim 4, characterized in that, The step of compressing the complete dynamic scene to generate a bitstream for transmission includes: Entropy coding is performed on the multi-resolution motion mesh, and the data distribution of the multi-resolution motion mesh and the entropy-coded data are transmitted. The static Gaussian set, the dynamic Gaussian set of the starting frame, and the compensated Gaussian set are compressed using a point cloud compression framework. The spatial position, scale, rotation, opacity, and spherical harmonic coefficient of each Gaussian are quantized and entropy encoded as independent attribute channels to generate a Gaussian compressed bitstream.

9. A dynamic scene volumetric video stream transmission device, characterized in that, include: The module is divided into a description module, a prediction generation module, and a generation and transmission module; wherein: The separation description module is used to acquire a dynamic scene video sequence and separate the starting frame of the dynamic scene video sequence into a static Gaussian set and a dynamic Gaussian set; the static Gaussian set remains unchanged throughout the entire video sequence. The prediction generation module is used to generate a dynamic Gaussian set for each video frame after the starting frame in the dynamic scene video sequence, based on the motion transformation of the dynamic Gaussian set of the previous video frame, and to compensate and optimize the dynamic Gaussian set of each video frame to obtain the complete dynamic scene of the dynamic scene video sequence. The generation and transmission module is used to compress the complete dynamic scene and generate a bit stream for transmission.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the dynamic scene volume video stream transmission method according to any one of claims 1 to 8.

Images