A video stream processing method with dynamic Gaussian compression and adaptive bit rate control

Abstract

This invention discloses a video stream processing method with dynamic Gaussian compression and adaptive bitrate control, suitable for scenarios such as virtual reality, augmented reality, and three-dimensional video. The method comprises the following steps: S1. Gaussian attribute modeling and initialization; S2. Binary hash grid construction; S3. Deformation prediction network construction; S4. Mask pruning mechanism design; S5. Entropy modeling and arithmetic encoding and decoding module design; S6. Model training; and S7. Video stream transmission at multiple bitrates. This invention proposes a unified solution combining Gaussian volume cloud coding and adaptive video transmission, significantly reducing video data storage and transmission costs and achieving comprehensive performance superior to existing methods on multiple real and synthetic datasets.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional video representation and compression, and in particular to a video stream processing method with dynamic Gaussian compression and adaptive bit rate control. Background Art

[0002] With the development of augmented reality and virtual reality technologies, the demand for high-quality representation and real-time transmission of dynamic 3D scenes is becoming increasingly urgent. In recent years, the 3D Gaussian Splatting (3DGS) method has demonstrated extremely high rendering speed and image fidelity for static scene rendering. Compared to traditional Neural Radiance Field (NeRF) methods, it offers advantages such as differentiability, explicitness, and real-time performance. Consequently, it has been rapidly extended to the field of 3D video (GSV) representation.

[0003] However, extending Gaussian sputtering technology to dynamic scenes still faces significant technical bottlenecks, mainly reflected in the following aspects:

[0004] Low compression efficiency: Since dynamic scenes require Gaussian volume representation to be constructed frame by frame, the storage cost increases sharply with the time series. Current compression methods are mostly based on static Gaussian structures, which makes it difficult to effectively model the temporal changes of Gaussian properties, resulting in serious accumulation of redundant information.

[0005] Weak streaming media adaptability: The existing GSV encoding method does not take into account bandwidth changes, terminal buffering capabilities, and QoE (Quality of Experience) optimization strategies during network transmission, which often leads to problems such as playback interruptions and frequent bitrate changes.

[0006] The reconstruction-speed-storage trilemma: Traditional methods struggle to balance the optimization of compression rate, rendering speed, and reconstruction quality, making them difficult to deploy on mobile devices and in bandwidth-constrained scenarios.

[0007] Some studies have attempted to reduce storage overhead through pruning, quantization, hash grid reconstruction, etc., but most of them are limited to static scenes. In dynamic three-dimensional scenes, only a very few methods have achieved coarse-grained bit rate control. Their coding structures are redundant, the training overhead is high, and the real-time performance is poor. A set of efficient compression and transmission systems that can be widely used in three-dimensional videos has not yet been formed.

[0008] Therefore, there is an urgent need for a new 3D video representation method that can simultaneously model dynamic property deformation, efficiently compress Gaussian features, and support adaptive bitrate control in streaming transmission, so as to adapt to the playback requirements in real network environments and improve the overall user experience. Summary of the Invention

[0009] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a video stream processing method with dynamic Gaussian compression and adaptive bit rate control, which realizes efficient encoding, low-latency transmission and stable decoding of dynamic three-dimensional scenes.

[0010] The object of the present invention is achieved through the following technical solution: a video stream processing method with dynamic Gaussian compression and adaptive bit rate control, comprising the following steps:

[0011] S1. Gaussian attribute modeling and initialization: The dynamic 3D scene sequence is converted into a 3D Gaussian volume set representation, where each Gaussian volume is used to simulate the local properties of the object surface;

[0012] S2. Based on a multi-resolution binary hash grid, construct a two-dimensional feature mapping model that maps from two-dimensional spatiotemporal positions to two-dimensional composite feature vectors, and a three-dimensional feature mapping model that maps from three-dimensional spatial features to two-dimensional composite feature vectors;

[0013] S3. Constructing a deformation prediction network: Based on binary spatial feature vectors, a composite feature representation that integrates space and time is constructed. This feature is then fed into a multi-head neural network to predict the temporal increments of 3D Gaussian volume attributes, thereby enabling continuous modeling of point attributes in dynamic 3D scenes.

[0014] S4. Mask pruning mechanism design: By constructing a spatiotemporal-sensitive learnable mask module, we automatically identify and remove Gaussian volumes that have a low impact on reconstruction quality in the spatial or temporal dimensions.

[0015] S5. Entropy Modeling and Arithmetic Encoding and Decoding Module Design: By building a deep entropy model network, we probabilistically model the distribution of continuous attributes. We compress continuous attributes and discrete values ​​into bitstreams using an arithmetic encoder and restore them using an arithmetic decoder.

[0016] S6. Perform image rendering and use RGB images collected from multiple perspectives as supervision signals to jointly train the deformation prediction network, mask module, and deep entropy model network;

[0017] S7. Design a multi-bitrate video streaming transmission scenario, and obtain the deformation prediction network and mask module corresponding to different bitrates by adjusting the number of Gaussian volumes and the feature grid accuracy. During the video transmission process, switch the video transmission bitrate based on the network quality for data transmission.

[0018] The beneficial effects of the present invention are: the present invention overcomes the problems existing in existing three-dimensional video compression and streaming transmission methods, such as high compression redundancy, uncontrollable bit rate, non-reusable model, and poor playback experience, and realizes efficient encoding, low-latency transmission and stable decoding of dynamic three-dimensional scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the overall principle of the present invention;

[0020] Figure 2 Schematic diagram of the deformation prediction network;

[0021] Figure 3 Schematic diagram of the compression and decoding process. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0023] like Figure 1 As shown, a video stream processing method with dynamic Gaussian compression and adaptive bit rate control includes the following steps:

[0024] S1. Gaussian attribute modeling and initialization: The dynamic 3D scene sequence is converted into a 3D Gaussian volume set representation, where each Gaussian volume is used to simulate the local properties of the object surface;

[0025] S2. Based on a multi-resolution binary hash grid, construct a two-dimensional feature mapping model that maps from two-dimensional spatiotemporal positions to two-dimensional composite feature vectors, and a three-dimensional feature mapping model that maps from three-dimensional spatial features to two-dimensional composite feature vectors;

[0026] S3. Constructing a deformation prediction network: Based on binary spatial feature vectors, a composite feature representation that integrates space and time is constructed. This feature is then fed into a multi-head neural network to predict the temporal increments of 3D Gaussian volume attributes, thereby enabling continuous modeling of point attributes in dynamic 3D scenes.

[0027] S4. Mask pruning mechanism design: By constructing a spatiotemporal-sensitive learnable mask module, we automatically identify and remove Gaussian volumes that have a low impact on reconstruction quality in the spatial or temporal dimensions.

[0028] S5. Entropy Modeling and Arithmetic Encoding and Decoding Module Design: By building a deep entropy model network, we probabilistically model the distribution of continuous attributes. We compress continuous attributes and discrete values ​​into bitstreams using an arithmetic encoder and restore them using an arithmetic decoder.

[0029] S6. Perform image rendering and use RGB images collected from multiple perspectives as supervision signals to jointly train the deformation prediction network, mask module, and deep entropy model network;

[0030] S7. Design a multi-bitrate video streaming transmission scenario, and obtain the deformation prediction network and mask module corresponding to different bitrates by adjusting the number of Gaussian volumes and the feature grid accuracy. During the video transmission process, switch the video transmission bitrate based on the network quality for data transmission.

[0031] The step S1 includes the following sub-steps:

[0032] S101. Dynamic scene preprocessing:

[0033] The positions of multiple cameras are fixed during synchronous shooting. The first frame of the multi-view motion video sequence captured by the camera is sparsely reconstructed using the motion recovery structure algorithm to obtain the 3D point cloud and camera pose, and to construct the initial sparse point cloud frame set. ;

[0034] S102. Initial configuration of Gaussian body:

[0035] Based on the initial sparse point cloud frame set Each point in , initialize a three-dimensional Gaussian body , whose initial attribute set is defined as follows:

[0036] Spatial location ;

[0037] RGB color , represents the color of the Gaussian body;

[0038] Rotation Quaternion , used to represent The orientation of the Gaussian body at the moment;

[0039] Zoom factor , respectively control Variance in direction;

[0040] Opacity , used to control the illumination superposition process;

[0041] The initial Gaussian set is defined as:

[0042]

[0043] in is the number of initialized Gaussian bodies; the parameters of the Gaussian bodies are continuously optimized and adjusted during the subsequent training process;

[0044] S103. Gaussian Volume Merging and Normalization:

[0045] Spatial proximity merging: cluster and merge Gaussian bodies within the local spatial radius δ, and use weighted average for the merged attributes;

[0046] Color normalization: normalize the color Normalize to the interval [0,1] to unify the input scale;

[0047] Normalize the pose: Normalize to ensure it is a valid quaternion.

[0048] In step S2, the multi-resolution binary hash grid includes a resolution binary hash grid for a two-dimensional feature mapping model and a resolution binary hash grid for a three-dimensional feature mapping model;

[0049] According to the multi-resolution binary hash grid for the two-dimensional feature mapping model, a two-dimensional feature mapping model is constructed from the two-dimensional spatiotemporal position mapping to the two-dimensional composite feature vector, including:

[0050] A1. Initialize the multi-layer hash grid structure:

[0051] Build The hash grid structure with increasing layer resolution is denoted as , where each layer Contains hash table entries , used to store binary features, each hash table item is a key-value pair ,satisfy:

[0052] is the hash result of the two-dimensional integer coordinates;

[0053] is a binary feature component;

[0054] A2. Hash function and resolution setting:

[0055] Set the corresponding resolution for each layer of hash grid , whose values ​​are incremented as follows:

[0056]

[0057] in is the minimum layer resolution;

[0058] A3. Construct a two-dimensional feature mapping model from two-dimensional spatiotemporal position mapping to a two-dimensional composite feature vector:

[0059] For any two-dimensional space-time position , first use the following hash function to convert the space-time coordinates Map to Integer keys on the layer hash grid :

[0060]

[0061] Then find the integer coordinates corresponding to the adjacent k value, will The values ​​are arranged into initial vectors in ascending order, and bilinear interpolation is used to obtain smooth eigenvectors. , repeat the above process and calculate the layer feature vector;

[0062] Then concatenate the features of all layers into a composite feature representation:

[0063]

[0064] in, ; represents the dimension of the composite feature, Indicates the Smoothed feature vector obtained by layer hash table Dimensions;

[0065] For the two-dimensional feature mapping model, since there are two coordinate axes, there are 4 adjacent integer coordinates;

[0066] According to the multi-resolution binary hash grid for the three-dimensional feature mapping model, a three-dimensional feature mapping model is constructed from the three-dimensional spatiotemporal position mapping to the two-dimensional composite feature vector, including the following sub-steps:

[0067] B1. Initialize the multi-layer hash grid structure:

[0068] Build The hash grid structure with increasing layer resolution is denoted as , where each layer Contains hash table entries , used to store binary features, each hash table item is a key-value pair ,satisfy:

[0069] is the hash result of the three-dimensional integer coordinates;

[0070] is a binary feature component;

[0071] B2. Hash function and resolution setting:

[0072] Set the corresponding resolution for each layer of hash grid , whose values ​​are incremented as follows:

[0073]

[0074] in is the minimum layer resolution;

[0075] B3. Construct a 3D feature mapping model that maps 3D spatiotemporal positions to 2D composite feature vectors:

[0076] For any input position , use the following hash function to convert the floating point coordinates Maps to integer keys :

[0077]

[0078] Then find the integer coordinates corresponding to the adjacent k value, will The values ​​are arranged into initial vectors in ascending order, and trilinear interpolation is used to obtain smooth eigenvectors. , repeat the above process and calculate the Layer feature vector; then concatenate the features of all layers into a composite feature representation:

[0079]

[0080] in, ; represents the dimension of the composite feature, Indicates the Smoothed feature vector obtained by layer hash table dimension.

[0081] For the three-dimensional feature mapping model, since there are three coordinate axes, there are 8 adjacent integer coordinates;

[0082] like Figure 2 As shown, step S3 includes the following sub-steps:

[0083] S301. Input feature construction:

[0084] On the basis of the three-dimensional coordinate system, a time t coordinate axis is added, and for each Gaussian body, its spatial position is , timestamp is , construct a four-dimensional joint input vector ;The input is projected onto the following set of 2D planes:

[0085] Space Plane: 、 、 ;

[0086] Space-time plane: 、 、 ;

[0087] S302. Planar Feature Extraction and Fusion:

[0088] S3021. Use the two-dimensional feature mapping model on each plane to obtain the composite features on the plane, denoted as ,in, ;

[0089] S3022. Use the three-dimensional feature mapping model on the entire space to obtain the composite features of the space, which are recorded as three-dimensional network features ;

[0090] S3023. Perform Hadamard product fusion on all plane features and combine them with 3D mesh features Splicing to construct a multi-resolution composite feature vector:

[0091]

[0092] in Gaussian body position point set, is a plane set;

[0093] S303. Deformation Prediction Network Design:

[0094] Will Input to the multi-head MLP network ,in:

[0095]

[0096] Represent the branch network parameters of position, color, rotation, scale, and opacity respectively;

[0097] Each branch network is used to predict five types of attributes, including:

[0098] Position increment: ;

[0099] Color Increment: ;

[0100] Rotation increment: ;

[0101] Zoom increment: ;

[0102] Opacity Increment: ;

[0103] The final output attributes are updated as follows:

[0104]

[0105] The step S4 includes the following sub-steps:

[0106] S401. Mask parameter initialization:

[0107] For each Gaussian Initialize a learnable saliency parameter , used to calculate the contribution of the Gaussian to the final image reconstruction. This parameter is used as the input of the neural network for training and updating;

[0108] S402. Constructing a mask prediction network:

[0109] Build a small MLP as a mask predictor , the input is , the output is the masked activation value:

[0110]

[0111] in represents the sigmoid activation function;

[0112] S403. Mask Binarization and Differentiable Processing:

[0113] Using output activation and set threshold Compare and calculate the binary mask :

[0114]

[0115] in: is an indicator function, if , the indicator function takes 1, if it is not satisfied, the indicator function takes 0, Indicates stopping gradient propagation;

[0116] S404. Mask applied to Gaussian properties:

[0117] The mask generated Applied to various Gaussian attribute encoding processes:

[0118] like , then the Gaussian body is retained to participate in deformation, encoding and rendering;

[0119] like , then the point is eliminated in the current frame and does not participate in rendering and compression.

[0120] The step S5 includes the following sub-steps:

[0121] S501. Attribute set division:

[0122] The original Gaussian attributes are classified according to whether they are compressible or not, and the attribute set to be compressed is defined as:

[0123]

[0124] Geometric position Due to the sensitivity to precision, lossless compression is performed separately;

[0125] S502. Construct context feature vector:

[0126] Construct context information for each Gaussian , as the input of the deep entropy model network, where 3D mesh features:

[0127]

[0128] S503. Conditional Probability Distribution Modeling:

[0129] The conditional probability distribution is calculated for the continuous-valued attributes of the Gaussian volume and the binary multi-resolution hash grid for subsequent arithmetic coding compression:

[0130] S5031. For continuous attributes: building a deep entropy model network , which consists of several layers of MLP, and its parameters are Indicates that the input is , used to predict attribute values The conditional probability distribution of attribute value Including color, scaling, rotation, transparency: predict its mean and standard deviation , construct Gaussian distribution, recorded as:

[0131]

[0132] Optimize during subsequent training;

[0133] S5032. For discrete values, including binary multi-resolution hash grids, the percentage is calculated based on the number of binary feature components +1 and -1, and a Bernoulli distribution is constructed:

[0134] S504. Arithmetic encoder compression process:

[0135] This step is divided into continuous attribute compression and discrete value compression, such as Figure 3 As shown;

[0136] S5041. Continuous Attribute Compression: Arithmetic Encoder Using Deep Entropy Model Network The obtained Gaussian distribution , for attribute values Perform compression;

[0137] For continuous-valued attributes First perform discrete quantization:

[0138]

[0139] The cumulative distribution function CDF is then constructed based on the predicted probability distribution:

[0140]

[0141] in The cumulative distribution function of the standard normal distribution is used to assign a probability interval in the range [0,1) to each quantized symbol as the input of the arithmetic coding.

[0142] According to the preset encoding accuracy , the interval Divide evenly into subintervals, determine The subinterval it falls into:

[0143]

[0144] Arithmetic coding: Use an arithmetic encoder to The interval Encoded as a bitstream , as a continuous attribute A compressed representation of

[0145] Repeat the above process to encode each attribute component independently; finally, output the bitstream segments of all attributes to form the overall compressed bitstream;

[0146] S5042. Discrete Value Compression: The arithmetic encoder uses the Bernoulli distribution in S5032 as the probability model input; it compresses the discrete values, i.e., the multi-resolution binary hash grid:

[0147] First, a fixed mapping of symbols is performed:

[0148]

[0149] This constructs a discrete probability distribution:

[0150]

[0151] And based on this, the cumulative distribution function CDF is defined as:

[0152]

[0153] Arithmetic coding: using the corresponding interval in CDF Perform arithmetic coding; the resulting bitstream fragment This is a compressed representation; all symbol values ​​are encoded independently, ultimately generating a compressed bit stream;

[0154] S505 arithmetic decoder to restore the compressed bit stream process: the arithmetic decoder decodes the continuous attributes and discrete values ​​?? respectively;

[0155] S5051. Arithmetic decoder recovery process for Gaussian continuous value attributes:

[0156] The corresponding bit stream segment is sent as input to the arithmetic decoder according to the probability distribution in S5031. Perform interval decoding operation:

[0157] Interval recovery: based on current probability distribution , the bitstream Map back to the probability interval in which it is located , that is, determine the cumulative probability range corresponding to the attribute value;

[0158] Decoding attribute value: according to the center value of the probability interval , inversely solve its position in the predicted Gaussian distribution and calculate the inverse quantized value:

[0159]

[0160] in represents the inverse cumulative distribution function of the standard normal distribution;

[0161] According to the attribute values ​​obtained by decoding, the decoded Gaussian volume is restored;

[0162] S5052. Arithmetic decoder decodes discrete values:

[0163] The bit stream segment in S5042 Input arithmetic decoder;

[0164] According to the Bernoulli distribution used in compression in S5042, it is located in the interval [0,1) The probability interval is:

[0165]

[0166] Finally, perform symbolic inverse mapping, Restore to actual value :

[0167]

[0168] After decoding, the multi-resolution binary hash grid is restored.

[0169] The step S6 comprises:

[0170] S601. Model building and rendering process:

[0171] The process input timestamp And camera parameters, camera parameters include intrinsic parameters and extrinsic parameters, output rendered images, including:

[0172] S6011. Calculate the Gaussian volume set at time t:

[0173] For the current Gaussian body set, according to the camera parameters, remove the Gaussian bodies outside the viewing cone, that is, the invisible Gaussian bodies, and obtain the remaining Gaussian body set ;

[0174] Gaussian set The compression properties in the deep entropy model network are Gaussian distributed, and based on the Gaussian distribution, the compression properties of each Gaussian in the arithmetic encoder Gaussian set are compressed into a bit stream, and the spatial position Due to the sensitivity to precision, lossless compression is performed separately, and the compression properties of each Gaussian volume are restored through an arithmetic decoder to obtain a restored Gaussian volume set;

[0175] The position information of the Gaussian body in the restored Gaussian body set and time Input into the deformation prediction network to calculate the deformation field ,Will Acts on The attribute value of is deformed to obtain Gaussian set at the moment ;

[0176] Input time Go to the mask module to obtain the mask value corresponding to each Gaussian body, remove the Gaussian bodies below the threshold, and obtain the Gaussian body set after removing the threshold ;

[0177] S6012. Remove the Gaussian volume set after the threshold is removed Rendering into RGB image: Depth information is calculated based on the camera projection plane and Gaussian body position information, the three-dimensional Gaussian ellipsoids are sorted and the pixel projection weights are determined; using the rasterization algorithm, the surface points of the sorted ellipsoids are projected onto a two-dimensional plane according to their positions and projection weights to obtain a rendered RGB image.

[0178] S602. A dynamic Gaussian coding system is used to optimize the parameters of each submodule through end-to-end training, including the deformation prediction network, mask module, and deep entropy model network. The training process relies on RGB images collected from multiple views as supervision signals. The specific process includes joint optimization objective design, network optimization, and training strategy. The process is as follows:

[0179] Projection and depth setting: Calculate the vertical distance from each point of the 3D Gaussian ellipsoid to the image plane, and take the shortest distance as the ellipsoid depth information;

[0180] Ellipsoid sorting: Sort the 3D Gaussian ellipsoids from small to large according to depth information, ensuring that the near-field ellipsoid is rendered first;

[0181] Projection and weight calculation: Using a "snowballing" approach, the sorted ellipsoids are splashed onto a two-dimensional projection plane to obtain projection points. The projection weight of the pixel corresponding to the projection point is calculated by combining the projection point coordinates, ellipsoid morphology, and depth information; the ellipsoid morphology includes shape and size.

[0182] Pixel superposition and processing: The projected point attributes, including color and brightness, are superimposed on the pixel value according to the weight. If multiple projected points correspond to the same pixel, the final pixel value is obtained through weighted averaging.

[0183] Differentiable rendering for training: Gaussian splash is a differentiable rendering that can be used for neural network training. It optimizes the color, position, transparency, rotation, and size of the Gaussian by minimizing the pixel loss between the rendered image and the real image.

[0184] A dynamic Gaussian coding system is used to optimize the parameters of each submodule through end-to-end training, including the deformation prediction network, mask module, and deep entropy model network. The training process relies on RGB images collected from multiple perspectives as supervision signals, and the specific process includes joint optimization goal design, network optimization, and training strategy.

[0185] D1. Joint optimization objective design:

[0186] The deformation prediction network, mask module and deep entropy model network participate in joint training, and the optimization goal is the joint loss function :

[0187]

[0188] in:

[0189] : compression entropy;

[0190] : reconstruction error;

[0191] : loss balance coefficient;

[0192] : Mask regularization loss;

[0193] Reconstruction error:

[0194] After obtaining the properties of the Gaussian scene, we render the scene to get the rendered image, compare the rendered image with the collected RGB image, and calculate the reconstruction error. The reconstruction error is calculated by the joint loss function based on L1 norm, SSIM and Total Variation. To measure, the loss function is defined as follows:

[0195]

[0196] in is the pixel-by-pixel L1 norm difference between the rendered image and the real image; It is a loss function based on structural similarity, which is used to measure the structural fidelity of the image; It is a total variation regularization term on the voxel grid, which is used to suppress the non-smooth changes of the predicted attributes in the spatial and temporal dimensions; It is the weighted coefficient of the loss term, and the hyperparameter is set according to the task weight;

[0197] Mask regularization loss:

[0198] Introducing mask regularization loss during training , which is used to constrain the total number of retained Gaussian volumes and balance the reconstruction quality with the compression rate:

[0199]

[0200] in is the weight coefficient, which represents the total Gaussian budget that the model expects to use;

[0201] Compression entropy loss

[0202] In order to control the balance between compression ratio and reconstruction quality, compression entropy loss is introduced :

[0203]

[0204] D2. Network Optimization

[0205] Training data preparation:

[0206] The input data is a sequence of RGB images taken from different perspectives at multiple time frames, which serves as the system's supervisory signal, i.e., the real image, and also provides the camera's internal and external parameters for the corresponding frame.

[0207] Model forward reasoning;

[0208] For each training sample, after converting it into a Gaussian volume set, it is processed according to step S6011 and rendered according to step S6012;

[0209] Loss function calculation:

[0210] Compare the rendered image with the captured RGB image and calculate the joint loss function;

[0211] Parameters involved in training optimization:

[0212] Then, a dynamic Gaussian coding system is used to optimize the parameters of each submodule through end-to-end training, including:

[0213] Gaussian properties , Multi-resolution binary hash grid , deformation prediction network , mask module and deep entropy model networks in ; Among them, the Gaussian properties are obtained by position information and optimization Construct the optimized Gaussian body.

[0214] Gradient calculation and parameter update:

[0215] The back-propagation algorithm is used to calculate the gradient of the total loss with respect to the parameters of each module of the model, and the Adam optimizer is used to perform gradient descent updates to gradually optimize the model weight parameters.

[0216] Iterative optimization at different times and shooting angles:

[0217] Repeat the above steps until the loss function converges and the model achieves optimal performance in Gaussian attribute prediction, mask estimation, and entropy modeling.

[0218] In the embodiment of the present application, the network optimization adopts a progressive strategy of "freezing deformation first and then optimizing in linkage", optimizing the attribute convergence on the static frame only in the first 20% of the iteration period, and then introducing the time dimension deformation update in the subsequent iteration period. , improve dynamic consistency;

[0219] Initial stage:

[0220] Without introducing the deformable network in step S303, the model only optimizes the initial properties of the three-dimensional Gaussian volume in the static frame. The training samples are uniformly sampled from 20% of the frames in the time series. By not introducing deformation in the time dimension, the Gaussian properties and distribution between frames are stably converged, reducing the difficulty of initial training of the deformable network.

[0221] Linkage phase:

[0222] The deformable network module is introduced, and the timestamp is used as input to jointly optimize the attribute prediction network and the deformable network. At this stage, the model begins to learn the continuous change law of Gaussian attributes in the temporal dimension, realizing the modeling of space-time consistency.

[0223] The step S7 includes the following sub-steps:

[0224] S701. Video Slicing and Frame Grouping:

[0225] The complete dynamic 3D Gaussian scene is divided into several transmission blocks according to time, and each transmission block covers a fixed number of frames. , recorded as:

[0226]

[0227] in is the total number of transmission blocks;

[0228] S702. Determine the bit rate level and parameter configuration:

[0229] Set each transport block to support bitrate versions, respectively recorded as:

[0230]

[0231] in Indicates the The first transmission block Level bitrate version;

[0232] S703. Multi-rate model generation:

[0233] For different bit rates, by specifying different , according to step S6, the Gaussian sets corresponding to different bit rates are obtained , deep entropy model network, deformation prediction network, mask module, binary multi-resolution hash grid;

[0234] S704. The network conditions are divided into multiple categories, and a bit rate is set for each type of network condition (the network condition is defined by the network speed, and the network speed is divided into different intervals, each interval corresponds to a bit rate, the higher the network speed, the higher the bit rate);

[0235] During data transmission, the network status is detected in real time, and the bit rate corresponding to the current network status is selected, as well as the deep entropy model network, deformation prediction network and mask module corresponding to the bit rate;

[0236] S705. The transmitter adaptively selects the optimal bitrate version of the dynamic Gaussian model based on the network conditions at time t, including: a Gaussian volume set, a deep entropy model network, a deformation prediction network, a mask module, and a binary multi-resolution hash grid; the Gaussian volume set is initialized by the first frame of the video stream to be transmitted according to step S1, and the parameters are trained and optimized according to step S6 to obtain the final Gaussian volume set;

[0237] The Gaussian set is passed through the deep entropy model network to obtain a Gaussian distribution, and based on the Gaussian distribution, the compression properties of each Gaussian in the Gaussian set are compressed into a bit stream using the arithmetic encoder; the Bernoulli distribution of the binary multi-resolution hash grid is calculated and the grid is compressed into a bit stream using the arithmetic encoder; the mask module parameters are directly transmitted without processing; the spatial position Due to the sensitivity to accuracy, lossless compression transmission is performed separately;

[0238] The sending end transmits the mask module parameters, the compressed bit stream, and the spatial position to the receiving end. The receiving end decompresses the bit stream using an arithmetic decoder, recovers the Gaussian volume set and the binary multi-resolution hash grid according to step S506, receives the mask module parameters, and performs image rendering at time t.

[0239] S706. When the transmitting end sends a stereoscopic video stream composed of a dynamic Gaussian model, step S705 is repeated at each moment, and the receiving end receives a continuous stereoscopic video stream and plays it.

[0240] The foregoing description shows and describes a preferred embodiment of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Instead, the present invention is applicable to various other combinations, modifications, and environments and is capable of modification within the scope of the inventive concept described herein, through the teachings above, or through techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. A video stream processing method with dynamic Gaussian compression and adaptive bit rate control, characterized by: The following steps are involved: S1. Gaussian attribute modeling and initialization: The dynamic 3D scene sequence is converted into a 3D Gaussian volume set representation, where each Gaussian volume is used to simulate the local properties of the object surface; S2. Based on a multi-resolution binary hash grid, construct a two-dimensional feature mapping model that maps from two-dimensional spatiotemporal positions to two-dimensional composite feature vectors, and a three-dimensional feature mapping model that maps from three-dimensional spatial features to two-dimensional composite feature vectors; S3. Constructing a deformation prediction network: Based on binary spatial feature vectors, a composite feature representation that integrates space and time is constructed. This feature is then fed into a multi-head neural network to predict the temporal increments of 3D Gaussian volume attributes, thereby enabling continuous modeling of point attributes in dynamic 3D scenes. S4. Mask pruning mechanism design: By constructing a spatiotemporal-sensitive learnable mask module, we automatically identify and remove Gaussian volumes that have a low impact on reconstruction quality in the spatial or temporal dimensions. S5. Entropy Modeling and Arithmetic Encoding and Decoding Module Design: By building a deep entropy model network, we probabilistically model the distribution of continuous attributes. We compress continuous attributes and discrete values ​​into bitstreams using an arithmetic encoder and restore them using an arithmetic decoder. S6. Perform image rendering and use RGB images collected from multiple perspectives as supervision signals to jointly train the deformation prediction network, mask module, and deep entropy model network; S7. Design a multi-bitrate video streaming transmission scenario, and obtain the deformation prediction network and mask module corresponding to different bitrates by adjusting the number of Gaussian volumes and the feature grid accuracy. During the video transmission process, switch the video transmission bitrate based on the network quality for data transmission.

2. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 1, characterized in that: The step S1 includes the following sub-steps: S101. Dynamic scene preprocessing: The positions of multiple cameras are fixed during synchronous shooting. The first frame of the multi-view motion video sequence captured by the camera is sparsely reconstructed using the motion recovery structure algorithm to obtain the 3D point cloud and camera pose, and to construct the initial sparse point cloud frame set. ; S102. Initial configuration of Gaussian body: Based on the initial sparse point cloud frame set Each point in , initialize a three-dimensional Gaussian body , whose initial attribute set is defined as follows: Spatial location ; RGB color , represents the color of the Gaussian body; Rotation Quaternion , used to represent The orientation of the Gaussian body at the moment; Zoom factor , respectively control Variance in direction; Opacity , used to control the illumination superposition process; The initial Gaussian set is defined as: in is the number of initialized Gaussian bodies; the parameters of the Gaussian bodies are continuously optimized and adjusted during the subsequent training process; S103. Gaussian Volume Merging and Normalization: Spatial proximity merging: cluster and merge Gaussian bodies within the local spatial radius δ, and use weighted average for the merged attributes; Color normalization: normalize the color Normalize to the interval [0,1] to unify the input scale; Normalize the attitude: Rotate the quaternion Normalize to ensure it is a valid quaternion.

3. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 1, characterized in that: In step S2, the multi-resolution binary hash grid includes a resolution binary hash grid for a two-dimensional feature mapping model and a resolution binary hash grid for a three-dimensional feature mapping model; According to the resolution binary hash grid used for the two-dimensional feature mapping model, a two-dimensional feature mapping model is constructed from the two-dimensional spatiotemporal position mapping to the two-dimensional composite feature vector, including: A1. Initialize the resolution binary hash grid for the 2D feature map model: Build The hash grid structure with increasing layer resolution is denoted as , where each layer Contains hash table entries , used to store binary features, each hash table item is a key-value pair ,in: is the hash result of the two-dimensional integer coordinates; is a binary feature component; A2. Hash function and resolution setting: Set the corresponding resolution for each layer of hash grid , whose values ​​are incremented as follows: in is the minimum layer resolution; A3. Construct a two-dimensional feature mapping model from two-dimensional spatiotemporal position mapping to a two-dimensional composite feature vector: For any two-dimensional space-time position , first use the following hash function to convert the space-time coordinates Map to Integer keys on the layer hash grid : Then find the integer coordinates corresponding to the adjacent k value, will The values ​​are arranged into initial vectors in ascending order, and bilinear interpolation is used to obtain smooth eigenvectors. , repeat the above process and calculate the layer feature vector; Then concatenate the features of all layers into a composite feature representation: in, ; represents the dimension of the composite feature, Indicates the Smoothed feature vector obtained by layer hash table Dimensions; According to the resolution binary hash grid used for the three-dimensional feature mapping model, a three-dimensional feature mapping model is constructed from the three-dimensional spatiotemporal position mapping to the two-dimensional composite feature vector, including: B1. Initialize the resolution binary hash grid for the 3D feature map model: Build The hash grid structure with increasing layer resolution is denoted as , where each layer Contains hash table entries , used to store binary features, each hash table item is a key-value pair ,satisfy: is the hash result of the three-dimensional integer coordinates; is a binary feature component; B2. Hash function and resolution setting: Set the corresponding resolution for each layer of hash grid , whose values ​​are incremented as follows: in is the minimum layer resolution; B3. Construct a 3D feature mapping model that maps 3D spatiotemporal positions to 2D composite feature vectors: For any input position , use the following hash function to convert the floating point coordinates Maps to integer keys : Then find the integer coordinates corresponding to the adjacent k value, will The values ​​are arranged into initial vectors in ascending order, and trilinear interpolation is used to obtain smooth eigenvectors. , repeat the above process and calculate the Layer feature vector; then concatenate the features of all layers into a composite feature representation: in, ; represents the dimension of the composite feature, Indicates the Smoothed feature vector obtained by layer hash table dimension.

4. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 1, characterized in that: The step S3 includes the following sub-steps: S301. Input feature construction: On the basis of the three-dimensional coordinate system, a time t coordinate axis is added, and for each Gaussian body, its spatial position is , timestamp is , t is a real number, construct a four-dimensional joint input vector ;The input vector is projected onto the following set of 2D planes: Space Plane: 、 、 ; Space-time plane: 、 、 ; S302. Planar Feature Extraction and Fusion: S3021. Use the two-dimensional feature mapping model on each plane to obtain the composite features on the plane, denoted as ,in, ; S3022. Use the three-dimensional feature mapping model on the entire space to obtain the composite features of the space, which are recorded as three-dimensional network features ; S3023. Perform Hadamard product fusion on all plane features and combine them with 3D mesh features Splicing to construct a multi-resolution composite feature vector: in Gaussian body position point set, is a plane set; S303. Deformation Prediction Network Design: Will Input to the multi-head MLP network ,in Represent the branch network parameters of position, color, rotation, scale, and opacity respectively; Each branch network is used to predict five types of attributes, including: Position increment: ; Color Increment: ; Rotation increment: ; Zoom increment: ; Opacity Increment: ; The final output attributes are updated as follows: 。 5. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 4, characterized in that: The step S4 includes the following sub-steps: S401. Mask parameter initialization: For each Gaussian Initialize a learnable saliency parameter , used to calculate the contribution of the Gaussian to the final image reconstruction. This parameter is used as the input of the neural network for training and updating; S402. Constructing a mask prediction network: Build a small MLP as a mask predictor , the input is , the output is the masked activation value: in represents the sigmoid activation function; S403. Mask Binarization and Differentiable Processing: Using output activation and set threshold Compare and calculate the binary mask : in: is an indicator function, if , the indicator function takes 1, if it is not satisfied, the indicator function takes 0, Indicates stopping gradient propagation; S404. Mask applied to Gaussian properties: The mask generated Applied to various Gaussian attribute encoding processes: like , then the Gaussian body is retained to participate in deformation, encoding and rendering; like , then the Gaussian body is eliminated in the current frame and does not participate in rendering and compression.

6. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 5, characterized in that: The step S5 includes the following sub-steps: S501. Attribute set division: The original Gaussian attributes are classified according to whether they are compressible or not, and the attribute set to be compressed is defined as: Spatial location Due to the sensitivity to precision, lossless compression is performed separately; S502. Construct context feature vector: Construct context information for each Gaussian , as the input of the deep entropy model network, where 3D mesh features: S503. Conditional Probability Distribution Modeling: The conditional probability distribution is calculated for the continuous-valued attributes of the Gaussian volume and the binary multi-resolution hash grid for subsequent arithmetic coding compression: S5031. For continuous attributes: building a deep entropy model network , which consists of several layers of MLP, and its parameters are Indicates that the input is , used to predict attribute values The conditional probability distribution of attribute value Including color, scaling, rotation, transparency: predict its mean and standard deviation , construct Gaussian distribution, recorded as: Optimize during subsequent training; S5032. For discrete values, including binary multi-resolution hash grids, the percentage is calculated based on the number of binary feature components +1 and -1, and a Bernoulli distribution is constructed: S504. Arithmetic encoder compression process: S5041. Continuous Attribute Compression: Arithmetic Encoder Using Deep Entropy Model Network The obtained Gaussian distribution , for attribute values Perform compression; Repeat the above process to encode each attribute component independently; finally, output the bitstream segments of all attributes to form the overall compressed bitstream; S5042. Discrete Value Compression: The arithmetic encoder uses the Bernoulli distribution in S5032 as the probability model input; it compresses the discrete values, i.e., the multi-resolution binary hash grid: All symbol values ​​are encoded independently, ultimately generating a compressed bitstream; S505 arithmetic decoder to restore the compressed bit stream process: the arithmetic decoder decodes the continuous attributes and discrete values ​​?? respectively; S5051. The arithmetic decoder recovers the continuous value attribute of the Gaussian body and restores the decoded Gaussian body according to the attribute value obtained by decoding; S5052. The arithmetic decoder decodes the discrete values ​​and restores the binary multi-resolution hash grid.

7. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 6, characterized in that: The step S6 comprises: S601. Model building and rendering process: The process input timestamp And camera parameters, camera parameters include intrinsic parameters and extrinsic parameters, output rendered images, including: S6011. Calculate the Gaussian volume set at time t: For the current Gaussian body set, according to the camera parameters, remove the Gaussian bodies outside the viewing cone, that is, the invisible Gaussian bodies, and obtain the remaining Gaussian body set ; Gaussian set The compression properties in the deep entropy model network are Gaussian distributed, and based on the Gaussian distribution, the compression properties of each Gaussian in the arithmetic encoder Gaussian set are compressed into a bit stream, and the spatial position Due to the sensitivity to precision, lossless compression is performed separately, and the compression properties of each Gaussian volume are restored through an arithmetic decoder to obtain a restored Gaussian volume set; The position information of the Gaussian body in the restored Gaussian body set and time Input into the deformation prediction network to calculate the deformation field ,Will Acts on The attribute value of is deformed to obtain Gaussian set at the moment ; Input time Go to the mask module to obtain the mask value corresponding to each Gaussian body, remove the Gaussian bodies below the threshold, and obtain the Gaussian body set after removing the threshold ; S6012. Remove the Gaussian volume set after the threshold is removed Rendering to RGB image: Depth information is calculated based on the camera projection plane and Gaussian volume position information, the three-dimensional Gaussian ellipsoids are sorted and pixel projection weights are determined; using the rasterization algorithm, the surface points of the sorted ellipsoids are projected onto a two-dimensional plane according to their positions and projection weights to obtain the rendered RGB image; S602. Optimize the parameters of each submodule, including the deformation prediction network, mask module, and deep entropy model network, through end-to-end training using a dynamic Gaussian coding system. The training process relies on RGB images collected from multiple views as supervision signals. The specific process includes joint optimization goal design, network optimization, and training strategy. D1. Joint optimization objective design: The deformation prediction network, mask module and deep entropy model network participate in joint training, and the optimization goal is the joint loss function : in: : compression entropy loss; : reconstruction error; : loss balance coefficient; : Mask regularization loss Reconstruction error: After obtaining the properties of the Gaussian scene, we render the scene to get the rendered image, compare the rendered image with the collected RGB image, and calculate the reconstruction error. The reconstruction error is calculated by the joint loss function based on L1 norm, SSIM and Total Variation. To measure, the loss function is defined as follows: in is the pixel-by-pixel L1 norm difference between the rendered image and the real image; It is a loss function based on structural similarity, which is used to measure the structural fidelity of the image; It is a total variation regularization term on the voxel grid, which is used to suppress the non-smooth changes of the predicted attributes in the spatial and temporal dimensions; It is the weighted coefficient of the loss term, and the hyperparameter is set according to the task weight; Mask regularization loss: Introducing mask regularization loss during training , which is used to constrain the total number of retained Gaussian volumes and balance the reconstruction quality with the compression rate: in is the weight coefficient, which represents the total Gaussian budget that the model expects to use; Compression entropy loss In order to control the balance between compression ratio and reconstruction quality, compression entropy loss is introduced : D2. Network Optimization Training data preparation: The input data is a sequence of RGB images taken from different perspectives at multiple time frames, which serves as the system's supervisory signal, i.e., the real image, and also provides the camera's internal and external parameters for the corresponding frame. Model forward reasoning; For each training sample, after converting it into a Gaussian volume set, it is processed according to step S6011 and rendered according to step S6012; Loss function calculation: Compare the rendered image with the captured RGB image and calculate the joint loss function; Parameters involved in training optimization: The dynamic Gaussian coding system is used to optimize the parameters of each submodule through end-to-end training, including: Gaussian properties , Multi-resolution binary hash grid , deformation prediction network , mask module and deep entropy model networks in ; Among them, the Gaussian properties are obtained by position information and optimization Construct an optimized Gaussian body; Gradient calculation and parameter update: The back-propagation algorithm is used to calculate the gradient of the total loss with respect to the parameters of each module of the model, and the Adam optimizer is used to perform gradient descent updates to gradually optimize the model weight parameters. Iterative optimization at different times and shooting angles: The above steps are repeated until the loss function converges and the model achieves optimal performance in Gaussian attribute prediction, mask estimation, and entropy modeling.

8. The video stream processing method with dynamic Gaussian compression and adaptive bit rate control according to claim 1, characterized in that: The step S7 includes the following sub-steps: S701. Video Slicing and Frame Grouping: The complete dynamic 3D Gaussian scene is divided into several transmission blocks according to time, and each transmission block covers a fixed number of frames. , recorded as: in is the total number of transmission blocks; S702. Determine the bit rate level and parameter configuration: Set each transport block to support bitrate versions, respectively recorded as: in Indicates the The first transmission block Level bitrate version; S703. Multi-rate model generation: For different bit rates, by specifying different , according to step S6, obtain Gaussian sets, deep entropy model networks, deformation prediction networks, mask modules, and binary multi-resolution hash grids corresponding to different bit rates; S704. Divide the network conditions into multiple categories, and set a bit rate for each network condition; During data transmission, the network status is detected in real time, and the bit rate corresponding to the current network status is selected, as well as the deep entropy model network, deformation prediction network and mask module corresponding to the bit rate; S705. The transmitter adaptively selects the optimal bitrate version of the dynamic Gaussian model based on the network conditions at time t, including: a Gaussian volume set, a deep entropy model network, a deformation prediction network, a mask module, and a binary multi-resolution hash grid; the Gaussian volume set is initialized by the first frame of the video stream to be transmitted according to step S1, and the parameters are trained and optimized according to step S6 to obtain the final Gaussian volume set; The Gaussian set is passed through the deep entropy model network to obtain a Gaussian distribution, and based on the Gaussian distribution, the compression properties of each Gaussian in the Gaussian set are compressed into a bit stream using the arithmetic encoder; the Bernoulli distribution of the binary multi-resolution hash grid is calculated and the grid is compressed into a bit stream using the arithmetic encoder; the mask module parameters are directly transmitted without processing; the spatial position Due to the sensitivity to accuracy, lossless compression transmission is performed separately; The sending end transmits the mask module parameters, the compressed bit stream, and the spatial position to the receiving end. The receiving end decompresses the bit stream using an arithmetic decoder, recovers the Gaussian volume set and the binary multi-resolution hash grid according to step S506, receives the mask module parameters, and performs image rendering at time t. S706. When the transmitting end sends a stereoscopic video stream composed of a dynamic Gaussian model, step S705 is repeated at each moment, and the receiving end receives a continuous stereoscopic video stream and plays it.

Images