A dynamic 3D grid video transmission method based on embedded deformation

Abstract

The present invention relates to the field of three-dimensional mesh video transmission, and more specifically to a dynamic three-dimensional mesh video transmission method based on embedded deformation. The solution includes: at the encoding end, dividing the original dynamic mesh sequence into multiple frame groups, each frame group containing a complete key frame and multiple predicted frames, and encoding the geometric data of the key frames using Huffman compression; for each predicted frame, sampling m control nodes on the mesh surface, and constructing a control node graph G, fitting the rigid body transformation parameters of each node on the control node graph, where is a rotation matrix and is a translation vector; during network transmission, using a sliding window to predict the bandwidth upper limit of a future set time, and dynamically scheduling based on the prediction results; at the decoding end, restoring the received predicted frames. The present invention is suitable for three-dimensional mesh video transmission.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional grid video transmission, and in particular to a dynamic three-dimensional grid video transmission method based on embedded deformation. Background Art

[0002] Conventional volumetric videos include point clouds, 3D meshes, and neural network-based 3D rendering technologies. Point cloud transmission uses discrete points to represent 3D scenes. This results in small data volumes but low visual quality, and often results in "holes" when zoomed in due to a lack of surface continuity. Neural network-based 3D rendering technologies, such as neural radiance fields and 3D Gaussian splatter representation, can achieve high-quality, high-fidelity reconstruction effects. However, these solutions have high computational overhead, require high-performance GPUs for training and rendering, and cannot be applied in real time. They also require large amounts of data and require the transmission of complete model data, resulting in high bandwidth requirements. Traditional 3D mesh transmission uses vertices, edges, and faces to represent 3D surfaces, supporting high-precision rendering. However, this solution cannot utilize inter-frame deformation information, resulting in wasted bandwidth, and cannot dynamically adjust transmission strategies based on network fluctuations, making it prone to lag. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a dynamic three-dimensional grid video transmission method based on embedded deformation, which reduces redundant data transmission and ensures stable and smooth playback under network fluctuations.

[0004] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. The present invention provides a dynamic three-dimensional grid video transmission method based on embedded deformation, comprising:

[0005] S1. At the encoding end, the original dynamic mesh sequence is divided into multiple frame groups. Each frame group contains a complete key frame and multiple predicted frames. The geometric data of the key frame is encoded using Huffman compression, and the texture sequence is compressed using the H.264 video compressor.

[0006] S2. For each prediction frame, sample m control nodes on the grid surface , and construct the control node graph G, fitting the rigid body transformation parameters of each node on the control node graph , where represents the rotation matrix of the j-th control node during the deformation process, represents the translation vector of the j-th control node during the deformation process;

[0007] S3: During network transmission, the bandwidth limit for the future set time is predicted through a sliding window, and dynamic scheduling is performed based on the prediction results.

[0008] S4. At the decoding end, for the received prediction frame, the vertex positions are restored in the following manner:

[0009] ;

[0010] Where, is the reconstructed position of vertex i in the current frame, is the weight of the control node corresponding to this point, Indicates the location of the control node, To predict the position of the vertex.

[0011] Furthermore, the transformation parameters are obtained by optimizing the following energy function:

[0012] ;

[0013] Where, , Indicates the geometric difference between the deformed vertex and the target frame, represents the deformed vertex, Indicates the corresponding vertex in the target frame;

[0014] , Indicates that the constrained rotation matrix is ​​approximately orthogonal, and T represents the transpose of the matrix;

[0015] , represents the deformation continuity of adjacent control nodes, represents the initial position coordinates of the kth control node, represents the rotation matrix of the kth node during the deformation process, represents the translation vector of the kth node during the deformation process;

[0016] They are 、 、 The weighting coefficient of

[0017] The optimized rigid body transformation parameters are compressed through entropy coding to form a predicted frame output.

[0018] Furthermore, in step S3, dynamic scheduling according to the prediction result specifically includes:

[0019] The Lyapunov algorithm is used to find the optimal coding configuration under the predicted bandwidth limit to maximize the global QoE function. The QoE function is designed as follows:

[0020] ;

[0021] Where, represents the reconstruction quality score of the i-th frame, represents the reconstruction quality score of the i-1th frame, represents the quality fluctuation term, Indicates playback delay; are the corresponding experience weights respectively.

[0022] Furthermore, in step S4, the normal vector is updated at the decoding end in the following manner:

[0023] , represents the original normal vector of vertex i in the reference mesh keyframe, represents the updated normal vector obtained after the weighted rigid body transformation of the predicted frame, and T represents the transpose of the matrix.

[0024] The beneficial effects of the present invention are:

[0025] By introducing an inter-frame deformation-driven compression method, this invention transmits only key frames and lightweight deformation parameters (rotation matrices and translation vectors), significantly reducing redundant data transmission and achieving a compression rate exceeding 90%. Furthermore, this invention constructs a QoE-based perception model based on Lyapunov optimization. Combined with bandwidth prediction, it adjusts key frame and predicted frame strategies in real time to ensure stable and smooth playback despite network fluctuations. On the client side, GPU-based parallel-accelerated reconstruction and texture mapping achieve low-latency rendering at 30 FPS, supporting adaptive strategy configuration in a variety of scenarios, and demonstrating extremely high transmission efficiency, flexibility, and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a dynamic three-dimensional grid video transmission method based on embedded deformation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] The present invention provides a dynamic three-dimensional grid video transmission method based on embedded deformation, such as Figure 1 As shown in the figure, it includes three stages: encoding-side processing, bandwidth-aware network scheduling, and decoding-side reconstruction. Its goal is to ensure a balance between the compression rate, real-time performance, and reconstruction quality of the 3D video stream under bandwidth-limited and network-fluctuating scenarios.

[0029] On the encoding side, the system first divides the original dynamic mesh sequence into several frame groups, each of which contains a complete key frame and multiple predicted frames. The key frame's geometric data is encoded using Huffman compression, and the texture sequence is compressed using the H.264 video compressor.

[0030] For each prediction frame, m control nodes are sampled on the mesh surface. , and construct a control node graph, fitting the rigid body transformation parameters of each node on the control node graph , where represents the rotation matrix of the j-th control node during the deformation process, Represents the translation vector of the j-th control node during the deformation process.

[0031] The rigid body transformation parameters are obtained by optimizing the following energy function:

[0032] , represents the energy function, They are 、 、 The weighting coefficient of

[0033] Where, , Indicates the geometric difference between the deformed vertex and the target frame, represents the deformed vertex, Indicates the corresponding vertex in the target frame;

[0034] , Indicates that the constraint rotation matrix is ​​approximately orthogonal, represents the identity matrix, the superscript T represents the transpose of the matrix, Represents the rotation matrix The transposed matrix of

[0035] , represents the deformation continuity of adjacent control nodes, where is the local position of the control point, represents the initial position coordinates of the kth control node, represents the rotation matrix of the kth node during the deformation process, represents the translation vector of the kth node during the deformation process;

[0036] Optimized transformation parameters The predicted frame output is compressed through entropy coding (such as Huffman coding).

[0037] During network transmission, the bandwidth upper limit of the future set time is predicted through a sliding window, and dynamic scheduling is performed based on the prediction results.

[0038] The Lyapunov algorithm is used to find the optimal coding configuration (i.e., frame compression strategies with different numbers of control nodes) under the predicted bandwidth limit to maximize the global QoE function. The QoE function is designed as follows:

[0039] ;

[0040] Where, Indicates the reconstruction quality score of the i-th frame. The i-th frame can be a key frame or a predicted frame in the display order. represents the reconstruction quality score of the i-1th frame, represents the quality fluctuation term, Indicates playback delay; are the corresponding empirical weights respectively. Here, the QoE function compares the reconstruction quality of the current frame with that of the previous frame, regardless of the frame type.

[0041] At the decoding end, for the received prediction frame, the vertex position is restored in the following way:

[0042] ;

[0043] Where, is the reconstructed position of vertex i in the current frame, is the weight of the control node corresponding to this point, Indicates the location of the control node, To predict the position of the vertex.

[0044] On the decoder side the normal vector is updated as follows:

[0045] , represents the original normal vector of vertex i in the reference mesh keyframe, Represents the updated normal vector obtained after the predicted frame undergoes weighted rigid body transformation, which is used for final rendering.

[0046] The keyframes are directly decompressed at the decoding end to obtain the complete mesh and texture, without the need for pose recovery based on control nodes.

[0047] The decoding process supports GPU acceleration and adapts to real-time 30FPS rendering requirements. The final reconstructed mesh is rendered and output through a rendering engine (such as OpenGL / Vulkan).

[0048] Tests of the present invention on multiple real dynamic 3D datasets (such as FaceCap, Coaster, and HandStand) show that, while maintaining acceptable visual quality, only 1.6% to 13.4% of the original frame data volume needs to be transmitted, which is far superior to traditional frame-by-frame compression methods (such as Draco+HEVC or VView), significantly reducing transmission bandwidth.

[0049] In a simulation environment where network bandwidth changes dynamically, the present invention can effectively avoid rebuffering, and the comprehensive QoE score is improved by 13.6% to 22.4% compared with the baseline, ensuring a more coherent user viewing experience.

[0050] The present invention supports real-time operation of end-side decoding at 30FPS and is scalable. It can run smoothly on client devices of different complexities and meet the real-time requirements of immersive interactive applications.

[0051] The present invention performs stably in both subjective and objective visual indicators (such as PSNR and Hausdorff distance), ensuring that no obvious distortion or visual jump is introduced while significantly compressing the transmission.

[0052] In summary, the present invention is superior to existing technologies in terms of bandwidth saving, stable quality, smooth decoding and network adaptability. It has wide practicality and industrialization potential, and is particularly suitable for application scenarios such as VR / AR and remote interaction that have high requirements for real-time and immersiveness.

[0053] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A dynamic three-dimensional grid video transmission method based on embedded deformation, characterized in that: include: S1. At the encoding end, the original dynamic mesh sequence is divided into multiple frame groups. Each frame group contains a complete key frame and multiple predicted frames. The geometric data of the key frame is encoded using Huffman compression, and the texture sequence is compressed using the H.264 video compressor. S2. For each prediction frame, sample m control nodes on the grid surface , and construct the control node graph G, fitting the rigid body transformation parameters of each node on the control node graph ,in represents the rotation matrix of the j-th control node during the deformation process, represents the translation vector of the j-th control node during the deformation process; S3: During network transmission, the bandwidth limit for the future set time is predicted through a sliding window, and dynamic scheduling is performed based on the prediction results. S4. At the decoding end, for the received prediction frame, the vertex positions are restored and the normal vectors are updated at the decoding end; The vertex position is restored as follows: ； Where, is the reconstructed position of vertex i in the current frame, is the weight of the control node j corresponding to the vertex, Indicates the location of the control node, To predict the position of the vertex; The normal vector is updated as follows: , represents the original normal vector of vertex i in the reference mesh keyframe, represents the updated normal vector obtained after the weighted rigid body transformation of the predicted frame, and T represents the transpose of the matrix.

2. The method for dynamic three-dimensional grid video transmission based on embedded deformation according to claim 1, characterized in that: The transformation parameters are obtained by optimizing the following energy function: , represents the energy function, They are 、 、 The weighting coefficient of Where, , Indicates the geometric difference between the deformed vertex and the target frame, represents the deformed vertex, Indicates the corresponding vertex in the target frame; , Indicates that the constraint rotation matrix is ​​approximately orthogonal, T represents the transpose of the matrix, represents the identity matrix; , represents the deformation continuity of adjacent control nodes, represents the initial position coordinates of the kth control node, represents the rotation matrix of the kth control node during the deformation process, represents the translation vector of the kth control node during the deformation process; The optimized rigid body transformation parameters are compressed through entropy coding to form a predicted frame output.

3. The method for dynamic three-dimensional grid video transmission based on embedded deformation according to claim 1, characterized in that: In step S3, dynamic scheduling based on the prediction results specifically includes: The Lyapunov algorithm is used to find the optimal coding configuration under the predicted bandwidth limit to maximize the global QoE function. The QoE function is designed as follows: ； Where, represents the reconstruction quality score of the i-th frame, represents the reconstruction quality score of the i-1th frame, represents the mass fluctuation term, Indicates playback delay; are the corresponding experience weights respectively.

Images