Multi-layer residual compression method for six-degree-of-freedom video

Abstract

The invention relates to the technical field of video coding, in particular to a multi-layer residual compression method for a six-degree-of-freedom video, which comprises the following steps of: carrying out space and visual angle joint sampling on an original six-degree-of-freedom video, and constructing a multi-visual angle texture and depth image set; constructing a self-adaptive multi-layer residual error coding structure; executing cross-modal joint residual modeling in each layer, extracting the spatial gradient correlation between the texture and the depth map through a shared convolution kernel, and generating a joint residual signal; performing time domain prediction of motion and view angle consistency constraint in combination with six-degree-of-freedom motion parameters, and performing motion compensation and view angle interpolation optimization on residual signals; and dynamically allocating a code rate by adopting a hierarchical rate distortion optimization strategy, and outputting a compressed code stream. According to the method, the compression efficiency and the reconstruction quality are improved, the picture continuity and the structural stability during view angle switching are guaranteed, and high-immersion real-time six-degree-of-freedom video interaction experience is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, and particularly relates to a multi-layer residual compression method for six-degree-of-freedom video. BACKGROUND

[0002] With the continuous evolution of immersive media technology, six-degree-of-freedom video is gradually becoming the core content form of the next generation of interactive video applications as a key carrier to realize the free movement and view switching of users in three-dimensional space. Six-degree-of-freedom video not only needs to record the visual information of any view in space, but also needs to support the continuous interaction of users in six dimensions of translation and rotation, which poses unprecedented challenges to the data size, structural organization and transmission efficiency of video.

[0003] The prior art generally adopts a fixed layered structure and static residual modeling, which is difficult to adapt to the non-uniform information distribution caused by the dynamic changes of views and the differences in geometric complexity in six-degree-of-freedom video, resulting in low compression efficiency; at the same time, the traditional residual coding does not fully consider the cross-modal correlation between depth maps and texture maps, causing geometric distortion and visual artifacts, which seriously affects the immersive experience. SUMMARY

[0004] The purpose of the present application is to provide a multi-layer residual compression method for six-degree-of-freedom video to solve the problems in the background art.

[0005] The present application provides a multi-layer residual compression method for six-degree-of-freedom video, comprising: spatial and view joint sampling of original six-degree-of-freedom video data, construction of a multi-view image set containing texture maps and depth maps, and dynamic division of spatial coding units based on user interaction trajectory prediction of view distribution hot spot areas; construction of an adaptive multi-layer residual coding structure, decomposition of the multi-view image set into a base layer and at least two enhancement layers, wherein the base layer covers the low-frequency geometry and texture information of all view areas, and the enhancement layers code high-frequency residual information according to view importance and geometric complexity; cross-modal joint residual modeling within each layer, synchronous processing of spatial gradient features of texture maps and depth maps, extraction of cross-modal correlation through shared convolution kernels, and generation of joint residual signals; temporal prediction of the joint residual signals with motion and view consistency constraints, construction of a view continuity model using six-degree-of-freedom motion parameters between adjacent frames, and implementation of motion compensation and view interpolation optimization on residual signals; quantization and entropy coding of the coding data of each layer using a layered rate-distortion optimization strategy, dynamic adjustment of the code rate allocation weight of each enhancement layer according to bandwidth constraints, and output of compressed code streams.

[0006] In some embodiments, the space and view angle joint sampling adopts an octree space division strategy to divide a three-dimensional scene into cubic units with an edge length of 0.5 meters to 2 meters, and constructs a view angle probability heat map in combination with historical interaction data of a user, and the sampling density of a region with a heat value greater than 0.7 is encrypted to 2 times the original density.

[0007] In some embodiments, the base layer adopts full-view angle uniform sampling, and the sampling interval is one view angle every 15 degrees; the first enhancement layer covers a region with a heat value greater than 0.5, and the view angle interval is encrypted to one view angle every 5 degrees; the second enhancement layer focuses on a high interaction region with a heat value greater than 0.8, and the view angle interval is encrypted to one view angle every 2 degrees, and a local geometric detail enhancement module is introduced.

[0008] In some embodiments, the cross-modal joint residual modeling adopts a double-branch convolutional neural network structure, the texture branch and the depth branch share the first 3 layers of convolution kernels, the convolution kernel size is 3 by 3, and the channel number is 64, and the subsequent branches independently process the respective modal features, and generate a joint residual map through a channel attention mechanism to fuse the cross-modal residual responses.

[0009] In some embodiments, the motion and view angle consistency constraint time domain prediction is based on a six-degree-of-freedom motion parameter matrix, which includes 3 translation components and 3 rotation components, and generates a motion trajectory of an intermediate view angle between adjacent key frames through Lie group interpolation algorithm, and implements block matching compensation based on a motion vector field on a residual signal, and the compensation block size is 16 by 16 pixels.

[0010] In some embodiments, the hierarchical rate-distortion optimization strategy adopts a Lagrange multiplier method, sets the distortion weight coefficient of the base layer to 1.0, the first enhancement layer to 0.6, and the second enhancement layer to 0.3, and dynamically adjusts the quantization parameters of each layer according to real-time bandwidth feedback, and the quantization step size ranges from 2 to 32, and the step size adjustment granularity is 1.

[0011] In some embodiments, multi-layer residual reconstruction and view angle synthesis are performed at the decoding end, specifically: The base layer is decoded to generate an initial six-degree-of-freedom video; The enhancement layer residual signals are sequentially superimposed for detail enhancement; A depth map guided view angle rendering algorithm is used to generate an output picture of an arbitrary user requested view angle, and the rendering delay is less than 50 milliseconds.

[0012] In some embodiments, the depth map guided view angle rendering algorithm adopts a voxel traversal strategy based on ray casting, the traversal step size is 0.01 meters, edge preserving smoothing processing is performed on depth discontinuous regions, the smoothing window size is 5 by 5 pixels, and the edge detection threshold is a depth gradient greater than 0.1 meters per pixel.

[0013] In some embodiments, the original six-degree-of-freedom video data is derived from a multi-camera array acquisition system, the multi-camera array acquisition system includes no less than 32 synchronous cameras, the camera baseline distance is 0.3 meters to 1.5 meters, the acquisition frame rate is 30 frames per second, the depth map precision is plus or minus 2 millimeters, and the texture map resolution is 4096 by 2048 pixels.

[0014] Compared with the prior art, the present application has the following beneficial effects: By constructing a space and view angle joint sampling mechanism and an adaptive multi-layer residual coding structure, precise modeling of non-uniform information distribution in six-degree-of-freedom video is realized, and compression efficiency is significantly improved. By introducing cross-modal joint residual modeling, the internal correlation between texture maps and depth maps is effectively mined, geometric distortion and visual artifacts are suppressed, and reconstruction quality is improved. Through the motion and view angle consistency constraint temporal prediction mechanism, the picture continuity and structure stability of the user in the process of rapid movement or view angle switching are guaranteed, and the interactive immersion is enhanced; through the hierarchical rate-distortion optimization and dynamic code rate allocation strategy, different bandwidth conditions can be flexibly adapted to realize the optimal visual fidelity and interactive response performance under limited resources. The present application realizes collaborative optimization in compression ratio, reconstruction quality and interactive consistency, and provides reliable technical support for large-scale real-time application of six-degree-of-freedom video. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is the technical architecture diagram of the multi-layer residual compression method of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described below. Figure 1 The technical solutions in the embodiments of the present application will be described below.

[0018] EMBODIMENT First, the original six-degree-of-freedom video data is spatially and angularly sampled to construct a multi-view image set including texture maps and depth maps, and the view angle distribution hot spot area is predicted based on the user interaction track, and the spatial coding unit is dynamically divided.

[0019] Understandably, the original six-degree-of-freedom video data comes from a multi-camera array acquisition system, which contains no fewer than 32 synchronized cameras with a baseline distance of 0.3 to 1.5 meters, an acquisition frame rate of 30 frames per second, a depth map accuracy of ±2 millimeters, and a texture map resolution of 4096 x 2048 pixels.

[0020] The joint sampling process of space and perspective is as follows: the three-dimensional scene space is divided into several cubic units, and an octree space partitioning strategy is adopted. The initial partitioning unit side length is set to 1 meter, and adaptive subdivision is performed according to the geometric complexity of the scene. Finally, the unit side length range is controlled between 0.5 meters and 2 meters.

[0021] Within each spatial cell, the multi-camera array acquisition system simultaneously acquires texture maps and depth maps from different perspectives, forming a local multi-view image set. At the same time, the multi-camera array acquisition system continuously records and analyzes the user's historical interaction trajectory data, including translation coordinates and rotation angle sequences. A view probability heatmap is constructed using a kernel density estimation method. The heatmap resolution is aligned with the spatial cell grid, and the heat value of each grid point represents the probability that the spatial area will be visited in future interactions.

[0022] For high-probability areas with thermal values ​​greater than 0.7, the multi-camera array acquisition system automatically densifies the sampling density in the area to twice the original density. This means increasing the number of camera viewpoints or improving image acquisition resolution within the same spatial unit, thereby ensuring that hotspot areas receive more refined data representation. This dynamic partitioning mechanism makes the granularity of spatial coding units highly coupled with user interaction behavior, avoiding redundant sampling of low-interaction areas and significantly reducing the total amount of original data.

[0023] Next, an adaptive multi-layer residual coding structure is constructed, which decomposes the multi-view image set into a base layer and at least two enhancement layers. The base layer covers the low-frequency geometric and texture information of all view regions, and the enhancement layers encode high-frequency residual information hierarchically according to view importance and geometric complexity.

[0024] Understandably, the adaptive multi-layer residual coding structure receives and outputs a set of multi-view images and the corresponding view probability heatmaps to generate a base layer. The base layer adopts a uniform sampling strategy across all views, with a fixed view interval of 15 degrees, covering a complete spherical space from 0 to 360 degrees horizontally and from -90 to +90 degrees vertically, ensuring that any user's view can be initially reconstructed through the base layer.

[0025] Furthermore, based on this, a first enhancement layer is constructed, which only covers areas with a view probability thermal value greater than 0.5, and the view interval is encrypted to one view every 5 degrees, in order to encode the high-frequency detail residuals of these moderately important areas.

[0026] Furthermore, based on this, a second enhancement layer is constructed. The second enhancement layer focuses on high-interaction areas with thermal values ​​greater than 0.8, the view interval is encrypted to one view every 2 degrees, and a local geometric detail enhancement module is introduced. This module is specifically designed to handle minor undulations, edge sharpness, and high-frequency texture components in the depth map.

[0027] It should be noted that the content encoded by the enhancement layer is the residual signal relative to the reconstruction result of the lower layer, rather than the original signal. This hierarchical structure allows the coding resources to be dynamically tilted according to the importance of the viewpoint. High-interaction areas obtain richer detail representations, while low-interaction areas maintain basic availability at a lower bit rate.

[0028] Secondly, cross-modal joint residual modeling is performed within each layer, and the spatial gradient features of the texture map and depth map are processed simultaneously. Cross-modal correlations are extracted through shared convolution kernels, and joint residual signals are generated.

[0029] Understandably, cross-modal joint residual modeling employs a dual-branch convolutional neural network structure, processing the texture map and depth map separately. The two branches share the same convolutional kernel in the first three layers, with a kernel size of 3 by 3 and 64 channels. This sharing mechanism forces the network to learn common features of texture and depth in the low-level spatial structure, such as edge positions and surface normal directions.

[0030] After passing through the shared layer, the two branches independently extract deep features. The texture branch focuses on color, lighting, and high-frequency texture details, while the depth branch focuses on geometry, surface curvature, and depth discontinuities. Next, the feature maps from the two branches are fused using a channel attention mechanism. This mechanism calculates the importance weight of each channel, dynamically adjusting the contribution ratio of texture and depth features to residual generation. Finally, the fused feature map is fed into a lightweight decoder to generate a joint residual map.

[0031] The joint residual map includes high-frequency information of both texture and depth. Due to the explicit modeling of cross-modal correlation, it effectively avoids geometric distortion and visual artifacts caused by the independent processing of two modalities in traditional methods, such as texture blurring at depth edges or geometric collapse at texture details.

[0032] Next, temporal prediction with motion and viewpoint consistency constraints is performed on the joint residual signal. A viewpoint continuity model is constructed using the six-degree-of-freedom motion parameters between adjacent frames, and motion compensation and viewpoint interpolation optimization are performed on the residual signal.

[0033] Understandably, at the encoding end, a six-degree-of-freedom motion parameter matrix between adjacent video frames is simultaneously acquired or estimated. This matrix contains three translational components and three rotational components, accurately describing the rigid body motion of the scene or camera in three-dimensional space. Based on this matrix, a Lie group interpolation algorithm is used to generate smooth intermediate viewpoint motion trajectories between adjacent keyframes, ensuring the continuity of viewpoint transformations.

[0034] For the joint residual signal of the current frame, global motion compensation is first performed using the motion parameter matrix, projecting the residual signal of the reference frame onto the coordinate system of the current frame. Then, for regions with inconsistent local motion, block matching compensation based on the motion vector field is calculated, with each compensation block being 16 x 16 pixels. The residual prediction accuracy is further optimized by searching for the optimal matching block. The motion and viewpoint consistency constraint mechanism ensures that even in scenes with rapid user movement or drastic viewpoint changes, the reconstructed video maintains the spatiotemporal continuity of the image, effectively suppressing image jumps and structural breaks.

[0035] Finally, a hierarchical rate-distortion optimization strategy is adopted to quantize and entropy encode the encoded data of each layer, dynamically adjust the bitrate allocation weight of each enhancement layer according to the bandwidth constraint, and output the compressed bitstream.

[0036] Understandably, the layered distortion optimization strategy uses the Lagrange multiplier method to set differentiated distortion weight coefficients for different layers. The base layer is set to 1.0, the first enhancement layer to 0.6, and the second enhancement layer to 0.3. This weight allocation reflects the contribution of each layer to the final reconstruction quality. As the cornerstone of reconstruction, the base layer has the highest distortion penalty.

[0037] Based on real-time network bandwidth monitoring, the quantization parameters of each layer are dynamically adjusted, with a quantization step size ranging from 2 to 32 and a step size adjustment granularity of 1. When bandwidth is sufficient, the quantization step size is reduced to retain more high-frequency details; when bandwidth is limited, priority is given to ensuring the quality of the base layer, and high-enhancement layer data is gradually discarded. The quantized coefficients are compressed by a context-adaptive arithmetic entropy encoder, and finally a structured compressed bitstream is output. This bitstream supports scalability, and the decoder can selectively decode different layers according to the available bandwidth to achieve dynamic bitrate switching.

[0038] On the decoding side, multi-layer residual reconstruction and viewpoint synthesis are performed, specifically as follows: First, decode the base layer data to generate an initial six-DOF video, which contains complete low-frequency geometry and texture information and can be rendered from any viewpoint. Next, the residual signals of the first enhancement layer and the second enhancement layer are decoded and superimposed sequentially to enhance the details of the initial video; Finally, a depth-map-guided viewpoint rendering algorithm generates the corresponding output image based on the user's real-time requested six-DOF pose. This rendering algorithm employs a voxel traversal strategy based on ray casting, with a traversal step size of 0.01 meters to ensure rendering accuracy. For regions with discontinuous depth, the rendering algorithm implements edge-preserving smoothing processing with a smoothing window size of 5 x 5 pixels and an edge detection threshold set to a depth gradient greater than 0.1 meters per pixel, thereby eliminating rendering noise while preserving geometric sharpness. The latency of the entire rendering process is controlled within 50 milliseconds, and the viewpoint switching response time is less than 100 milliseconds. With bandwidth fluctuations ranging from 5 megabits per second to 50 megabits per second, the structural similarity index of the reconstructed video can be maintained at greater than 0.92, meeting the requirements for a high-fidelity immersive experience.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0040] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-layer residual compression method for six-degree-of-freedom video, characterized in that, include: Spatial and viewpoint joint sampling is performed on the original six-degree-of-freedom video data to construct a multi-viewpoint image set containing texture maps and depth maps. Based on the user interaction trajectory, viewpoint distribution hotspot areas are predicted, and spatial coding units are dynamically divided. An adaptive multi-layer residual coding structure is constructed, which decomposes the multi-view image set into a base layer and at least two enhancement layers. The base layer covers the low-frequency geometric and texture information of all view regions, and the enhancement layers encode high-frequency residual information hierarchically according to view importance and geometric complexity. Within each layer, cross-modal joint residual modeling is performed, spatial gradient features of texture maps and depth maps are processed simultaneously, cross-modal correlations are extracted through shared convolutional kernels, and joint residual signals are generated. The joint residual signal is subjected to temporal prediction with motion and viewpoint consistency constraints. A viewpoint continuity model is constructed using the six-degree-of-freedom motion parameters between adjacent frames. Motion compensation and viewpoint interpolation optimization are then performed on the residual signal. A hierarchical rate-distortion optimization strategy is adopted to quantize and entropy encode the encoded data of each layer. The bit rate allocation weight of each enhancement layer is dynamically adjusted according to the bandwidth constraint, and a compressed bit stream is output.

2. The multilayer residual compression method according to claim 1, characterized in that, The joint spatial and viewpoint sampling adopts an octree spatial partitioning strategy, dividing the 3D scene into cubic units with side lengths of 0.5 meters to 2 meters, and constructing a viewpoint probability heatmap by combining user historical interaction data, and increasing the sampling density to twice the original density for areas with heat values ​​greater than 0.

7.

3. The multilayer residual compression method according to claim 1, characterized in that, The base layer uses uniform sampling across the entire field of view, with a sampling interval of one field of view every 15 degrees; the first enhancement layer covers areas with a thermal value greater than 0.5, with the field of view interval reduced to one field of view every 5 degrees; the second enhancement layer focuses on highly interactive areas with a thermal value greater than 0.8, with the field of view interval reduced to one field of view every 2 degrees, and introduces a local geometric detail enhancement module.

4. The multilayer residual compression method according to claim 1, characterized in that, The cross-modal joint residual modeling adopts a dual-branch convolutional neural network structure. The texture branch and the depth branch share the first 3 layers of convolutional kernels. The kernel size is 3 by 3 and the number of channels is 64. Subsequent branches independently process their respective modal features and fuse the cross-modal residual responses through a channel attention mechanism to generate a joint residual map.

5. The multilayer residual compression method according to claim 1, characterized in that, The temporal prediction of the motion and viewpoint consistency constraint is based on a six-degree-of-freedom motion parameter matrix, which contains three translation components and three rotation components. The motion trajectory of the intermediate viewpoint is generated between adjacent keyframes using a Lie group interpolation algorithm, and block matching compensation based on the motion vector field is performed on the residual signal. The compensation block size is 16 by 16 pixels.

6. The multilayer residual compression method according to claim 1, characterized in that, The layered rate-distortion optimization strategy adopts the Lagrange multiplier method, setting the distortion weight coefficient to 1.0 for the base layer, 0.6 for the first enhancement layer, and 0.3 for the second enhancement layer. The quantization parameters of each layer are dynamically adjusted according to the real-time bandwidth feedback, with a quantization step size ranging from 2 to 32 and a step size adjustment granularity of 1.

7. The multilayer residual compression method according to claim 1, characterized in that, Multi-layer residual reconstruction and viewpoint synthesis are performed at the decoding end, specifically as follows: Decoding the base layer generates the initial six-DOF video; Then, the residual signals of the enhancement layers are superimposed sequentially for detail enhancement; The depth-map-guided perspective rendering algorithm generates output images from any user-requested perspective with a rendering latency of less than 50 milliseconds.

8. The multilayer residual compression method according to claim 7, characterized in that, The depth map-guided view rendering algorithm adopts a voxel traversal strategy based on ray projection, with a traversal step size of 0.01 meters. It performs edge-preserving smoothing processing on depth discontinuous regions, with a smoothing window size of 5 x 5 pixels and an edge detection threshold of depth gradient greater than 0.1 meters per pixel.

9. The multilayer residual compression method according to claim 1, characterized in that, The original six-degree-of-freedom video data comes from a multi-camera array acquisition system, which contains no fewer than 32 synchronized cameras with a baseline distance of 0.3 to 1.5 meters, an acquisition frame rate of 30 frames per second, a depth map accuracy of ±2 millimeters, and a texture map resolution of 4096 x 2048 pixels.

Images