A three-dimensional video coding compression method

Abstract

The present invention belongs to the field of volumetric video coding and compression, and specifically relates to a three-dimensional volumetric video coding and compression method, comprising: constructing a voxel space based on a sparse point cloud of a three-dimensional volumetric video and initializing information of each anchor point therein; determining the three-dimensional position of each neural Gaussian at the anchor point based on the anchor point information; obtaining static attribute information of each neural Gaussian at the anchor point using a multi-level projection (MLP) of each static attribute based on the anchor point information and the current view to be rendered; obtaining dynamic attribute information of each neural Gaussian at the anchor point using a multi-level projection (MLP) of each dynamic attribute based on the anchor point information, the current view to be rendered, and the frame to be rendered; obtaining neural Gaussian points in a deformable space using a temporal projection model based on the dynamic attribute information of the neural Gaussian points in a canonical space, thereby obtaining the view to be rendered and the frame to be rendered; optimizing the parameters of all MLPs and the anchor point information of all anchor points, and ultimately using all the anchor point information and all the MLPs as the coding and compression result. The present invention can significantly improve the compression rate.

Description

Technical Field

[0001] The present invention belongs to the field of volumetric video coding and compression, and more particularly, relates to a three-dimensional volumetric video coding and compression method. Background Art

[0002] In recent years, with the growing demand for digital media vision, the presentation format of multimedia information has undergone tremendous changes: from images to videos, from standard-definition video to high-definition video, and from single two-dimensional (2D) video to three-dimensional (3D) video with freely switchable viewing angles. The amount of multimedia data accompanying this has increased exponentially, creating an urgent need for a video coding technology that can efficiently compress high-resolution 3D video.

[0003] Among the many 3D video encoding methods, direct encoding strategies, such as simulcast 3D video encoding, are based on traditional 2D video coding strategies. This method compresses each video channel independently. For example, in unbalanced stereo video encoding, different quantizers are used for each viewpoint, resulting in different decoded quality and potentially saving some bitrate. However, this encoding method fails to exploit the correlation between the left and right viewpoints and removes redundant information between them, significantly limiting the compression ratio.

[0004] Improving the coding efficiency of 3D volumetric video requires considering the temporal and inter-view correlations in 3D video. This is where 3D video coding based on motion estimation and disparity estimation comes in. Over a decade ago, the MPEG-2 Multiview Profile proposed leveraging this feature to improve 3D video coding efficiency by combining the cross-correlation between left and right views with the spatiotemporal correlation within the same viewpoint. Relatedly, methods based on disparity compensation have been proposed to eliminate view correlation. Transform-domain disparity compensation and disparity estimation methods, along with their associated multi-view video coding schemes, have also been proposed. Current multi-view video coding standards are based on motion estimation and disparity estimation. To maximize compression, encoding employs a view-temporal pyramid prediction structure based on hierarchical B frames. This structure is adopted by the official MVC test model, JMVM. However, the compression ratios achieved by this approach still fall short of current compression requirements. Summary of the Invention

[0005] In view of the defects and improvement needs of the existing technology, the present invention provides a 3D video coding compression method, which aims to improve the 3D video compression rate to reduce the bit rate required for video transmission.

[0006] To achieve the above object, according to one aspect of the present invention, a 3D volumetric video coding and compression method is provided, comprising:

[0007] Based on the sparse point cloud obtained from the multi-view 3D volumetric video to be compressed, a voxel space is constructed and the information of each anchor point therein is initialized, including static attribute information, dynamic attribute information and static features, dynamic features and offset scaling information for decoding the static attribute information, dynamic attribute information and 3D position of the corresponding neural Gaussian in the preset canonical space;

[0008] The offset scaling information of each anchor point is used to determine the three-dimensional position of each neural Gaussian at the anchor point; the static features of each anchor point and the current viewing angle to be rendered are input into the fully connected neural network model MLP corresponding to each static attribute to obtain the static attribute information of each neural Gaussian at the anchor point; the dynamic features of each anchor point, the current viewing angle to be rendered, and the time of the frame to be rendered are input into the MLP corresponding to each dynamic attribute to obtain the dynamic attribute information of each neural Gaussian at the anchor point; the three-dimensional position of each neural Gaussian and the attribute information constitute a neural Gaussian point in the canonical space;

[0009] Based on each static attribute information to be projected and its corresponding dynamic attribute information of each neural Gaussian point in the canonical space, a temporal projection model of the static attribute to be projected is used to project the static attribute information to the deformation space corresponding to the frame to be rendered, thereby obtaining a neural Gaussian point in the deformation space; the static attribute to be projected is selected from one or more of static attributes and three-dimensional positions;

[0010] Based on all neural Gaussian points in the deformation space, the image of the perspective to be rendered and the frame to be rendered are obtained; by comparing the image with its real image, the parameters of all MLPs and the anchor point information of all anchor points are optimized, and the above operation is repeated until the termination condition is reached. The final anchor point information of all anchor points and all MLPs are used as the encoding compression results to complete the three-dimensional video encoding compression.

[0011] Furthermore, the static attributes include opacity, color, rotation information and size.

[0012] Furthermore, the static attributes to be projected include three-dimensional position, opacity, and rotation information, and the dynamic attributes include a position change coefficient, an opacity change coefficient, and a rotation information change coefficient.

[0013] Furthermore, the time projection model of the opacity is expressed as:

[0014] σ i (t)=σ i (t0)+tanh(s i )*(t-t0)

[0015] Where, σ i (t) represents the opacity of the i-th neural Gaussian at time t, σ i (t0) represents the opacity of the i-th neural Gaussian in the canonical space, si represents the opacity variation coefficient, and t0 represents the video frame time corresponding to the standard space.

[0016] Furthermore, the time projection model of the three-dimensional position is expressed as:

[0017]

[0018] Where μ i (t) represents the three-dimensional position of the i-th neural Gaussian at time t, μ i (t0) represents the three-dimensional position of the i-th neural Gaussian in the canonical space, b i,k represents the polynomial coefficient, k represents the polynomial order, n p represents the upper limit of k, and t0 represents the video frame time corresponding to the standard space.

[0019] Furthermore, the time projection model of the rotation information is expressed as:

[0020]

[0021] Where q i (t) represents the rotation information of the i-th neural Gaussian at time t, q i (t0) represents the rotation information of the i-th neural Gaussian in the canonical space, k represents the polynomial order, c i,k Represents the rotation information change coefficient, n p represents the upper limit of k, and t0 represents the video frame time corresponding to the standard space.

[0022] The present invention also provides a three-dimensional video coding and compression device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the three-dimensional video coding and compression method described above when executing the computer program.

[0023] The present invention also provides a 3D video decoding method, comprising:

[0024] receiving a coding compression result sent by the coding end, including anchor point information of all anchor points and all MLPs, wherein the coding compression result is obtained by the above-mentioned 3D volumetric video coding compression method;

[0025] The offset scaling information of each anchor point is used to determine the three-dimensional position of each neural Gaussian at the anchor point; the static features of each anchor point and the current viewing angle to be rendered are input into the fully connected neural network model MLP corresponding to each static attribute to obtain the static attribute information of each neural Gaussian at the anchor point; the dynamic features of each anchor point, the current viewing angle to be rendered, and the time of the frame to be rendered are input into the MLP corresponding to each dynamic attribute to obtain the dynamic attribute information of each neural Gaussian at the anchor point; the three-dimensional position of each neural Gaussian and the attribute information constitute a neural Gaussian in the canonical space;

[0026] Based on each static attribute information to be projected and its corresponding dynamic attribute information of each neural Gaussian point in the canonical space, a temporal projection model of the static attribute to be projected is used to project the static attribute information to the deformation space corresponding to the frame to be rendered, thereby obtaining a neural Gaussian point in the deformation space; the static attribute to be projected is selected from one or more of static attributes and three-dimensional positions;

[0027] Based on all neural Gaussian points in the deformation space, the image of the perspective to be rendered and the frame to be rendered are obtained.

[0028] The present invention also provides a 3D video decoding device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned 3D video decoding method when executing the computer program.

[0029] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned three-dimensional video encoding and compression method or the above-mentioned three-dimensional video decoding method.

[0030] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0031] (1) The present invention proposes a three-dimensional volumetric video coding compression method. First, with the help of anchor points in voxel space and a fully connected neural network model, a neural Gaussian point cloud in a preset standard space is constructed, wherein each neural Gaussian (point) in the standard space is determined by the three-dimensional position and attribute information of the neural Gaussian point, the three-dimensional position of each neural Gaussian is determined by the offset scaling information of the anchor point corresponding to the neural Gaussian, and the static attribute information and dynamic attribute information of each neural Gaussian are respectively decoded and determined by the corresponding attribute fully connected neural network model under the input of the static features and dynamic features of the anchor point corresponding to the neural Gaussian; secondly, the images corresponding to other video frames are obtained by temporally projecting and rendering the neural Gaussian point cloud in the preset standard space, and by comparing with the real image, the anchor point information of each anchor point and the parameters of each neural network model are back-transferred and optimized, and the final compression result contains the trained anchor point information and each neural network model. Since the compression result of the present invention only contains the trained anchor point information and each neural network model, and the same neural Gaussian attribute of different anchor points shares the same neural network model, the final compression result occupies less memory, ensuring a higher compression rate, reducing the bandwidth requirement of the network to transmit a large amount of bulk video data, and the bit rate required for video transmission is greatly reduced; in addition, since the input of the neural network model corresponding to each static attribute includes the time and perspective of the video frame, the compression result obtained by the method of the present invention can be decoded into an image of any frame and any perspective according to actual needs during subsequent decoding, that is, video decoding of any perspective and any frame can be realized on the terminal, thereby improving the user's viewing experience and not being restricted by the initial perspective and video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a three-dimensional video coding and compression method provided by an embodiment of the present invention;

[0033] Figure 2 A schematic flow chart of a 3D volumetric video encoding and compression method according to an embodiment of the present invention;

[0034] Figure 3 Schematic diagram of decoding anchor point features and time projection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0036] Example 1

[0037] A three-dimensional video coding compression method, such as Figure 1 As shown, including:

[0038] Based on the sparse point cloud obtained from the multi-view 3D volumetric video to be compressed, a voxel space is constructed and the information of each anchor point therein is initialized, including static attribute information, dynamic attribute information and static features, dynamic features and offset scaling information for decoding the static attribute information, dynamic attribute information and 3D position of the corresponding neural Gaussian in the preset canonical space;

[0039] The offset scaling information of each anchor point is used to determine the three-dimensional position of each neural Gaussian at the anchor point; the static features of each anchor point and the current viewing angle to be rendered are input into the fully connected neural network model MLP corresponding to each static attribute to obtain the static attribute information of each neural Gaussian at the anchor point; the dynamic features of each anchor point, the current viewing angle to be rendered, and the time of the frame to be rendered are input into the MLP corresponding to each dynamic attribute to obtain the dynamic attribute information of each neural Gaussian at the anchor point; the three-dimensional position of each neural Gaussian and the attribute information constitute a neural Gaussian point in the canonical space;

[0040] Based on each static attribute information to be projected and its corresponding dynamic attribute information of each neural Gaussian point in the canonical space, a temporal projection model of the static attribute to be projected is used to project the static attribute information to the deformation space corresponding to the frame to be rendered, thereby obtaining a neural Gaussian point in the deformation space; the static attribute to be projected is selected from one or more of static attributes and three-dimensional positions;

[0041] Based on all neural Gaussian points in the deformation space, the image of the perspective to be rendered and the frame to be rendered are obtained; by comparing the image with its real image, the parameters of all MLPs and the anchor point information of all anchor points are optimized, and the above operation is repeated until the termination condition is reached. The final anchor point information of all anchor points and all MLPs are used as the encoding compression results to complete the three-dimensional video encoding compression.

[0042] The above-mentioned method constructs a voxel space and initializes the anchor point information therein based on the sparse point cloud of the multi-view 3D volumetric video to be compressed. For example, the method specifically includes: using motion recovery structure (SfM) to process the 3D volumetric video to be compressed to obtain a sparse point cloud; constructing a voxel space based on the sparse point cloud of the multi-view 3D volumetric video to be compressed and initializing the anchor point information of each anchor point in the voxel space to activate the anchor point.

[0043] The three-dimensional position of each neural Gaussian and its attribute information constitute a neural Gaussian point in the canonical space, and all neural Gaussian points corresponding to all anchor points constitute a neural Gaussian point cloud in the canonical space.

[0044] The above-mentioned frame to be rendered and the image of the perspective to be rendered are obtained based on all the neural Gaussian points in the deformation space. For example, specifically: the neural Gaussian point cloud in the deformation space is sequentially projected into two-dimensional space and rapidly rasterized to obtain the rendered image of the frame to be rendered.

[0045] In the method of this embodiment, the same neural Gaussian attributes of different anchor points share the same neural network model, so that the final compression result occupies less memory and ensures a higher compression rate.

[0046] The compression result obtained by the method of this embodiment can be decoded into images of any frame and any perspective according to actual needs during subsequent decoding, that is, video decoding of any perspective and any frame can be realized on the terminal, thereby improving the user's viewing experience without being restricted by the initial perspective and video frame.

[0047] This method is a volumetric video coding and compression method, belonging to the field of coding and compression. This method compresses the 3D volumetric video input from multiple perspectives, expresses it using a neural network model, and can adapt quantization and entropy coding to achieve further compression. The overall process of the present invention is as follows: Figure 2 and Figure 3 This differs from the current 4DGS technology, which aims to achieve high-quality 3D reconstruction. This method, however, is dedicated to obtaining a more compact volumetric video representation in the time domain. While maintaining a low bit rate, it can still achieve high-quality 3D volumetric video decoding results from any perspective.

[0048] It should be noted that, since a 3D volumetric video typically has a large number of frames, to ensure encoding quality, the method of this embodiment requires dividing the 3D volumetric video into multiple segments, each with a preset number of frames, for example, 15 frames. Each segment is considered a Group of Pictures (GOP), and the volumetric video is compressed in units of 15 frames during each encoding operation. Furthermore, when compressing a 3D volumetric video, the number of frames per GOP can be adjusted based on actual conditions; different GOPs can be the same or different, and there is no limit on the GOP length.

[0049] As a preferred embodiment, the static attributes include opacity, color, rotation information and size.

[0050] As a preferred implementation, the attributes to be projected include three-dimensional position, opacity, and rotation, and the dynamic attributes include a position change coefficient, an opacity change coefficient, and a rotation information change coefficient.

[0051] The mathematical form of each time projection model may preferably be a polynomial form.

[0052] Specifically, as a preferred embodiment, the time projection model of the three-dimensional position is expressed as:

[0053]

[0054] Where μ i (t) represents the three-dimensional position of the i-th neural Gaussian at time t, μ i (t0) represents the three-dimensional position of the i-th neural Gaussian in the canonical space, b i,k represents the polynomial coefficient, k represents the polynomial order, n p represents the upper limit of k, and t0 represents the video frame time corresponding to the standard space.

[0055] Specifically, as a preferred embodiment, the time projection model of opacity is expressed as:

[0056] σ i (t)=σ i (t0)+tanh(s i )*(t-t0)

[0057] Where, σ i (t) represents the opacity of the i-th neural Gaussian at time t, σ i (t0) represents the opacity of the i-th neural Gaussian in the canonical space, s i represents the opacity variation coefficient, and t0 represents the video frame time corresponding to the standard space.

[0058] Specifically, as a preferred embodiment, the time projection model of the rotation information is expressed as:

[0059]

[0060] Where q i (t) represents the rotation information of the i-th neural Gaussian at time t, q i (t0) represents the rotation information of the i-th neural Gaussian in the canonical space, k represents the polynomial order, n p represents the upper limit of k, c i,k represents the rotation information change coefficient, and t0 represents the video frame time corresponding to the standard space.

[0061] It should be noted that the method in this embodiment achieves compression by encoding 3D volumetric video into a compact 3D point cloud. Due to the performance limitations of dynamic Gaussian 3D modeling, the input 3D video must have a sufficient number of viewing angles and a sufficiently dense viewing angle. Otherwise, the point cloud reconstruction quality will be compromised, resulting in poor rendered video quality. In other words, a greater number of initial viewing angles results in better video quality and compression performance.

[0062] Example 2

[0063] A 3D volumetric video coding and compression device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements a 3D volumetric video coding and compression method as described in the above embodiment.

[0064] The relevant technical solutions are the same as those in Example 1 and will not be described again here.

[0065] Example 3

[0066] A 3D video decoding method, comprising:

[0067] receiving a coding compression result sent by the coding end, including anchor point information of all anchor points and all MLPs, wherein the coding compression result is obtained by a 3D volumetric video coding compression method as described in the above embodiment;

[0068] The offset scaling information of each anchor point is used to determine the three-dimensional position of each neural Gaussian at the anchor point; the static features of each anchor point and the current viewing angle to be rendered are input into the fully connected neural network model MLP corresponding to each static attribute to obtain the static attribute information of each neural Gaussian at the anchor point; the dynamic features of each anchor point, the current viewing angle to be rendered, and the time of the frame to be rendered are input into the MLP corresponding to each dynamic attribute to obtain the dynamic attribute information of each neural Gaussian at the anchor point; the three-dimensional position of each neural Gaussian and the attribute information constitute a neural Gaussian in the canonical space;

[0069] Based on each static attribute information to be projected and its corresponding dynamic attribute information of each neural Gaussian point in the canonical space, a temporal projection model of the static attribute to be projected is used to project the static attribute information to the deformation space corresponding to the frame to be rendered, thereby obtaining a neural Gaussian point in the deformation space; the static attribute to be projected is selected from one or more of static attributes and three-dimensional positions;

[0070] Based on all neural Gaussian points in the deformation space, the images of the perspective to be rendered and the frames to be rendered are obtained.

[0071] The relevant technical solutions are the same as those in the previous embodiment and will not be described again here.

[0072] Example 4

[0073] A 3D video decoding device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the 3D video decoding method described in the previous embodiment is implemented.

[0074] The relevant technical solutions are the same as those in the previous embodiment and will not be described again here.

[0075] Example 5

[0076] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the computer program implements the three-dimensional video encoding and compression method described in the first embodiment or the three-dimensional video decoding method described in the third embodiment.

[0077] The relevant technical solutions are the same as those in the previous embodiment and will not be described again here.

[0078] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional video coding and compression method, characterized in that: include: Based on the sparse point cloud obtained from the multi-view 3D volumetric video to be compressed, a voxel space is constructed and the information of each anchor point therein is initialized, including static attribute information, dynamic attribute information and static features, dynamic features and offset scaling information for decoding the static attribute information, dynamic attribute information and 3D position of the corresponding neural Gaussian in the preset canonical space; The offset scaling information of each anchor point is used to determine the three-dimensional position of each neural Gaussian at the anchor point; the static features of each anchor point and the current viewing angle to be rendered are input into the fully connected neural network model MLP corresponding to each static attribute to obtain the static attribute information of each neural Gaussian at the anchor point; the dynamic features of each anchor point, the current viewing angle to be rendered, and the time of the frame to be rendered are input into the MLP corresponding to each dynamic attribute to obtain the dynamic attribute information of each neural Gaussian at the anchor point; the three-dimensional position of each neural Gaussian and the attribute information constitute a neural Gaussian point in the canonical space; Based on each static attribute information to be projected and its corresponding dynamic attribute information of each neural Gaussian point in the canonical space, a temporal projection model of the static attribute to be projected is used to project the static attribute information to the deformation space corresponding to the frame to be rendered, thereby obtaining a neural Gaussian point in the deformation space; the static attribute to be projected is selected from one or more of static attributes and three-dimensional positions; Based on all neural Gaussian points in the deformation space, the image of the perspective to be rendered and the frame to be rendered are obtained; by comparing the image with its real image, the parameters of all MLPs and the anchor point information of all anchor points are optimized, and the above operation is repeated until the termination condition is reached. The final anchor point information of all anchor points and all MLPs are used as the encoding compression results to complete the three-dimensional video encoding compression.

2. The 3D volumetric video coding and compression method according to claim 1, wherein: The static properties include opacity, color, rotation information, and size.

3. The 3D volumetric video coding and compression method according to claim 1, wherein: The static attributes to be projected include three-dimensional position, opacity and rotation information, and the dynamic attributes include a position change coefficient, an opacity change coefficient and a rotation information change coefficient.

4. The 3D volumetric video coding and compression method according to claim 3, wherein: The temporal projection model of opacity is expressed as: σ i (t)=σ i (t0)+tanh(s i )*(t-t0) Where, σ i (t) represents the opacity of the i-th neural Gaussian at time t, σ i (t0) represents the opacity of the i-th neural Gaussian in the canonical space, s i represents the opacity variation coefficient, and t0 represents the video frame time corresponding to the standard space.

5. The 3D volumetric video coding and compression method according to claim 3, wherein: The time projection model of the three-dimensional position is expressed as: Where μ i (t) represents the three-dimensional position of the i-th neural Gaussian at time t, μ i (t0) represents the three-dimensional position of the i-th neural Gaussian in the canonical space, b i,k represents the polynomial coefficient, k represents the polynomial order, n p represents the upper limit of k, and t0 represents the video frame time corresponding to the standard space.

6. The 3D volumetric video coding and compression method according to claim 3, wherein: The time projection model of the rotation information is expressed as: Where q i (t) represents the rotation information of the i-th neural Gaussian at time t, q i (t0) represents the rotation information of the i-th neural Gaussian in the canonical space, k represents the polynomial order, c i,k Represents the rotation information change coefficient, n p represents the upper limit of k, and t0 represents the video frame time corresponding to the standard space.

7. A 3D video coding and compression device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the three-dimensional video coding compression method according to any one of claims 1 to 6 is implemented.

8. A 3D video decoding method, characterized in that: include: receiving a coding compression result sent by an encoding end, including anchor point information of all anchor points and all MLPs, wherein the coding compression result is obtained by the 3D volumetric video coding compression method according to any one of claims 1 to 6; The offset scaling information of each anchor point is used to determine the three-dimensional position of each neural Gaussian at the anchor point; the static features of each anchor point and the current viewing angle to be rendered are input into the fully connected neural network model MLP corresponding to each static attribute to obtain the static attribute information of each neural Gaussian at the anchor point; the dynamic features of each anchor point, the current viewing angle to be rendered, and the time of the frame to be rendered are input into the MLP corresponding to each dynamic attribute to obtain the dynamic attribute information of each neural Gaussian at the anchor point; the three-dimensional position of each neural Gaussian and the attribute information constitute a neural Gaussian in the canonical space; Based on each static attribute information to be projected and its corresponding dynamic attribute information of each neural Gaussian point in the canonical space, a temporal projection model of the static attribute to be projected is used to project the static attribute information to the deformation space corresponding to the frame to be rendered, thereby obtaining a neural Gaussian point in the deformation space; the static attribute to be projected is selected from one or more of static attributes and three-dimensional positions; The image of the perspective to be rendered and the frame to be rendered are obtained based on all neural Gaussian points in the deformation space.

9. A 3D video decoding device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the three-dimensional video decoding method according to claim 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements a 3D video encoding and compression method according to any one of claims 1 to 6 or a 3D video decoding method according to claim 8.

Images