A volumetric video coding method based on inter-view multiplexing

Abstract

The application discloses a volumetric video coding method based on inter-frame multiplexing, and relates to the field of volumetric video coding.The method mainly comprises the following steps: picture similarity detection, point cloud block similarity detection, inter-frame multiplexing, octree optimization coding algorithm, inverse Morton order rearrangement, 2D video coding algorithm, inter-frame decoding, octree optimization decoding algorithm and 2D video decoding algorithm.The application solves the problem of insufficient decoding frame rate of the decoding algorithm based on inter-frame multiplexing in a mobile terminal, realizes the target of greatly reducing the required bandwidth of transmission by adopting the inter-frame multiplexing technology in the field of volumetric video streaming media system, uses the deep learning point cloud registration model based on geometric information and color information to perform motion estimation between adjacent frames of the volumetric video for the first time, and realizes fast and accurate elimination of redundant information, and uses the 2D video coding algorithm to compress the color information of the volumetric video for the first time, thereby realizing inter-frame coding of the color information of the volumetric video.

Description

Technical Field

[0001] This invention relates to the field of volumetric video encoding and decoding technology, and specifically to a volumetric video encoding and decoding method based on inter-frame multiplexing. Background Technology

[0002] Volumetric video is hailed as a next-generation media technology, a novel media technology capable of providing a highly immersive and interactive user experience. Unlike 2D and 360-degree videos, volumetric video, composed of 3D data, allows users to watch with six degrees of freedom, providing a more immersive experience. However, despite its enormous potential, volumetric video streaming systems face two key technical challenges: (1) The sheer size of volumetric video data. The most common format for volumetric video is point cloud, with single-frame sizes typically ranging from 4MB to 15MB. The uncompressed transmission of the raw video stream requires a transmission rate of 1Gbps to 3.6Gbps, far exceeding the capabilities of current WiFi or 5G networks. (2) Decoding speed cannot meet 30fps. For users to watch six-degree-of-freedom volumetric video, content decoding and updates require 30fps. Existing commonly used volumetric video encoding and decoding algorithms still cannot meet this requirement.

[0003] Currently, some methods have reduced the size of video streaming media using viewport adaptation and 3D super-resolution, while optimizing decoding speed through multi-threading and improved decoding algorithms. Viewport adaptation reduces bandwidth requirements by optimizing data according to the user's viewport, maintaining high definition for content within the viewport and processing lower definition for content outside the viewport. While this method reduces bandwidth needs, it requires real-time prediction of the user's viewing angle on the mobile device and real-time optimization of the point cloud streaming media on the server to meet motion perception latency. This optimization method relies heavily on prediction accuracy, and prediction errors can lead to a poor viewing experience. 3D super-resolution requires deploying a super-resolution model on the mobile device, placing high demands on its performance. Multi-threading to improve mobile decoding speed typically requires high-performance mobile devices, and this method consumes a lot of power when deployed on mobile devices. Improving decoding algorithms to increase mobile decoding speed has achieved good results, but the bandwidth required for the optimized video streaming media system is still relatively high. Overall, the above optimization methods still require bandwidth of over 100 Mbps for compressed video streaming media to ensure transmission, indicating significant room for improvement in video streaming media systems. After investigation, it was found that the volumetric video encoding and decoding algorithms used in volumetric video streaming systems still focus on single-frame processing and ignore the inter-frame content redundancy of volumetric videos. The volumetric video encoding and decoding algorithms can only provide a compression ratio of 4 to 8, which is still far behind the compression ratio of hundreds of times of 2D video encoding algorithms H.264 and H.265 that take into account inter-frame coding. Summary of the Invention

[0004] The purpose of this invention is to provide a volumetric video encoding and decoding method based on inter-frame multiplexing to solve the problem of excessively high bandwidth requirements for current volumetric video transmission. This invention reduces the bandwidth required for volumetric video transmission by eliminating redundant information during transmission, and simultaneously optimizes the decoding algorithm based on inter-frame multiplexing to meet the decoding frame rate.

[0005] The technical solution adopted by this invention to solve the technical problem is as follows:

[0006] The present invention provides a volumetric video encoding and decoding method based on inter-frame multiplexing, comprising the following steps:

[0007] Step S1: Inter-frame multiplexing coding;

[0008] Step S1.1: Image similarity detection;

[0009] The volumetric video of different frames is projected into a two-dimensional image from different angles, and then a structural similarity algorithm is used to quickly detect the image similarity between adjacent frames.

[0010] Step S1.2: Point cloud block similarity detection;

[0011] Adjacent frames that are similar after image similarity detection are divided into blocks, and a heuristic algorithm is used to determine the similarity between two point cloud blocks;

[0012] Step S1.3: Inter-frame multiplexing;

[0013] After downsampling two point cloud blocks that are similar after point cloud block similarity detection, they are used as input to a deep learning point cloud registration model based on sparse convolution for registration. The transformation matrix between the two point cloud blocks is calculated, and this transformation matrix is ​​used as an intermediate file for inter-frame multiplexing.

[0014] Step S1.4: Octree optimized coding algorithm;

[0015] An octree optimized coding algorithm is used to intra-frame encode the geometric information of the volumetric video, and an intermediate file of the geometric information is output.

[0016] Step S1.5: Reverse Morton order rearrangement;

[0017] The color information of each frame of the volumetric video is rearranged in reverse Morton order and projected onto a two-dimensional plane to form an image. Multiple frames of images are then stitched together to form a video stream.

[0018] Step S1.6: 2D video coding algorithm;

[0019] The color information compression algorithm based on 2D video compression technology is used to stitch the images of different frames of a large video into a video stream in their original order, and the H.264 algorithm is used to perform intra-frame and inter-frame compression on the video stream to output an intermediate file of color information.

[0020] Step S2: Inter-frame multiplexing decoding;

[0021] Step S2.1: Inter-frame decoding;

[0022] First, determine whether the intermediate file is the intermediate file for inter-frame multiplexing in step S1.3. If so, reconstruct the point cloud block using the transformation matrix of the point cloud block in the previous frame. If not, execute steps S2.2 and S2.3 respectively.

[0023] Step S2.2: Octree-optimized decoding algorithm;

[0024] For intermediate files that are not used in the inter-frame multiplexing process in step S1.3, the volumetric video geometric information is decoded using an octree-optimized decoding algorithm, and the decoded volumetric video geometric information is output. The decoding frame rate is guaranteed by improving the octree structure and reasonably allocating the GPU and CPU resources used during mobile decoding.

[0025] Step S2.3: 2D video decoding algorithm;

[0026] For intermediate files that are not used in the inter-frame multiplexing process in step S1.3, the volumetric video color information is decoded using a 2D video decoding algorithm, and the decoded volumetric video color information is output. The Morton order is used to reorder the data, and the decoded volumetric video geometric information and volumetric video color information are spliced ​​together to obtain single-frame volumetric video data.

[0027] Furthermore, the specific steps of step S1.2 are as follows:

[0028] By traversing each point in the source point cloud block, the point closest to the target point cloud block is found, thus obtaining the matching relationship between the points in the two point cloud blocks. Two point cloud blocks are considered similar when the following three rules are met simultaneously:

[0029] (1) The error between N(i,j) and N(i+1,j) does not exceed 10%;

[0030] (2) The average value is less than or equal to 0.01m;

[0031] (3) 80% Less than 0.01m.

[0032] Where N(i,j) represents the number of points in the j-th point cloud block of the i-th frame, N(i+1,j) represents the number of points in the j-th point cloud block of the (i+1)-th frame, and P(i,j,u) represents point u in the j-th point cloud block of the i-th frame. This represents the point that is closest to the j-th point cloud block in the (i+1)-th frame. Representing point u and point The Euclidean distance between them; similar point cloud blocks are selected by filtering out the above three rules.

[0033] Furthermore, the specific steps of step S1.3 are as follows:

[0034] The point cloud geometry information in the point cloud block is projected into the mesh; for each mesh cell, the center point inside is calculated; for each center point, the relative position of its surrounding points is calculated and stored in a sparse tensor. The point cloud geometry information is converted into a sparse tensor and used as input to a deep learning point cloud registration model.

[0035] Further, in step S1.3, the deep learning point cloud registration model based on sparse convolution includes: a geometric information encoding module, a color information encoding module, an attention fusion module, and a decoder module; the geometric information encoding module uses four sparse tensor convolutional layers to extract local and global features of the point cloud geometric information; the color information encoding module uses a pre-trained ResNet34 to extract color information features; the attention fusion module fuses the outputs of the geometric information encoding module and the color information encoding module based on an attention mechanism to generate a unique point cloud description vector for each point cloud block; the decoder module takes as input the point cloud description vectors encoded by the two point cloud blocks respectively; the decoder module decodes each point cloud description vector using four sparse tensor convolutional layers and simultaneously inputs the decoded information of the two point cloud description vectors into a fully connected layer, which outputs the transformation matrix between the two point cloud blocks as an intermediate file for inter-frame multiplexing.

[0036] Furthermore, in step S1.3, the intermediate file for inter-frame multiplexing is represented as follows:

[0037]

[0038] Where H(i,j) represents the intermediate file when the j-th point cloud block in the i-th frame uses inter-frame multiplexing, which is the transformation matrix relative to the point cloud block in the previous frame. A 3X3 Let T represent the rotation matrix. 3X1 O represents the translation vector. 1X3 Let S represent the zero vector, and let a represent the overall scaling factor. 11 -a 13These represent the scaling factors on the x-axis, y-axis, and z-axis, respectively. 21 -a 23 These represent the shearing factors on the x-axis, y-axis, and z-axis, respectively. 31 -a 33 t represents the rotation factors on the x-axis, y-axis, and z-axis, respectively. x t represents the translation along the x-axis. y t represents the translation along the y-axis. z This represents the translation along the z-axis.

[0039] Furthermore, the specific steps of step S1.4 are as follows:

[0040] During the encoding process, different methods are used to encode different parts of the octree structure. The octree structure is divided into two parts. The part above the third-to-last level of the octree is encoded using the existing octree encoding algorithm. For the part below the third-to-last level of the octree, the dependency relationship between nodes and upper-level nodes is broken. For each leaf node, a path for its next three levels is generated separately. The path contains the node number of each level, and the path information of each leaf node is stored using a hash method with the node number of the fourth-to-last level as the key, so as to ensure that the corresponding node can be found quickly after decoding.

[0041] Furthermore, the specific steps of step S1.5 are as follows:

[0042] The inverse Morton sequence decodes the one-dimensional position index of a point into a two-dimensional coordinate, where the values ​​of the two coordinate axes are the even-numbered bits and odd-numbered bits in the binary representation of the one-dimensional position index, respectively. Then, the generated two-dimensional coordinate is used to map the color information of the point to the corresponding position in the image.

[0043] Furthermore, in step S2.1, the inter-frame multiplexing formula used to reconstruct the point cloud blocks using the transformation matrix of the previous frame's point cloud blocks is as follows:

[0044] C(i,j)=A 3X3 C(i-1,j)S+T 3X1 S

[0045] Where C(i,j) represents the j-th point cloud block in the i-th frame, and C(i-1,j) represents the j-th point cloud block in the (i-1)-th frame. Based on the transformation matrix H(i,j) of the point cloud block in the previous frame (step S1.3) and the j-th point cloud block C(i-1,j) in the (i-1)-th frame (i.e., the point cloud block in the previous frame), the point cloud block in the current frame can be obtained by rotating, translating, and scaling the point cloud block in the previous frame.

[0046] Furthermore, the specific steps of step S2.2 are as follows:

[0047] Step S2.2.1: Based on the fact that the decoding speed of the last three layers of the octree drops sharply, the octree structure is divided into two parts at the third-to-last layer of the octree. The encoded stream above the third-to-last layer of the octree is decoded using the existing octree decoding method to obtain the first part of the encoded stream; the encoded stream below the third-to-last layer of the octree is decoded in parallel for each node to obtain the second part of the encoded stream.

[0048] Step S2.2.2: Optimize the allocation of computing resources during mobile decoding. Use CPU resources to decode the first part of the encoded stream, and use GPU resources to decode the second part of the encoded stream in parallel to refine the point cloud position, and render it directly in a fully parallel manner.

[0049] Furthermore, the specific steps of step S2.3 are as follows:

[0050] The Morton sequence converts the two-dimensional coordinates of a point on a two-dimensional image into binary, and uses the even and odd bits of the binary representation as one-dimensional position indices to obtain one-dimensional position indices. Then, the decoded color information and the decoded geometric information are aligned and stitched together to restore the original volumetric video.

[0051] The beneficial effects of this invention are:

[0052] This invention provides a volumetric video encoding and decoding method based on inter-frame multiplexing. It utilizes the redundancy between volumetric video frames to further compress data, thereby reducing the volumetric video transmission stream and significantly lowering the bandwidth required by the volumetric video streaming system, thus optimizing volumetric video playback. Compared with existing technologies, this invention has the following advantages:

[0053] 1) This invention achieves the goal of significantly reducing the bandwidth required for transmission by employing inter-frame multiplexing technology in the field of volumetric video streaming systems;

[0054] 2) This invention is the first to use a deep learning point cloud registration model based on geometric and color information to perform motion estimation between adjacent frames of a volumetric video, achieving fast and accurate removal of redundant information;

[0055] 3) This invention is the first to utilize a 2D video coding algorithm to compress the color information of a volumetric video, thereby achieving inter-frame coding of the color information of the volumetric video;

[0056] 4) This invention solves the problem of insufficient decoding frame rate on mobile devices in decoding algorithms based on inter-frame multiplexing. Attached Figure Description

[0057] Figure 1 This is a flowchart of a volumetric video encoding and decoding method based on inter-frame multiplexing according to the present invention. Detailed Implementation

[0058] Firstly, such as Figure 1 As shown, this invention provides a volumetric video encoding and decoding method based on inter-frame multiplexing. See also... Figure 1 The present invention provides a volumetric video encoding and decoding method based on inter-frame multiplexing, which mainly includes the following steps:

[0059] Step S1: Inter-frame multiplexing coding;

[0060] The inter-frame multiplexing coding section mainly includes six steps: frame similarity detection, point cloud block similarity detection, inter-frame multiplexing, octree optimized coding algorithm, inverse Morton order reordering, and 2D video coding algorithm. The specific operation flow is as follows:

[0061] Step S1.1: Image similarity detection;

[0062] In the field of 2D video research, determining the similarity of images using algorithms such as structural similarity is a common practice. Volumetric video, composed of stereoscopic video data (3D data), uses millions of colored 3D points to realistically map reality into the digital world, providing users with a highly immersive experience. However, performing similarity matching on millions of point clouds is computationally and time-consuming. Therefore, this invention designs an image similarity detection module based on 2D images. It projects volumetric video (stereoscopic video data) from different frames at different angles into 2D images, then uses structural similarity algorithms to quickly detect the similarity of these 2D images to determine if there is significant inter-frame redundancy in the point clouds of adjacent frames. If the redundancy is significant, a computationally intensive and time-consuming point cloud block similarity detection module is then used for matching.

[0063] Step S1.2: Point cloud block similarity detection;

[0064] For adjacent frames that are relatively similar after image similarity detection, block processing and further point cloud block similarity detection are required. Therefore, this invention designs a point cloud block similarity detection module and embeds a heuristic algorithm within it to quickly detect whether two point cloud blocks are similar. First, it is necessary to generate the matching relationship between the points of the two point cloud blocks: traversing each point in the source point cloud block to find the point closest to the target point cloud block, and obtaining the matching relationship of the corresponding points. Two point cloud blocks are considered similar when the following three rules are met simultaneously:

[0065] (1) The error between N(i,j) and N(i+1,j) does not exceed 10%;

[0066] (2) The average value is less than or equal to 0.01m;

[0067] (3) 80% Less than 0.01m.

[0068] Where N(i,j) represents the number of points in the j-th point cloud block of the i-th frame, N(i+1,j) represents the number of points in the j-th point cloud block of the (i+1)-th frame, and P(i,j,u) represents point u in the j-th point cloud block of the i-th frame. This represents the point that is closest to the j-th point cloud block in the (i+1)-th frame. Representing point u and point The Euclidean distance between them. These three rules can be used to filter out more similar point cloud blocks, which can then be used as input to the inter-frame multiplexing module.

[0069] Step S1.3: Inter-frame multiplexing;

[0070] Based on the similar point cloud patches obtained in step S1.2, downsampling is performed and used as input to the point cloud registration algorithm to quickly obtain the corresponding transformation matrix. However, traditional point cloud registration algorithms can usually only handle small-scale point cloud data, becoming very slow or impossible to handle large-scale point cloud data. Furthermore, traditional point cloud registration algorithms rely solely on the geometric information of the point cloud for registration, ignoring its color information. Therefore, this invention designs an inter-frame multiplexing module and embeds a deep learning point cloud registration model based on sparse convolution within it.

[0071] Specifically, the point cloud geometric information in the point cloud block is first projected onto the mesh; for each mesh cell, the center point inside is calculated; for each center point, the relative position of its surrounding points is calculated and stored in a sparse tensor; through the above steps, the point cloud geometric information can be converted into a sparse tensor as input to the deep learning point cloud registration model.

[0072] The deep learning point cloud registration model based on sparse convolution mainly comprises four components: a geometric information encoding module, a color information encoding module, an attention fusion module, and a decoder module. The geometric information encoding module uses four sparse tensor convolutional layers to extract local and global features of the point cloud's geometric information. The color information encoding module employs a pre-trained ResNet34 (deep residual network), which effectively extracts color information features. The attention fusion module fuses the outputs of the geometric and color information encoding modules using an attention mechanism to generate a unique point cloud description vector for each point cloud block. The decoder module takes the point cloud description vectors encoded from the two point cloud blocks as input. It decodes each point cloud description vector using four sparse tensor convolutional layers and simultaneously inputs the decoded information from both point cloud description vectors into a fully connected layer. The fully connected layer outputs the transformation matrix between the two point cloud blocks as an intermediate file for inter-frame multiplexing.

[0073] The obtained intermediate file can be represented as:

[0074]

[0075] Where H(i,j) represents the intermediate file when the j-th point cloud block in the i-th frame uses inter-frame multiplexing, which is the transformation matrix relative to the point cloud block in the previous frame. A 3X3 Let T represent the rotation matrix. 3X1 O represents the translation vector. 1X3 Let S represent the zero vector, and let a represent the overall scaling factor. 11 -a 13 These represent the scaling factors on the x-axis, y-axis, and z-axis, respectively. 21 -a 23 These represent the shearing factors on the x-axis, y-axis, and z-axis, respectively. 31 -a 33 t represents the rotation factors on the x-axis, y-axis, and z-axis, respectively. x t represents the translation along the x-axis. y t represents the translation along the y-axis. z This represents the translation along the z-axis.

[0076] Step S1.4: Octree optimized coding algorithm;

[0077] This invention uses an optimized octree coding algorithm to intra-frame encode the geometric information of volumetric video (stereo video data) and outputs an intermediate file of the geometric information. When encoding using the traditional octree coding algorithm, the decoding speed of the last three layers drops sharply, severely limiting the decoding speed on mobile devices. Therefore, this invention designs an optimized octree coding module. During the encoding process, different methods are used to encode different parts of the octree structure. The octree structure is divided into two parts: the part above the third-to-last layer is encoded using the traditional octree coding algorithm; for the part below the third-to-last layer, the dependency relationship between nodes and upper-level nodes is broken. For each leaf node, a separate path for its last three layers is generated. The path contains the node number of each layer, and the path information of each leaf node is stored using a hash method with the node number of the fourth-to-last layer as the key, thus ensuring that the corresponding node can be quickly found after decoding.

[0078] Step S1.5: Reverse Morton order rearrangement;

[0079] Compressing color information in volumetric video (stereoscopic video data) using 2D video coding algorithms requires projecting the color information of a single frame onto a two-dimensional plane. However, using the traditional raster scanning method, which arranges color information in an S-shape from left to right on the image, results in significant color differences in the vertical direction, leading to poor intra-frame compression. Therefore, this invention employs an inverse Morton order rearrangement module to rearrange the color information of each frame of the volumetric video and project it onto a two-dimensional plane to form an image. Specifically, the inverse Morton order decodes the one-dimensional position index of a point into a two-dimensional coordinate, where the values ​​of the two coordinate axes are the even-numbered and odd-numbered bits in the binary representation of the one-dimensional position index. Subsequently, the generated two-dimensional coordinates are used to map the color information of that point to the corresponding position in the image. This method maximizes the locality of color between adjacent points within an 8×8 pixel block, generating a smoother color pattern for the color information of each frame of the volumetric video, achieving better intra-frame compression.

[0080] Step S1.6: 2D video coding algorithm;

[0081] Existing methods have low compression ratios for color information in volumetric videos, failing to fully utilize inter-frame redundancy of color information and thus failing to adequately address the high bandwidth requirements for color information in volumetric videos. Therefore, this invention designs a 2D video encoding module and embeds a color information compression algorithm based on 2D video compression technology within it. In step S1.5, the color information of each frame of the volumetric video is projected onto a two-dimensional plane using inverse Morton order to form an image. The images from different frames of the volumetric video are then stitched together in their original order to form a video stream. The existing H.264 algorithm is used to perform intra-frame and inter-frame compression on this video stream. This algorithm considers the inter-frame redundancy of color information and achieves a better compression ratio, thereby reducing the bandwidth requirements during transmission.

[0082] Step S2: Inter-frame multiplexing decoding;

[0083] The inter-frame multiplexing decoding part mainly includes three steps: inter-frame decoding, octree optimized decoding algorithm, and 2D video decoding algorithm. The specific operation flow is as follows:

[0084] Step S2.1: Inter-frame decoding;

[0085] First, it is determined whether the intermediate file is the inter-frame multiplexing intermediate file in step S1.3, i.e., whether inter-frame multiplexing coding is used. If so, for intermediate files using inter-frame multiplexing coding, the present invention will reconstruct the point cloud blocks based on the point cloud blocks of the previous frame and the transformation matrix of the point cloud blocks of the previous frame. If not, steps S2.2 and S2.3 are executed respectively. The inter-frame multiplexing formula used during reconstruction is as follows:

[0086] C(i,j)=A 3X3 C(i-1,j)S+T 3X1 S

[0087] Where C(i,j) represents the j-th point cloud block in the i-th frame, and C(i-1,j) represents the j-th point cloud block in the (i-1)-th frame. Based on the transformation matrix H(i,j) of the point cloud block in the previous frame (step S1.3) and the j-th point cloud block C(i-1,j) in the (i-1)-th frame (i.e., the point cloud block in the previous frame), the point cloud block in the current frame can be obtained by rotating, translating, and scaling the point cloud block in the previous frame.

[0088] Step S2.2: Octree-optimized decoding algorithm;

[0089] For video frames that are not intermediate files in step S1.3 (i.e., those not using inter-frame multiplexing coding), the encoded volumetric video geometric information needs to be decoded. This invention uses an optimized octree decoding algorithm to decode the volumetric video geometric information and outputs the decoded volumetric video geometric information. The decoding frame rate is guaranteed by improving the octree structure and rationally allocating GPU and CPU resources used during mobile decoding. The specific operation steps are as follows:

[0090] Step S2.2.1: Based on the fact that the decoding speed of the last three layers of the octree drops sharply, the octree structure is divided into two parts at the third-to-last layer. The encoded stream above the third-to-last layer of the octree uses the traditional octree decoding method; the encoded stream below the third-to-last layer of the octree uses parallel decoding for each node.

[0091] Step S2.2.2 Optimizes the allocation of computing resources during mobile decoding. The encoded stream of volumetric video geometric information includes a traditional octree encoded stream and an optimized octree encoded stream of the last three layers of the octree. This invention uses CPU resources to decode the first part of the encoded stream, namely the traditional octree encoded stream, and simultaneously uses GPU resources to decode the second part of the encoded stream, namely the optimized octree encoded stream of the last three layers of the octree, in parallel to refine the point cloud position, and renders it directly in a fully parallel manner.

[0092] Step S2.3: 2D video decoding algorithm;

[0093] The volumetric video color information is decoded using an H.264 decoder and then reordered using Morton's order. Specifically, Morton's order converts the two-dimensional coordinates of a point on a two-dimensional image into binary, and uses the even and odd bits of the binary representation of the one-dimensional position number as the one-dimensional position number. Then, the decoded color information is aligned with the decoded geometric information and stitched together to restore the original volumetric video (stereoscopic video data).

[0094] Secondly, the present invention provides a volumetric video encoding and decoding system based on inter-frame multiplexing, mainly used to implement the volumetric video encoding and decoding method based on inter-frame multiplexing described in the first aspect above. The system mainly includes the following modules:

[0095] The inter-frame multiplexing coding module and the inter-frame multiplexing decoding module, wherein the inter-frame multiplexing coding module mainly includes:

[0096] The image similarity detection module is used to render a two-dimensional image from different angles for each frame of volume video, and then use a structural similarity algorithm to quickly detect the image similarity between adjacent frames.

[0097] The point cloud block similarity detection module is used to segment adjacent frames that are similar after image similarity detection and use heuristic algorithms to determine the similarity between point cloud blocks.

[0098] The inter-frame multiplexing module is used to register similar point cloud blocks using a deep learning point cloud registration model based on sparse convolution, calculate the transformation matrix between the two point cloud blocks to replace the original point cloud blocks, and use this transformation matrix as the intermediate file for inter-frame multiplexing.

[0099] The octree optimized coding module is used to perform intra-frame coding on the geometric information of a single-frame point cloud and output an intermediate file of the geometric information.

[0100] The reverse Morton order rearrangement module is used to rearrange the color information of each frame of a volumetric video and project it onto a two-dimensional plane to form an image.

[0101] The 2D video encoding module uses a color information compression algorithm based on 2D video compression technology to project the color information of each frame of the volumetric video onto a two-dimensional plane in reverse Morton order to form an image. The images of different frames of the volumetric video are then stitched together in their original order to form a video stream. The existing H.264 algorithm is used to perform intra-frame and inter-frame compression on the video stream, and an intermediate file of color information is output.

[0102] The judgment module is used to determine whether point cloud blocks use inter-frame multiplexing coding;

[0103] The inter-frame multiplexing decoding module mainly includes:

[0104] The inter-frame decoding module is used to reconstruct point cloud blocks in intermediate files using inter-frame multiplexing coding based on the point cloud blocks of the previous frame and the transformation matrix of the point cloud blocks of the previous frame.

[0105] The octree-optimized decoding module is used to decode the encoded volumetric video geometric information of video frames that do not use inter-frame multiplexing coding using an octree decoding algorithm, and output the decoded volumetric video geometric information.

[0106] The 2D video decoding module is used to decode the volumetric video color information encoding stream using an H.264 decoder and reorder it using Morton order to output the decoded volumetric video color information; it is also used to stitch together the decoded volumetric video geometric information and volumetric video color information to obtain single-frame volumetric video data.

[0107] To verify the feasibility of the volumetric video encoding and decoding method based on inter-frame reuse of the present invention, some verification experiments were conducted. The results show that, compared with the single-frame volumetric video encoding and decoding method, the overall video stream compression ratio of the volumetric video encoding and decoding method based on inter-frame reuse of the present invention is improved from approximately 9 to 31.27, with the compression ratio of geometric information improving from approximately 12 to approximately 19. Point cloud blocks with approximately 21% similarity between adjacent frames can be reused inter-frame. When using a lossless 2D video encoding and decoding algorithm, the color information compression ratio can be improved from approximately 7 to 10.4 without any loss of color information. When using a lossy 2D video encoding and decoding algorithm, the color information compression ratio can be improved from approximately 7 to approximately 50, and the structural similarity between the rendered image and the lossless image is maintained at approximately 0.95, with minimal loss of visual quality. Furthermore, the present invention achieves a decoding frame rate of over 50fps on mobile devices and supports smooth six-degrees-of-freedom rendering of volumetric video on mobile devices.

[0108] This invention discloses a volumetric video encoding and decoding method based on inter-frame multiplexing. By utilizing inter-frame multiplexing, redundant information between adjacent frames is considered during encoding, and redundant information is eliminated during transmission to reduce the bandwidth required for existing volumetric video transmission. This invention can further reduce the required bandwidth on the basis of existing volumetric video streaming systems, lowering the application threshold for volumetric videos. It can be applied to applications such as virtual reality and augmented reality, effectively solving the problem of high bandwidth requirements in volumetric video streaming systems.

[0109] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A volumetric video encoding and decoding method based on inter-frame multiplexing, characterized in that, Includes the following steps: Step S1: Inter-frame multiplexing coding; Step S1.1: Image similarity detection; The volumetric video of different frames is projected into a two-dimensional image from different angles, and then a structural similarity algorithm is used to quickly detect the image similarity between adjacent frames. Step S1.2: Point cloud block similarity detection; Adjacent frames that are similar after image similarity detection are divided into blocks, and a heuristic algorithm is used to determine the similarity between two point cloud blocks; Step S1.3: Inter-frame multiplexing; After downsampling two point cloud blocks that are similar after point cloud block similarity detection, they are used as input to a deep learning point cloud registration model based on sparse convolution for registration. The transformation matrix between the two point cloud blocks is calculated, and this transformation matrix is ​​used as an intermediate file for inter-frame multiplexing. Step S1.4: Octree optimized coding algorithm; An octree optimized coding algorithm is used to intra-frame encode the geometric information of the volumetric video, and an intermediate file of the geometric information is output. Step S1.5: Reverse Morton order rearrangement; The color information of each frame of the volumetric video is rearranged in reverse Morton order and projected onto a two-dimensional plane to form an image. Multiple frames of images are then stitched together to form a video stream. Step S1.6: 2D video coding algorithm; The color information compression algorithm based on 2D video compression technology is used to stitch the images of different frames of a large video into a video stream in their original order, and the H.264 algorithm is used to perform intra-frame and inter-frame compression on the video stream to output an intermediate file of color information. Step S2: Inter-frame multiplexing decoding; Step S2.1: Inter-frame decoding; First, determine whether the intermediate file is the intermediate file for inter-frame multiplexing in step S1.

3. If so, reconstruct the point cloud block using the transformation matrix of the point cloud block in the previous frame. If not, execute steps S2.2 and S2.3 respectively. Step S2.2: Octree-optimized decoding algorithm; For intermediate files that are not used in the inter-frame multiplexing process in step S1.3, the volumetric video geometric information is decoded using an octree-optimized decoding algorithm, and the decoded volumetric video geometric information is output. The decoding frame rate is guaranteed by improving the octree structure and reasonably allocating the GPU and CPU resources used during mobile decoding. Step S2.3: 2D video decoding algorithm; For intermediate files that are not used in the inter-frame multiplexing process in step S1.3, the volumetric video color information is decoded using a 2D video decoding algorithm, and the decoded volumetric video color information is output. The Morton order is used to reorder the data, and the decoded volumetric video geometric information and volumetric video color information are spliced ​​together to obtain single-frame volumetric video data.

2. The volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, The specific steps for step S1.2 are as follows: By traversing each point in the source point cloud block, the point closest to the target point cloud block is found, thus obtaining the matching relationship between the points in the two point cloud blocks. Two point cloud blocks are considered similar when the following three rules are met simultaneously: (1) The error between N(i,j) and N(i+1,j) does not exceed 10%; (2) The average value is less than or equal to 0.01m; (3) 80% Less than 0.01m; Where N(i,j) represents the number of points in the j-th point cloud block of the i-th frame, N(i+1,j) represents the number of points in the j-th point cloud block of the (i+1)-th frame, and P(i,j,u) represents point u in the j-th point cloud block of the i-th frame. This represents the point that is closest to the j-th point cloud block in the (i+1)-th frame. Representing point u and point The Euclidean distance between them; similar point cloud blocks are selected by filtering out the above three rules.

3. The volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, The specific steps for step S1.3 are as follows: The point cloud geometry information in the point cloud block is projected into the mesh; for each mesh cell, the center point inside is calculated; for each center point, the relative position of its surrounding points is calculated and stored in a sparse tensor. The point cloud geometry information is converted into a sparse tensor and used as input to a deep learning point cloud registration model.

4. The volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, In step S1.3, the deep learning point cloud registration model based on sparse convolution includes: a geometric information encoding module, a color information encoding module, an attention fusion module, and a decoder module. The geometric information encoding module uses four sparse tensor convolutional layers to extract local and global features of the point cloud geometric information. The color information encoding module uses a pre-trained ResNet34 to extract color information features. The attention fusion module fuses the outputs of the geometric information encoding module and the color information encoding module based on an attention mechanism to generate a unique point cloud description vector for each point cloud block. The decoder module takes as input the point cloud description vectors encoded by the two point cloud blocks respectively. The decoder module decodes each point cloud description vector using four sparse tensor convolutional layers and simultaneously inputs the decoded information from the two point cloud description vectors into a fully connected layer. The fully connected layer outputs the transformation matrix between the two point cloud blocks as an intermediate file for inter-frame multiplexing.

5. A volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, In step S1.3, the intermediate file for inter-frame multiplexing is represented as follows: Where H(i,j) represents the intermediate file when the j-th point cloud block in the i-th frame uses inter-frame multiplexing, which is the transformation matrix relative to the point cloud block in the previous frame. A 3X3 Let T represent the rotation matrix. 3X1 O represents the translation vector. 1X3 Let S represent the zero vector, and let a represent the overall scaling factor. 11 -a 13 These represent the scaling factors on the x-axis, y-axis, and z-axis, respectively. 21 -a 23 These represent the shearing factors on the x-axis, y-axis, and z-axis, respectively. 31 -a 33 t represents the rotation factors on the x-axis, y-axis, and z-axis, respectively. x t represents the translation along the x-axis. y t represents the translation along the y-axis. z This represents the translation along the z-axis.

6. The volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, The specific steps for step S1.4 are as follows: During the encoding process, different methods are used to encode different parts of the octree structure. The octree structure is divided into two parts. The part above the third-to-last level of the octree is encoded using the existing octree encoding algorithm. For the part below the third-to-last level of the octree, the dependency relationship between nodes and upper-level nodes is broken. For each leaf node, a path for its next three levels is generated separately. The path contains the node number of each level, and the path information of each leaf node is stored using a hash method with the node number of the fourth-to-last level as the key, so as to ensure that the corresponding node can be found quickly after decoding.

7. The volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, The specific steps for step S1.5 are as follows: The inverse Morton sequence decodes the one-dimensional position index of a point into a two-dimensional coordinate, where the values ​​of the two coordinate axes are the even-numbered bits and odd-numbered bits in the binary representation of the one-dimensional position index, respectively. Then, the generated two-dimensional coordinate is used to map the color information of the point to the corresponding position in the image.

8. A volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, In step S2.1, the inter-frame multiplexing formula used to reconstruct the point cloud blocks using the transformation matrix of the previous frame's point cloud blocks is as follows: C(i,j)=A 3X3 C(i-1,j)S+T 3X1 S Where C(i,j) represents the j-th point cloud block in the i-th frame, and C(i-1,j) represents the j-th point cloud block in the (i-1)-th frame; based on the transformation matrix H(i,j) of the point cloud block in the previous frame (step S1.3) and the j-th point cloud block C(i-1,j) in the (i-1)-th frame, i.e. the point cloud block in the previous frame, the point cloud block in the current frame can be obtained by rotating, translating and scaling the point cloud block in the previous frame.

9. A volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, The specific steps for step S2.2 are as follows: Step S2.2.1: Based on the fact that the decoding speed of the last three layers of the octree drops sharply, the octree structure is divided into two parts at the third-to-last layer of the octree. The encoded stream above the third-to-last layer of the octree is decoded using the existing octree decoding method to obtain the first part of the encoded stream; the encoded stream below the third-to-last layer of the octree is decoded in parallel for each node to obtain the second part of the encoded stream. Step S2.2.2: Optimize the allocation of computing resources during mobile decoding. Use CPU resources to decode the first part of the encoded stream, and use GPU resources to decode the second part of the encoded stream in parallel to refine the point cloud position, and render it directly in a fully parallel manner.

10. A volumetric video encoding and decoding method based on inter-frame multiplexing according to claim 1, characterized in that, The specific steps for step S2.3 are as follows: The Morton sequence converts the two-dimensional coordinates of a point on a two-dimensional image into binary, and uses the even and odd bits of the binary representation as one-dimensional position indices to obtain one-dimensional position indices. Then, the decoded color information and the decoded geometric information are aligned and stitched together to restore the original volumetric video.

Images