A volume video transmission method based on viewport adaptation and hybrid coding

Abstract

A volumetric video transmission method based on viewport adaptation and hybrid coding, belonging to the field of volumetric video transmission, includes: viewport adaptation: predicting the viewport from which the user will view the next frame of volumetric video and transmitting only the volumetric video within that viewport; hybrid coding: segmenting the current frame into tiles and specifying either an inter-frame entropy coding algorithm or an intra-frame entropy coding algorithm for encoding each tile; adaptive multi-threaded decoding: employing an adaptive multi-threaded decoding method, where multiple threads simultaneously decode using different decoding methods based on the different encoding methods of the received tiles. This invention utilizes viewport adaptation technology to transmit only the content within the user's viewport, significantly reducing the amount of data transmitted; since there is redundant information between volumetric video frames, this invention uses inter-frame coding technology to further compress the volumetric video size, reducing the amount of data that needs to be transmitted; this invention achieves low-bandwidth, high-quality transmission of volumetric video, improving the user's viewing experience.

Description

Technical Field

[0001] This invention belongs to the field of volumetric video transmission technology, specifically relating to a volumetric video transmission method based on viewport adaptation and hybrid coding. Background Technology

[0002] Volumetric video is increasingly being used in virtual reality, augmented reality, telepresence, and remote collaboration. However, the transmission of volumetric video requires significant bandwidth, which can lead to high latency, low video quality, and high bandwidth costs. Therefore, much work has been dedicated to reducing the bandwidth consumption of volumetric video.

[0003] The paper "Bo Han, Yu Liu, and Feng Qian. 2020. ViVo: Visibility-Aware Mobile Volumetric Video Streaming. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (London, United Kingdom) (MobiCom'20). Association for Computing Machinery, New York, NY, USA, Article 11, 13 pages." discloses a volumetric video streaming system for mobile devices that considers the visibility of objects in a 3D scene. ViVo adaptively encodes and transmits only the visible portion of the volumetric video using visibility information, reducing the amount of data that needs to be transmitted and improving the overall video quality. ViVo uses linear regression to predict the user's viewport and then uses the predicted viewport to determine the encoding parameters for each object in the scene. This allows ViVo to use a low bitrate for invisible objects and a higher bitrate for visible objects, thereby improving the overall video quality.

[0004] The paper "Kyungjin Lee et al. 2020. GROOT: A Real-Time Streaming System of High-Fidelity Volumetric Videos. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (London, United Kingdom) (MobiCo m'20). Association for Computing Machinery, New York, NY, USA, Article 57, 14 pages." discloses a system that combines video compression and network optimization techniques to reduce the amount of data that needs to be transmitted while maintaining high video quality. A key technique used by GROOT is viewport adaptation, which adjusts the video resolution and bitrate based on the user's head posture and gaze direction to match the user's current viewport. This allows GROOT to stream only the volumetric video portion currently visible to the user, thereby reducing the amount of data that needs to be transmitted and improving video quality.

[0005] However, existing methods, including ViVo and GROOT, compress and transmit each frame independently, ignoring the large amount of redundant information between frames, thus missing the opportunity to reduce transmission bandwidth consumption by utilizing redundant information. Summary of the Invention

[0006] The purpose of this invention is to provide a volumetric video transmission method based on viewport adaptation and hybrid coding. This volumetric video transmission method solves the problem of missing reference frames in inter-frame coding, thereby enabling the simultaneous use of viewport adaptation and hybrid inter-frame coding to effectively reduce bandwidth consumption in volumetric video stream transmission while ensuring high video quality.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] The present invention provides a volumetric video transmission method based on viewport adaptation and hybrid coding, comprising the following steps:

[0009] Step 1: Viewport adaptation;

[0010] Predict the viewport from which the user will view the next frame of volumetric video, and only transmit the volumetric video within that viewport;

[0011] Step 2: Mixed encoding;

[0012] The current frame is divided into tiles, and each tile is assigned to either an inter-frame entropy coding algorithm or an intra-frame entropy coding algorithm for encoding.

[0013] Step 3: Adaptive multi-threaded decoding;

[0014] An adaptive multi-threaded decoding method is adopted, in which multiple threads simultaneously decode the tiles using different decoding methods based on the different encoding methods of the received tiles.

[0015] Furthermore, in step one, LSTM is used to predict the viewport of the next frame of volumetric video that the user will watch. First, the viewport information of the past 10 users is used as input to predict the viewport information of the next frame of volumetric video that the user will watch. The predicted viewport information is then converted into a view frustum, and only the volumetric video data within that view frustum is transmitted.

[0016] Furthermore, in the process of using LSTM to predict the viewport of the next frame of volumetric video viewed by the user, it is necessary to manipulate the tiles, that is, to mark any part of the tile as content within the user's viewport as such, provided that any part of the tile is within the view frustum.

[0017] Furthermore, in the process of using LSTM to predict the viewport of the next frame of volumetric video viewed by the user, the input and output of the LSTM model are as follows:

[0018] The input is a vector representation of the past 10 user viewport information; the vector representation of the past 10 user viewport information is X = [x1, y1, z1, yaw1, pitch1, roll1, ..., x10, y10, z10, yaw10, pitch10, roll10];

[0019] The output is a vector representation of the predicted viewport information of the next frame of volumetric video viewed by the user; the vector representation of the predicted viewport information of the next frame of volumetric video viewed by the user is Y = [x_next, y_next, z_next, ya w_next, pitch_next, roll_next];

[0020] Where x1-x10 represent the x-coordinates of the user's viewport in 3D space over the past 10 times, x_next represents the predicted x-coordinate of the user's viewport when viewing the next frame of volumetric video, y1-y10 represent the y-coordinates of the user's viewport in 3D space over the past 10 times, y_next represents the predicted y-coordinate of the user's viewport when viewing the next frame of volumetric video, z1-z10 represent the z-coordinates of the user's viewport in 3D space over the past 10 times, z_next represents the predicted z-coordinate of the user's viewport when viewing the next frame of volumetric video, and yaw1-yaw10 represent the x-coordinates of the user's viewport in 3D space over the past 10 times, respectively. The Euler angle of the viewport rotating around the vertical axis is the yaw angle. yaw_next represents the predicted Euler angle of the viewport rotating around the vertical axis when the user views the next frame of volumetric video. pitch1-pitch10 represent the Euler angles of the user viewport rotating around the horizontal axis (pitch angle) over the past 10 frames. pitch_next represents the predicted Euler angle of the user viewport rotating around the horizontal axis when the user views the next frame of volumetric video. roll1-roll10 represent the Euler angles of the user viewport rotating around the vertical axis (roll angle) over the past 10 frames. roll_next represents the predicted Euler angle of the user viewport rotating around the vertical axis when the user views the next frame of volumetric video.

[0021] Furthermore, in step two, the inter-frame entropy coding algorithm uses the viewport information of the previous frame's volume video to encode the current frame, while the intra-frame entropy coding algorithm only uses the viewport information of the current frame's volume video for encoding.

[0022] Furthermore, in step two, the tiles in the current frame are classified according to whether they are in the previous frame: Ci and Ci-1 represent the set of tile positions in the current frame and the set of tile positions in the previous frame, respectively. The tiles in the current frame are divided into two parts: Ci∩Ci-1 corresponds to the tiles in the previous frame, and Ci-Ci-1 corresponds to the tiles that are not in the previous frame. The inter-frame entropy coding algorithm is used to encode the tiles in Ci∩Ci-1, and the intra-frame entropy coding algorithm is used to encode the tiles in Ci-Ci-1.

[0023] Furthermore, in step three, decoder states are used to store the state of each tile, where state 0 indicates no transmission, state 1 indicates waiting for intra-frame decoding, state 2 indicates waiting for inter-frame decoding, state 3 indicates decoding in progress, and state 4 indicates decoding complete. At the same time, an encoding queue is used to store the encoding of the received frames, a state queue is used to store the state of the received frames, and a result queue is used to store the frames that need to be rendered and played after decoding.

[0024] Furthermore, in step three, in order to obtain a new decoding task, all tiles in the state queue are traversed to find the first tile with a state of 1, or a tile with a state of 2 but with a corresponding tile that has already been decoded in the previous frame. Then, a thread lock is requested and the tile is marked as state 3. Subsequently, the thread lock is released to execute the decoding task. When the task is completed, the thread lock is requested again and the tile is marked as state 4. Finally, the thread lock is released.

[0025] The beneficial effects of this invention are:

[0026] This invention aims to enable low-bandwidth, high-quality transmission of volumetric video. Among common volumetric data formats, point cloud format offers high flexibility and simplicity; however, the massive data volume of point cloud format results in video sizes significantly larger than 2D videos. Streaming media's raw volumetric video requires 1 to 3 Gbps of bandwidth, far exceeding the common bandwidth capabilities of mobile devices and causing playback delays. However, there is room for optimization. First, this invention can utilize viewport adaptation technology. Since users have limited viewports, if this invention only transmits content within the user's viewport, it will greatly reduce the amount of data transmitted. Second, there is redundant information between volumetric video frames. This invention can utilize inter-frame coding technology to further compress the volumetric video size, reducing the amount of data that needs to be transmitted. However, in a viewport-adaptive-based transmission system, inter-frame coding may be unusable due to: 1) missing data dependencies; 2) complex data dependencies, making multi-threaded decoding difficult. To solve these two problems, this invention develops a volumetric video transmission method based on viewport adaptation and hybrid coding. This invention not only significantly saves bandwidth for volumetric video transmission but also effectively improves the quality of video viewing for users. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] To improve computational efficiency and reduce the amount of data transmitted, this invention decomposes frames into smaller tiles for processing. First, viewport adaptation is performed. During viewport adaptation, the viewport from which the user will view the next frame of video is predicted. After obtaining the prediction result, this invention only encodes the tiles within the user's viewport. During this process, the server adjusts the frame encoding method based on network quality, adaptively transmitting frames generated by either inter-frame entropy encoding or intra-frame entropy encoding algorithms. Finally, the client receives the video stream and decodes it for display on the client device. The client employs an adaptive multi-threaded decoding method to improve decoding efficiency; that is, multiple threads simultaneously decode using different decoding methods based on the different encoding methods of the received tiles.

[0029] The present invention provides a volumetric video transmission method based on viewport adaptation and hybrid coding, which specifically includes the following steps:

[0030] Step 1: Viewport adaptation;

[0031] When watching volumetric video, the user's viewport is constantly changing, but this change follows a predictable pattern. By predicting the viewport from which the user will view the next frame of volumetric video and transmitting only the volumetric video within that viewport, bandwidth consumption during volumetric video transmission can be significantly reduced.

[0032] To achieve this goal, this invention proposes using LSTM (Long Short-Term Memory) networks to predict the viewport of the next frame of volumetric video viewed by the user. First, the viewport information (including position coordinates and Euler angles) of the past 10 users is used as input to predict the viewport information (including position coordinates and Euler angles) of the next frame of volumetric video viewed by the user. Finally, the predicted viewport information is converted into a view frustum, and only the volumetric video data within that view frustum is transmitted.

[0033] In using LSTM (Long Short-Term Memory) networks to predict the viewport of the next frame of volumetric video viewed by a user, tile manipulation is required. Specifically, any tile whose portion lies within the view frustum is marked as content within the user's viewport. To ensure the accuracy of viewport prediction, this invention increases the volume of the view frustum by 1.2 times.

[0034] The user viewport coordinates (x, y, z) are as follows:

[0035] x: The x-coordinate of the viewport in three-dimensional space;

[0036] y: The y-coordinate of the viewport in three-dimensional space;

[0037] z: The z-coordinate of the viewport in three-dimensional space.

[0038] The user viewport Euler angles (yaw, pitch, roll) are as follows:

[0039] yaw: Euler angle of rotation about the vertical axis, also known as yaw angle;

[0040] pitch: Euler angle of rotation about the horizontal axis, also known as pitch angle;

[0041] roll: The Euler angle of rotation about the longitudinal axis, also known as the roll angle.

[0042] The input and output of the LSTM model are as follows:

[0043] Input: A vector representation of the past 10 user viewport information, which can be represented as X = [x1, y1, z1, yaw1, pitch1, roll1, ..., x10, y10, z10, yaw10, pitch10, roll10].

[0044] Output: A vector representation of the predicted viewport information of the next frame of volumetric video viewed by the user. The vector of the predicted viewport information of the next frame of volumetric video viewed by the user can be represented as Y = [x_next, y_next, z_next, yaw_next, pitch_next, roll_next].

[0045] Step 2: Mixed encoding;

[0046] To adapt to dynamic changes in the user's viewport, two encoding algorithms are used to encode frames: (i) inter-frame entropy coding algorithm and (ii) intra-frame entropy coding algorithm. Inter-frame entropy coding algorithm and intra-frame entropy coding algorithm are the inter-frame version and intra-frame version of the S4D coding algorithm, respectively. The inter-frame entropy coding algorithm uses the viewport information of the previous frame's volume video to encode the current frame, while the intra-frame entropy coding algorithm only uses the viewport information of the current frame's volume video for encoding.

[0047] To reduce bandwidth consumption, the current frame needs to be segmented into tiles, and primarily the tiles in the viewport after viewport adaptation are transmitted. However, since the user's viewport is constantly changing, the tiles in the current frame may not be able to utilize the tiles from the previous frame for inter-frame coding, because users may prefer to view highly dynamic content, and the tiles in the previous frame are not similar to those in the current frame, failing to meet the conditions for inter-frame entropy coding algorithms. Therefore, this invention specifies that each tile should be encoded using either an inter-frame entropy coding algorithm or an intra-frame entropy coding algorithm.

[0048] When encoding tiles in the current frame using the inter-frame entropy coding algorithm, tiles at corresponding positions in the previous frame are required. However, since the user's viewport is constantly changing, the tiles at corresponding positions in the previous frame may not exist. Therefore, this invention classifies tiles in the current frame based on whether they are located in the previous frame. Specifically: C i and C i-1 These represent the set of tile positions in the current frame and the set of tile positions in the previous frame, respectively. The tiles in the current frame are divided into two parts: C i ∩C i-1 Corresponding to the tile in the previous frame, C i -C i-1 This corresponds to tiles that were not in the previous frame. During volumetric video stream transmission, an inter-frame entropy coding algorithm is selected to encode C. i ∩C i-1 The tiles in the image are selected, and an intra-frame entropy coding algorithm is chosen to encode C. i -C i-1 The tiles in the middle.

[0049] Step 3: Adaptive multi-threaded decoding;

[0050] Hybrid encoding leads to complex dependencies during decoding. Inter-frame decoding of a tile requires decoding the corresponding tile in the previous frame, while intra-frame decoding does not. To address this, this invention uses decoder states to store the state of each tile, where state 0 indicates no transmission, state 1 indicates waiting for intra-frame decoding, state 2 indicates waiting for inter-frame decoding, state 3 indicates decoding in progress, and state 4 indicates decoding complete. This invention uses an encoding queue to store the encoding of received frames, a state queue to store the states of received frames, and a result queue to store frames that need to be rendered and played after decoding. To prevent race conditions and ensure correct behavior, this invention introduces thread locks, ensuring that each thread has exclusive access to the decoder state before any modifications are made and releasing the lock after access, allowing this invention to adapt to computational resources of any scale.

[0051] To acquire a new decoding task, this invention iterates through all tiles in the state queue, finds the first tile with a state of 1, or a tile with a state of 2 but whose corresponding previous frame has already been decoded, then acquires a thread lock, marks the tile as state 3, releases the thread lock, executes the decoding task, and upon task completion, acquires the thread lock again, marks the tile as state 4, and finally releases the thread lock. This invention employs an adaptive multi-threaded decoding method, where each thread continuously acquires tasks, executes tasks, and updates state variables.

[0052] Existing methods for transmitting large-volume video employ various encoding techniques, but their compression efficiency is poor on highly dynamic frames, and they cannot achieve low-bandwidth transmission of large-volume video. This invention combines the advantages of viewport adaptation in other systems and innovates upon them by developing hybrid encoding and multi-threaded decoding, ultimately achieving high-efficiency low-bandwidth transmission of large-volume video.

[0053] By implementing and evaluating the volumetric video transmission method based on viewport adaptation and hybrid coding of the present invention on three public videos, it is demonstrated that the volumetric video transmission method of the present invention is a volumetric video streaming system that can be deployed at any time and can achieve a frame rate of 30FPS+ on everyday networks and commercial smartphones.

[0054] This invention experimentally verified multi-threaded decoding using different parameters. The results show that as the number of threads increases from 1 to 8, the system's decoding frame rate increases. When the number of threads reaches 3, the system's decoding frame rate can reach 30 FPS+ for high-quality volumetric videos. To further verify the effectiveness of hybrid coding, this invention transmits all tiles using an inter-frame coding algorithm, resulting in blank spaces in the display for tiles without reference tiles on the client side. This invention also uses SSIM as an indicator of visual quality. The results show that for Ricardo, Pizza, and Longdress (three classic volumetric video datasets), SSIM decreases from 0.9463, 0.9449, and 0.9510 to 0.8997, 0.7992, and 0.8691, respectively. This indicates that the hybrid coding method ensures low bandwidth consumption while maintaining visual quality.

[0055] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A volumetric video transmission method based on viewport adaptation and hybrid coding, characterized in that, Includes the following steps: Step 1: Viewport adaptation; Predict the viewport from which the user will view the next frame of volumetric video, and only transmit the volumetric video within that viewport; LSTM is used to predict the viewport of the next frame of volumetric video that the user will watch. First, the viewport information of the past 10 users is used as input to predict the viewport information of the next frame of volumetric video that the user will watch. The predicted viewport information is converted into a view frustum, and only the volumetric video data within the view frustum is transmitted. In the process of using LSTM to predict the viewport of the next frame of volumetric video viewed by the user, it is necessary to operate on the tiles. That is, as long as any part of the tile is within the view frustum, the tile is marked as content within the user's viewport. The input of the LSTM model is a vector representation of the past 10 user viewport information, and the output is a vector representation of the predicted viewport information of the next frame of volumetric video viewed by the user. Step 2: Mixed encoding; The current frame is divided into tiles, and each tile is assigned to either an inter-frame entropy coding algorithm or an intra-frame entropy coding algorithm for encoding. The inter-frame entropy coding algorithm uses the viewport information of the previous frame's volume video to encode the current frame, while the intra-frame entropy coding algorithm only uses the viewport information of the current frame's volume video for encoding. Tiles in the current frame are classified according to whether they are in the previous frame: Ci and Ci−1 represent the set of tile positions in the current frame and the set of tile positions in the previous frame, respectively. Tiles in the current frame are divided into two parts: Ci∩Ci−1 corresponds to tiles in the previous frame, and Ci−Ci−1 corresponds to tiles that are not in the previous frame. Inter-frame entropy coding algorithm is used to encode tiles in Ci∩Ci−1, and intra-frame entropy coding algorithm is used to encode tiles in Ci−Ci−1. Step 3: Adaptive multi-threaded decoding; An adaptive multi-threaded decoding method is adopted, in which multiple threads simultaneously decode the tiles using different decoding methods based on the different encoding methods of the received tiles. Store the state of each tile using the decoder state, store the encoding of the frame using the encoding queue, store the state of the frame using the state queue, and store the frames to be rendered and played after decoding using the result queue. Traverse the state queue to find the first tile waiting for the intra-frame decoding state, or the tile waiting for the inter-frame decoding state but which already has a corresponding tile that has completed decoding of the previous frame. Request a thread lock, mark the tile as being in the decoding process, release the thread lock, execute the decoding task, request a thread lock again, mark the tile as being in the decoding process, and release the thread lock.

2. The volumetric video transmission method based on viewport adaptation and hybrid coding according to claim 1, characterized in that, The vector representation of the past 10 user viewport information is X = [x1, y1, z1, yaw1, pitch1, roll1,..., x10, y10, z10, yaw10, pitch10, roll10]; the vector representation of the predicted user viewport information for the next frame of volumetric video is Y = [x_next, y_next, z_next, yaw_next, pitch_next, ..., x10, y10, z10, yaw10, pitch10 ... [roll_next]; where x1-x10 represent the x-coordinates of the user viewport in the past 10 times in 3D space, x_next represents the predicted x-coordinate of the user viewport when viewing the next frame of volumetric video, y1-y10 represent the y-coordinates of the user viewport in the past 10 times in 3D space, y_next represents the predicted y-coordinate of the user viewport when viewing the next frame of volumetric video, z1-z10 represent the z-coordinates of the user viewport in the past 10 times in 3D space, z_next represents the predicted z-coordinate of the user viewport when viewing the next frame of volumetric video, and yaw1-yaw10 represent the x-coordinates of the user viewport in the past 10 times in 3D space, respectively; where x1-x ... The Euler angles of the 10 user viewports rotating around the vertical axis are the yaw angles. yaw_next represents the predicted Euler angles of the user's viewport rotating around the vertical axis when viewing the next frame of volumetric video. pitch1-pitch10 represent the Euler angles of the past 10 user viewports rotating around the horizontal axis, i.e., the pitch angles. pitch_next represents the predicted Euler angles of the user's viewport rotating around the horizontal axis when viewing the next frame of volumetric video. roll1-roll10 represent the Euler angles of the past 10 user viewports rotating around the vertical axis, i.e., the roll angles. roll_next represents the predicted Euler angles of the user's viewport rotating around the vertical axis when viewing the next frame of volumetric video.