Viewport prediction and code rate decision optimization method for volume video stream

Abstract

The invention provides a viewport prediction and code rate decision optimization method for a volume video stream, and the method comprises the steps: 1, carrying out offline training, which comprises the following steps: 1-1, removing the prediction of a direction degree of freedom based on the focusing following characteristic of the sight line motion of a user and the inertia characteristic of the position motion; step 1-2, training a historical window selector; step 1-3, training a viewport prediction confidence evaluator; step 2, online reasoning which comprises the following steps: step 2-1, deploying a viewport prediction enhancement (VPE) module to carry out real-time prediction; and step 2-2, deploying a code rate decision enhancement BDE module to carry out adaptive decision. Through collaborative optimization of a viewport prediction mechanism and bandwidth resource dynamic allocation, low-lagging and high-visual-quality video transmission can be realized, a mainstream viewport prediction model and a self-adaptive code rate algorithm are compatible, and the six-degree-of-freedom volume video experience quality of a user is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia communication technology, and specifically to a viewport prediction and bitrate decision optimization method for volumetric video streams. Background Technology

[0002] With the development of the metaverse, users' demands for interactivity in video experiences are increasing. Volumetric video, as a key technology, provides complete six degrees of freedom (6-DoF) viewing capabilities, allowing users not only to change their gaze direction but also to move freely, thus delivering an immersive experience far exceeding that of traditional 360-degree videos. This technology has been driven by giants such as Apple and Google and has demonstrated enormous market potential in commercial applications.

[0003] However, the sheer volume of video data presents a significant challenge for streaming. While current mainstream viewport-aware transmission solutions can save bandwidth by transmitting only high-quality content within the user's visible area, they face two core problems in 6-DoF scenarios:

[0004] 1. High prediction complexity and large error: The prediction difficulty of six degrees of freedom is much higher than that of three degrees of freedom. The prediction error is easily amplified by nonlinearity, resulting in a significant reduction in viewport hit rate of 30%–60%.

[0005] 2. Disconnect between bitrate decision-making and prediction reliability: Existing adaptive bitrate algorithms often blindly trust prediction results, requesting high-quality content even if the prediction is unreliable. This not only wastes bandwidth but may also trigger a chain reaction of stuttering and other issues due to prediction errors, thus harming the user experience.

[0006] Therefore, there is an urgent need for a new method that can synergistically optimize the robustness of viewport prediction and the adaptability of bitrate decision-making to cope with the dynamic characteristics of 6-DoF interaction and ensure that users obtain a stable and high-quality immersive experience in complex network environments. Summary of the Invention

[0007] Objective of the Invention: The technical problem to be solved by the present invention is to address the shortcomings of existing technologies by providing a viewport prediction and bitrate decision optimization method for volumetric video streams, comprising the following steps:

[0008] Step 1, offline training, includes the following steps:

[0009] Step 1-1: Based on the focusing and following characteristics of the user's line of sight movement and the inertial characteristics of the positional movement, remove the prediction of directional degrees of freedom (pitch, yaw, roll).

[0010] Steps 1-2: Training the historical window selector: Collect user trajectory sequences as the training set, and train the agent using the Proximal Policy Optimization (PPO) algorithm. The reward function is related to the Viewport Hit Rate (VHR), enabling the agent to learn to dynamically select the historical window range that best reflects the motion inertia of the current position based on the motion trajectory displayed in the user trajectory.

[0011] Steps 1-3, train the viewport prediction confidence evaluator: construct a two-layer fully connected neural network, take the historical position acceleration variance and prediction time span as input, and the corresponding viewport hit rate as label, and train it through supervised learning so that the viewport prediction confidence evaluator can evaluate the reliability of the prediction based on real-time interaction features.

[0012] Step 2, online reasoning, includes the following steps:

[0013] Step 2-1: Deploy the Viewport Prediction Enhancement (VPE) module for real-time prediction. Before each request for a new video block, the VPE module obtains the latest user trajectory from the client buffer, calls the offline trained historical window selector, analyzes the current trajectory state, outputs the optimal window capture index, thereby filtering out the historical trajectory segments used for this viewport prediction, and performs future viewport prediction based on the historical trajectory segments.

[0014] Step 2-2: Deploy the Bitrate Decision Enhancement (BDE) module for adaptive decision-making.

[0015] In step 1-1, the historical window selector is a reinforcement learning selector built and trained in the offline training phase. The historical window selector takes the recent K position motion samples obtained by the client at a fixed sampling frequency as the state input. The recent refers to a number of position points sampled within a short time window corresponding to the prediction time span, such as K consecutive position samples sampled within 0.5 seconds to 1.5 seconds.

[0016] The historical window selector trains an agent using the Proximal Policy Optimization (PPO) algorithm. The agent refers to a policy network model optimized by the PPO algorithm. The agent's output is the historical window cutoff point index, and the action space is an integer from 1 to K, used to indicate the starting position of the trajectory window. The historical window cutoff point index is used to determine the dynamic window range. For example, when the index output by the agent is 4, it means that the trajectory interval from the latest sample to the 4th sample is used as the effective window, so that only the 4 most recent samples are retained and the earlier noise samples that are inconsistent with the current movement trend are removed.

[0017] The reward function is directly associated with the viewport hit rate calculated based on the dynamic trajectory window selected by the historical window selector. The viewport hit rate is obtained by calculating the overlap between the user's future location predicted by the spatial location prediction model (such as linear regression or LSTM) and the actual viewport area.

[0018] The agent learns and captures the inertial patterns of user position movement by maximizing the cumulative viewport hit rate. That is, the user's position coordinates in three-dimensional space (x represents the left and right direction, y represents the up and down direction, and z represents the forward and backward direction) usually maintain strong motion continuity in a short time scale. For example, in the prediction window of 0.5 to 1.5 seconds in the future, the deviation of its motion direction is usually less than a preset angle threshold (such as 20° to 40°), which is manifested as continuous motion such as walking at a constant speed or turning smoothly. The dynamic window selection strategy obtained by training with physical laws can replace the static mode of traditional fixed-length windows and significantly reduce the prediction error caused by the inclusion of non-inertial trajectories.

[0019] In step 1-1, the formula for calculating the viewport hit rate is:

[0020]

[0021] Among them, T pred T represents the set of viewport coverage tiles predicted by the client based on historical trajectories. actual This represents the set of visible segments actually viewed by the user at the future target time point corresponding to the prediction (i.e., the time after the predicted time span p); |·| represents the number of segments in the set; the viewport hit rate H is used to quantify the overlap accuracy between the predicted viewport and the actual viewport to reflect the accuracy of the viewport prediction.

[0022] In steps 1-2, the window selection problem is constructed as a sequential decision task to capture the inertial characteristics of user positional motion in a six-degrees-of-freedom experience;

[0023] The state space of the agent consists of the K most recent historical position samples, and the action space is defined as a discrete window truncation index a∈[1,K], which is used to indicate the starting position of the trajectory window;

[0024] The action space is consistent with the action space defined in step 1-1, both representing the index range from the latest sample to the historical sample.

[0025] The agent dynamically determines the length of the trajectory window from the latest sample back to the a-th sample by outputting action 'a'.

[0026] During the training phase, the viewport hit rate is used as the immediate reward, and the Proximal Policy Optimization (PPO) algorithm is used to update the policy network to maximize the cumulative reward. Through training, the agent can learn to identify and filter historical trajectory segments that best match the current motion trend. For example, when a user is moving at a constant speed in a three-dimensional space, the agent usually selects a shorter window to highlight recent inertial characteristics; while when the user turns or changes speed, the agent adaptively adjusts the window length to capture the transition phase of the motion pattern. The learning strategy based on this invention can effectively avoid fixed-length windows from including early noise samples that are irrelevant to the current motion trend, thereby improving prediction accuracy.

[0027] Steps 1-3 include: extracting historical viewport prediction cases from real user motion trajectories as training samples. The input features of each training sample include the variance of spatial degrees of freedom acceleration within the historical window and the user's acceleration value 'a' along the left-right direction (x-axis) in three-dimensional space. x The acceleration value a in the vertical direction (y-axis) y The acceleration value a in the forward and backward direction (z-axis) z And the predicted time span p;

[0028] a x a y a z Used to quantify the intensity of a user's location movement;

[0029] The output label is the viewport hit rate corresponding to the predicted viewport;

[0030] Based on the training samples, a two-layer fully connected neural network is constructed as a viewport prediction confidence evaluator. Supervised learning is used to optimize the model with Mean Squared Error (MSE) as the loss function. The training sample dataset is divided into a training set and a validation set according to a preset ratio (e.g., 8:2). The validation set is derived from the reserved portion of real user trajectory data and is used to monitor the model's generalization performance. During training, the fully connected neural network employs forward and backward propagation mechanisms: forward propagation generates the prediction confidence of the viewport hit rate, and backward propagation updates the model weights based on the error between the predicted value and the viewport hit rate. This process is iterated until the MSE metric on the validation set converges.

[0031] After training, the viewport prediction confidence estimator can predict based on real-time interaction patterns (e.g., vigorous movement leading to a...). x a y a z (When the variance increases, the output confidence level is lower) to quantify the reliability of viewport prediction and provide a risk perception basis for the adaptive bitrate ABR algorithm.

[0032] In steps 1-3, the two fully connected neural networks consist of one layer with 128 neurons and the other with 64 neurons, using ReLU as the activation function; ultimately, the two fully connected neural networks output a scalar between 0 and 1.

[0033] In step 2-1, during online inference, the viewport prediction enhancement module is first deployed. The viewport prediction enhancement module includes a historical window selector and a spatial location prediction model obtained during the offline training phase, which are used to perform real-time viewport prediction during playback.

[0034] When a new video block needs to be requested, the viewport prediction enhancement module obtains the latest user trajectory from the client buffer and calls the historical window selector trained by the Proximal Policy Optimization (PPO) algorithm in steps 1-2. The historical window selector contains an agent trained by the Proximal Policy Optimization (PPO) algorithm. The policy network corresponding to the agent is a policy function optimized by the Proximal Policy Optimization (PPO) algorithm. The agent outputs the optimal window index in real time according to the current trajectory state input, thereby dynamically selecting the relevant trajectory segment that is most consistent with the current motion trend.

[0035] The relevant trajectory segments are fed into a spatial location prediction model (such as computing efficient linear regression LR or LSTM that can capture complex temporal dependencies) to predict the future user spatial coordinates.

[0036] Finally, the client runs on the PICO4 head-mounted display device and, combining the device's field of view (FOV) parameters with the three-dimensional geometric coordinate system of the virtual scene, calculates the spatial coverage of the predicted viewport: Let the predicted head orientation be determined by Euler angles (θ, φ) or the equivalent unit direction vector d ∈ R. 3 Let R represent the real number space; the horizontal field of view of PICO 4 is FOV. h The vertical field of view is FOV. v The predicted viewport can be expressed by the following two inequalities:

[0037]

[0038] Where u and v are the horizontal and vertical vectors of the center point of any tile in the scene relative to the user's position, respectively.

[0039] All tiles that satisfy the two inequalities constitute the set T of visible tiles for the predicted viewport. pred This calculation process forms the predicted viewport geometry filtering method proposed in this invention, which is used to link the device's field of view model with the predicted position and predicted orientation to obtain an accurate three-dimensional viewport range.

[0040] In step 2-2, the bitrate decision enhancement module is used to dynamically adjust the bitrate selection based on the reliability of viewport prediction. The client calculates the corresponding trajectory features based on the historical trajectory segments used for viewport prediction determined in step 2-1, and calls the viewport prediction confidence evaluator trained in step 1-3 to generate a confidence value representing the reliability of this viewport prediction. The confidence value is integrated into the Adaptive Bitrate Streaming (ABR) algorithm to dynamically determine the video block bitrate level corresponding to the predicted viewport, while uniformly using the lowest bitrate for transmission of content outside the predicted viewport.

[0041] In step 2-2, the confidence level is deeply integrated into the Adaptive Bitrate (ABR) algorithm through three core mechanisms to achieve risk-aware adaptive decision-making: For rule-based algorithms, the confidence level serves as a threshold condition for direct intervention; for example, when the confidence level is below 0.5, a low bitrate is forced for the predicted viewport to avoid risk, while a high bitrate is allowed to be requested to improve quality when the confidence level is above 0.8. For optimization control algorithms, the confidence level is used as a modulation factor-weighted expected quality gain for the future viewport, thereby suppressing the attraction of low-reliability predictions for aggressive bitrate selection. For learning algorithms, the confidence level serves as a key state feature input to the policy network, driving the agent to autonomously learn to adopt aggressive strategies when the confidence level is high and to switch to conservative decisions when the confidence level is low. This multi-layered integration ensures that bitrate decisions no longer blindly trust prediction results, but dynamically adjust strategies based on the reliability of the prediction, ultimately significantly improving bandwidth utilization efficiency and playback stability.

[0042] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0043] Beneficial Effects: Unlike existing technologies, this invention first proposes a six-DOF volumetric video viewport prediction enhancement module—Viewport Prediction Enhancement (VPE). By using a dynamic history window selector based on RL (PPO algorithm) to retain only samples aligned with the current user's motion trend, it suppresses long-term random motion noise. Furthermore, through analysis of a large number of user trajectories, this module uses independent predictions with directional degrees of freedom (pitch, roll, yaw) removed, aligning the spatial position with the line-of-sight direction to avoid compounding and amplifying errors. This design significantly improves viewport hit rate in the long-term domain.

[0044] Furthermore, this invention proposes a volumetric video transmission bitrate decision enhancement module—Bitrate Decision Enhancement (BDE). Since viewport prediction for six-DOF volumetric video exhibits significant fluctuations, traditional ABR algorithms assume complete reliability, leading to high false positive costs. By quantifying the prediction confidence level, the ABR algorithm can effectively mitigate these risks. This module trains a viewport prediction confidence evaluator, enabling it to dynamically assess the reliability of the prediction window result based on the user's motion characteristics and prediction duration, outputting a confidence scalar of 0-1. Based on this confidence level, the ABR decision is dynamically adjusted: high confidence allows requests for high bitrate content, while low confidence forces a reduction in bitrate to reserve bandwidth buffers. This mechanism significantly reduces network bandwidth waste.

[0045] During actual playback, after the VPE module outputs a high-precision viewport, the BDE immediately injects confidence to guide the ABR bitrate selection. The rendering layer forces the lowest bitrate rendering for non-viewport areas and records the new trajectory, feeding it back to the prediction module. This closed loop supports robustness in complex scenes and can significantly improve the user's perceived quality of experience (QoE). Attached Figure Description

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0047] Figure 1 This is a flowchart of the method architecture of the present invention.

[0048] Figure 2 This is a schematic diagram of the history window selector used in the VPE module of this invention.

[0049] Figure 3 This is a schematic diagram illustrating the training principle of the viewport prediction confidence evaluator used in the BDE module of this invention.

[0050] Figure 4 This is a diagram showing the system's operational effect as used in this invention.

[0051] Figure 5 This is a partial screenshot of the system page.

[0052] Figure 6 This is another attached image from the system page.

[0053] Figure 7 This is a performance comparison chart of the present invention under various network conditions (the comparison is between the algorithm using the collaborative optimization method described in the present invention and the basic algorithm that does not use the method, and the comparison index is the average user experience quality). Detailed Implementation

[0054] like Figure 1 As shown, this embodiment of the invention provides a viewport prediction and bitrate decision optimization method for volumetric video streams, including the following steps:

[0055] 1. Offline training

[0056] (1) Based on the focusing and following characteristics of user eye movement and the inertial characteristics of positional movement, the prediction of directional degrees of freedom (pitch, yaw, roll) is removed.

[0057] (2) Training the historical window selector: The collected user trajectory sequences are used as the training set. The PPO algorithm is optimized using the near-end strategy to train the agent. Its reward function is linked to the viewport hit rate, so that it learns to dynamically select the historical window range that best reflects the motion inertia of the current position based on the motion trajectory displayed in the user trajectory.

[0058] (3) Training the viewport prediction confidence evaluator: Construct a two-layer fully connected neural network with historical position acceleration variance and prediction time span as inputs and the corresponding viewport hit rate as labels. Train it through supervised learning so that it can evaluate the reliability of the prediction based on real-time interaction features.

[0059] 2. Online reasoning

[0060] (1) VPE module real-time prediction: The client obtains the latest user trajectory, calls the offline trained historical window selector to filter out relevant trajectory segments, inputs them into the spatial location prediction model (such as linear regression algorithm or LSTM algorithm) to obtain the future position, and calculates the predicted viewport in combination with the device field of view.

[0061] (2) BDE module adaptive decision-making: Based on the current trajectory features, the offline trained confidence estimator is called to generate the prediction confidence, and the confidence is integrated into the ABR algorithm to dynamically determine the bitrate level for the predicted viewport request; the content outside the predicted viewport is transmitted at the lowest bitrate; the system iteratively executes steps 2-1 and 2-2 in video blocks until the playback ends.

[0062] In this invention, the server-side implements lightweight video content organization based on the DASH standard: the volumetric video is arranged chronologically, each frame is divided into 3×3×4 spatial tiles, and multiple bitrate versions are generated and stored on the Nginx server through Draco encoding; the client-side player integrates two modules—the Viewport Prediction Enhancement (VPE) module dynamically filters the position motion inertia window and only predicts the spatial degrees of freedom x, y, z through reinforcement learning to eliminate orientation errors, and the Bitrate Decision Enhancement (BDE) module outputs confidence scores in real time (based on the variance of position acceleration α) through a lightweight neural network. x ,a y ,a z And the predicted duration p) drives the ABR algorithm to achieve risk-adaptive bitrate regulation (high confidence > 0.8 requests high bitrate, low confidence < 0.5 reduces bitrate to maintain smoothness); ultimately, when playing six-degrees-of-freedom volumetric videos, it can effectively reduce stuttering and improve the user's visual quality QoE.

[0063] In this invention, the VPE module significantly improves the prediction performance of six-DOF volumetric video through two core innovations: degree-of-freedom reduction and dynamic historical window selection. The module's design is based on a deep understanding of user behavior patterns: analysis shows that in most viewing scenarios, the user's gaze actively locks onto the focal object, making the directional degrees of freedom (pitch, yaw, roll) essentially derived from its spatial position, rather than independent variables. Based on this, the VPE module innovatively eliminates the independent prediction of directional degrees of freedom, simplifying the complex six-DOF problem into a three-DOF prediction of spatial position (x, y, z). This significantly reduces computational complexity and fundamentally avoids the compound amplification of prediction errors from multiple degrees of freedom. Simultaneously, the VPE module introduces a reinforcement learning-driven dynamic historical window selector to accurately capture the short-term inertial characteristics of the user's positional motion. This selector uses recent position samples as its state and dynamically selects historical trajectory windows that best match the current motion trend through agent decision-making (e.g., retaining only the most recent samples that reflect uniform or smooth turning trends), using the predicted viewport hit rate as a reward for training. This mechanism adaptively filters out early noise data that is irrelevant to the current inertia, ensuring the quality of trajectory data input to the prediction model. The VPE module improves the accuracy of six-DOF volumetric video stream viewport prediction through these two innovative optimization methods.

[0064] The BDE module proposed in this invention is centered on a lightweight viewport prediction confidence estimator. This estimator takes the prediction duration and the acceleration variance of the user's positional motion as input, and outputs a confidence score between 0 and 1 in real time through a pre-trained neural network, accurately quantifying the reliability of the current prediction. This confidence score is flexibly integrated into the ABR algorithm to achieve intelligent decision-making based on risk awareness: for rule-based algorithms, the confidence score serves as a decision threshold; for optimization-based algorithms, it modulates the weights of future rewards; and for learning-based algorithms, it serves as a key state feature. This mechanism enables the ABR algorithm to dynamically adjust its strategy—when the prediction confidence is high, it actively requests high-bitrate content for the predicted viewport to improve quality; when the confidence is low, it switches to a conservative strategy, reserving bandwidth to avoid waste. Ultimately, this module significantly optimizes bandwidth allocation efficiency, effectively improving visible image quality and reducing stuttering in complex network environments.

[0065] During the online inference phase, the VPE and BDE modules work together: the system acquires the user's trajectory in real time, and the VPE module first dynamically predicts the future viewport; then, the BDE module evaluates the confidence level of the prediction and inputs it into the ABR controller; based on the confidence level, the ABR dynamically decides to allocate a high or low bitrate to the predicted viewport, while non-viewport areas always use the lowest bitrate. This process is executed iteratively with each video block request, continuously optimizing bandwidth utilization and viewing experience.

[0066] like Figure 2 As shown, this invention constructs the dynamic selection process of the historical window as a closed-loop decision-making system based on reinforcement learning. The system first uses the K most recent user spatial position movement samples as state input, reflecting the user's movement trend, speed changes, and positional inertia information over a short period. The historical window selector, acting as an agent, parses this state through its internal policy network and outputs an action representing the historical window's capture position, i.e., the starting coordinate index of the historical window. This action is fed into the environment module, which is composed of a viewport prediction model, to generate a viewport prediction result based on this window and calculate the viewport hit rate (VHR).

[0067] The environment module returns the viewport hit rate to the agent as an immediate reward, reflecting the effectiveness of the window selection. During reinforcement learning training, the policy network is updated and optimized based on this reward, enabling the agent to gradually learn to automatically select the trajectory segment that best represents the current dynamic characteristics under different user movement patterns (such as constant speed movement, sharp turns, etc.). The overall training goal is to maximize the cumulative viewport hit rate throughout the sequence, allowing the window selector to stably and accurately capture the inertial patterns of user movement, providing reliable input for subsequent viewport prediction enhancement.

[0068] like Figure 3 As shown, the viewport prediction confidence estimator constructed in this invention employs a two-layer fully connected neural network structure, using multi-dimensional motion features within the historical trajectory window as input to perform regression prediction of the future viewport hit rate. The input features include the three-dimensional acceleration components {a} of the user at h consecutive sampling points prior to the prediction time. x,t-h ,a y,t-h ,a z,t-h The `}` parameter characterizes the intensity of user positional changes in the left-right, up-down, and forward-backward directions, while the prediction time span `p` describes the time range required for prediction. These features are mapped by a neural network during the forward propagation phase, outputting the predicted viewport hit rate. The training phase will predict the value VHR with Real Label t The mean squared error (MSE) is calculated, and the network parameters are continuously optimized through backpropagation, enabling the evaluator to accurately characterize the reliability of viewport prediction results under different motion modes. Finally, the model can output confidence scores in real time during the inference phase, providing a reliable risk perception basis for subsequent adaptive transmission decisions.

[0069] Figure 4 The diagram shows the interface of the volumetric video streaming system of the present invention during playback, including the current playback progress, rendering status, and the operation of core modules such as viewport prediction and bitrate decision, which can be used to reflect the overall working status of the system in the actual playback process. Figure 5The system provides network monitoring metrics during playback, including key parameters such as instantaneous bandwidth, bandwidth change trend, buffer occupancy, and the currently used quality level. Instantaneous bandwidth reflects the actual network throughput available to the client at any given moment and is a direct constraint on bitrate decisions. The bandwidth change trend depicts the direction of bandwidth increase or decrease over a short period, used to predict network stability and help avoid frequent bitrate fluctuations. Buffer occupancy represents the amount of video data currently cached by the client, reflecting the safety margin during playback and preventing stuttering caused by network fluctuations. The quality level corresponds to the encoding quality of the currently playing video and is used to visually demonstrate the decision results of the adaptive bitrate algorithm. Through real-time visual monitoring of these metrics, the system can dynamically assess the network operating status, providing necessary environmental awareness information for the viewport prediction enhancement module and the bitrate decision enhancement module. Figure 6 The system showcases a configurable user interface, allowing users to select different types of Adaptive Bit Rate (ABR) algorithms based on their needs and specify the viewport prediction algorithm used by the system. This ensures the system maintains high flexibility and adjustability across various network environments and usage scenarios.

[0070] 4. Effect Comparison

[0071] like Figure 7 As shown: This invention compares the constructed volumetric video transmission system with the following five existing advanced volumetric video transmission algorithms. The comparison is between algorithms using the collaborative optimization method described in this invention and basic algorithms that do not use this method. The comparison metric is average user experience quality. The compared algorithms include:

[0072] (1) Buffer-based (BB) (Reference: Te-Yuan Huang et al., A buffer-based approach to rate adaptation: Evidence from a large video streaming service, in Proceedings of the 2014 ACM conference on SIGCOMM, 2014, pp. 187–198.): This is an adaptive bit rate algorithm that makes quality decisions by sensing buffer occupancy and is widely deployed in real video streams.

[0073] (2) Rate-based (RB) (Reference: B. Han, Y. Liu, and F. Qian, “ViVo: Visibility-aware mobile volumetric video streaming,” in Proc. 26th Annu. Int. Conf. Mobile Comput. Netw., 2020, pp. 1–13.): The matching bitrate is directly selected by observing the current network throughput (or bandwidth) to minimize stuttering and bitrate fluctuations.

[0074] (3) Reinforcement Learning-Based (RLB) (Reference: Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural Adaptive Video Streaming with Pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM'17). Association for Computing Machinery, New York, NY, USA, 197–210): The ABR algorithm is modeled as a Markov decision process (MDP), where state = {buffer level, throughput, historical bitrate and other influencing factors}, and action = bitrate selection.

[0075] (4) Two-Tier (Reference: Sun, Liyang, et al. "A two-tier system for on-demand streaming of 360degree video over dynamic networks." in IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9.1(2019):43-57.): This is a popular scheme in video streaming. It first downloads the base layer of all blocks for the next few blocks to efficiently fill the buffer. When a new viewport is detected, it completely discards the previously downloaded content and acquires new, higher-quality content for the necessary upgrade.

[0076] (5) Robust Model-Predictive Control (RMPC) (Reference: J.Li, C.Zhang, Z.Liu, W.Sun, and Q.Li, “Joint communication and computational resource allocation for QoE-driven point cloud video streaming,” in Proc.IEEE Int.Conf.Commun.(ICC), 2020, pp.1–6): It makes real-time decisions on bitrate by predicting network conditions over a period of time and adjusting the bitrate based on optimization objectives (such as bitrate smoothness, stuttering minimization, etc.).

[0077] This invention was tested in various real-world network environments, including a 5G city center (average 36.55 Mbps), a WiFi office (25.49 Mbps), a WiFi shopping mall (20.08 Mbps), and a 5G suburb (14.90 Mbps). The tests used "redandblack" volumetric video from the 8iVFB dataset, which was divided into 40 segments of 0.5 seconds each, each segment spatially divided into 36 blocks, and encoded at three bitrates: {5, 20, 50} Mbps.

[0078] Experimental results show that this invention significantly improves QoE compared to existing mainstream ABR algorithms, demonstrating a significant advantage. The improvement is most pronounced in low-bandwidth, demanding environments such as 5G suburban areas. This proves that this invention improves prediction accuracy through the VPE module and achieves risk-aware rate adaptation using the BDE module, efficiently utilizing limited bandwidth and maximizing visible image quality while ensuring smoothness, thus significantly optimizing the overall experience.

[0079] This invention provides a viewport prediction and bitrate decision optimization method for volumetric video streams. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A viewport prediction and bitrate decision optimization method for volumetric video streams, characterized in that, Includes the following steps: Step 1, offline training, includes the following steps: Step 1-1: Based on the focusing and following characteristics of the user's gaze movement and the inertial characteristics of the positional movement, remove the prediction of the directional degree of freedom; Step 1-2, Training the historical window selector: Collect user trajectory sequences as training sets, use the near-end strategy to optimize the PPO algorithm to train the agent, and associate the reward function with the viewport hit rate (VHR) so that the agent can learn to dynamically select the historical window range that best reflects the motion inertia of the current position based on the motion trajectory displayed in the user trajectory. Steps 1-3, train the viewport prediction confidence evaluator: construct a two-layer fully connected neural network, take the historical position acceleration variance and prediction time span as input, and the corresponding viewport hit rate as label, and train it through supervised learning so that the viewport prediction confidence evaluator can evaluate the reliability of the prediction based on real-time interaction features. Step 2, online reasoning, includes the following steps: Step 2-1: Deploy the Viewport Prediction Enhancement (VPE) module for real-time prediction. Before each request for a new video block, the VPE module obtains the latest user trajectory from the client buffer and calls the offline trained historical window selector to analyze the current trajectory status, output the optimal window capture index, thereby filtering out the historical trajectory segments used for this viewport prediction, and performing future viewport prediction based on the historical trajectory segments. Step 2-2: Deploy the Bitrate Decision Enhancement (BDE) module for adaptive decision-making.

2. The method according to claim 1, characterized in that, In step 1-1, the historical window selector is a reinforcement learning selector built and trained in the offline training phase. The historical window selector takes the recent K position motion samples obtained by the client at a fixed sampling frequency as the state input, where recent refers to a number of position points sampled within a short time window corresponding to the prediction time span. The historical window selector trains an agent using the Proximal Policy Optimization (PPO) algorithm. The agent refers to the policy network model optimized by the PPO algorithm. The agent's output is the historical window cutoff point index, and the action space is an integer from 1 to K, used to indicate the starting position of the trajectory window. The historical window cutoff point index is used to determine the dynamic window range and remove noise samples that are inconsistent with the current motion trend. The reward function is directly associated with the viewport hit rate calculated based on the dynamic trajectory window selected by the historical window selector. The viewport hit rate is obtained by calculating the overlap between the spatial location prediction model, which predicts the user's future location, and the actual viewport area. The agent learns and captures the inertial patterns of user positional motion by maximizing the cumulative viewport hit rate.

3. The method according to claim 2, characterized in that, In step 1-1, the formula for calculating the viewport hit rate is: Among them, T pred T represents the set of viewport coverage tiles predicted by the client based on historical trajectories. actual This represents the set of visible segments actually viewed by the user at the future target time point corresponding to the prediction; |·| represents the number of segments in the set; the viewport hit rate H is used to quantify the overlap accuracy between the predicted viewport and the actual viewport.

4. The method according to claim 3, characterized in that, In steps 1-2, the window selection problem is constructed as a sequential decision task to capture the inertial characteristics of user positional motion in a six-degrees-of-freedom experience; The state space of the agent consists of the K most recent historical position samples, and the action space is defined as a discrete window truncation index a∈[1,K], which is used to indicate the starting position of the trajectory window; The agent dynamically determines the length of the trajectory window from the latest sample back to the a-th sample by outputting action 'a'. During the training phase, the viewport hit rate is used as the immediate reward, and the Proximal Policy Optimization (PPO) algorithm is used to update the policy network to maximize the cumulative reward. Through training, the agent can learn to identify and filter historical trajectory segments that are most consistent with the current motion trend.

5. The method according to claim 4, characterized in that, Steps 1-3 include: extracting historical viewport prediction cases from real user motion trajectories as training samples. The input features of each training sample include the variance of spatial degrees of freedom acceleration within the historical window and the user's acceleration value 'a' in the left-right direction in three-dimensional space. x The acceleration value a in the vertical direction y The acceleration value a in the forward and backward directions z And the predicted time span p; a x a y a z Used to quantify the intensity of a user's location movement; The output label is the viewport hit rate corresponding to the predicted viewport; Based on the training samples, a two-layer fully connected neural network is constructed as a viewport prediction confidence evaluator. Supervised learning is used to optimize the model with mean squared error (MSE) as the loss function. The training sample dataset is divided into a training set and a validation set according to a preset ratio. The validation set is derived from the reserved portion of real user trajectory data and is used to monitor the model's generalization performance. During training, the fully connected neural network employs forward and backward propagation mechanisms: forward propagation generates the prediction confidence of the viewport hit rate, and backward propagation updates the model weights based on the error between the predicted value and the viewport hit rate, iterating continuously until the mean squared error (MSE) metric on the validation set converges. After training, the viewport prediction confidence evaluator can quantify the reliability of viewport prediction based on the real-time interaction mode, providing a risk perception basis for the adaptive bitrate (ABR) algorithm.

6. The method according to claim 5, characterized in that, In steps 1-3, the two fully connected neural networks consist of one layer with 128 neurons and the other with 64 neurons, using ReLU as the activation function; ultimately, the two fully connected neural networks output a scalar between 0 and 1.

7. The method according to claim 6, characterized in that, In step 2-1, during online inference, the viewport prediction enhancement module is first deployed. The viewport prediction enhancement module includes a historical window selector and a spatial location prediction model obtained during the offline training phase, which are used to perform real-time viewport prediction during playback. When a new video block needs to be requested, the viewport prediction enhancement module obtains the latest user trajectory from the client buffer and calls the historical window selector trained by the near-end policy optimization PPO algorithm in steps 1-2. The historical window selector contains an agent trained by the near-end policy optimization PPO algorithm. The policy network corresponding to the agent is a policy function optimized by the near-end policy optimization PPO algorithm. The agent outputs the optimal window index in real time according to the current trajectory state input, thereby dynamically selecting the relevant trajectory segment that is most consistent with the current motion trend. The relevant trajectory segments are fed into a spatial location prediction model to predict the future user spatial coordinates; Finally, the client runs on the PICO4 head-mounted display device. Combining the device's field of view parameters with the three-dimensional geometric coordinate system of the virtual scene, it calculates the spatial coverage of the predicted viewport: Let the predicted head orientation be determined by Euler angles (θ, φ) or the equivalent unit direction vector d ∈ R. 3 Let R represent the real number space; the horizontal field of view of PICO 4 is FOV. h The vertical field of view is FOV. v The predicted viewport can be expressed by the following two inequalities: Where u and v are the horizontal and vertical vectors of the center point of any tile in the scene relative to the user's position, respectively. All tiles that satisfy the two inequalities constitute the set T of visible tiles for the predicted viewport. pred .

8. The method according to claim 7, characterized in that, In step 2-2, the bitrate decision enhancement module is used to dynamically adjust the bitrate selection based on the reliability of viewport prediction; the client calculates the corresponding trajectory features based on the historical trajectory segments for viewport prediction determined in step 2-1, and calls the viewport prediction confidence evaluator trained in step 1-3 to generate a confidence value representing the reliability of this viewport prediction; the confidence value is integrated into the adaptive bitrate algorithm ABR to dynamically determine the video block bitrate level corresponding to the predicted viewport, while the lowest bitrate is uniformly used for transmission of content outside the predicted viewport.

9. The method according to claim 8, characterized in that, In step 2-2, the confidence level is deeply integrated into the adaptive bit rate (ABR) algorithm through three core mechanisms to achieve adaptive decision-making for risk perception: for rule-based algorithms, the confidence level serves as a threshold condition for direct intervention; for optimization control algorithms, the confidence level serves as the expected quality gain of the future viewport weighted by the modulation factor; and for learning algorithms, the confidence level serves as a key state feature input to the policy network, driving the agent to autonomously learn to adopt aggressive strategies when the confidence level is high and to switch to conservative decisions when the confidence level is low.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 9.

Images