An immersive video perceptual transmission method based on implicit integrated viewport prediction and representation learning code rate decision

Abstract

An immersive video perception transmission method based on implicit ensemble viewport prediction and representation learning-based bitrate decision-making is proposed, relating to the fields of immersive video transmission and intelligent content-aware coding optimization. To address the problems of insufficient generalization in viewport prediction, non-adaptive user preference modeling, and poor stability under network disturbances in existing technologies, this method establishes an implicit ensemble viewport prediction model with a multi-input / output structure by collecting user head motion trajectory and field-of-view heatmap data; it constructs a bitrate decision engine with mutual information constraints by combining representation learning and reinforcement learning, and achieves tile-level dynamic bitrate allocation based on prediction results and user preferences; it introduces an adversarial training mechanism against bandwidth jitter and packet loss disturbances to enhance robustness; and it adopts a collaborative deployment of cloud, edge, and terminal to achieve end-to-end adaptive optimization. This method is suitable for high-bandwidth, low-latency scenarios, significantly saving bandwidth and improving user experience continuity while ensuring the clarity of the core view area.

Description

Technical Field

[0001] This involves the fields of immersive video transmission and intelligent content-aware coding optimization, specifically immersive video-aware transmission based on implicit integrated viewport prediction and representation learning bitrate decision-making. Background Technology

[0002] With the widespread adoption of 5G / gigabit broadband and the improvement of terminal computing power, cloud gaming and immersive video such as VR / AR are entering a service phase characterized by high resolution, low latency, and strong interactivity. These services are sensitive to end-to-end latency, bandwidth usage, and jitter: cloud gaming typically requires sustained medium-to-high bitrates and stable inter-frame latency, while VR scenarios face even stricter constraints on bitrate and latency at 4K resolution and high refresh rates per eye. Traditional transmission methods, which use uniform encoding parameters superimposed with adaptive bitrates, primarily switch between several fixed levels based on bandwidth feedback. While this can maintain usable image quality in general scenarios, it easily leads to significant image quality fluctuations, increased stuttering, and inconsistencies in the user's subjective experience under complex conditions involving high-speed motion, frequent changes in viewpoint, and network disturbances.

[0003] To improve transmission efficiency and subjective quality, content awareness has become an important direction. Existing solutions, based on saliency detection or object recognition, divide the image into regions of interest and regions of non-interest, increasing encoding resources for regions of interest and reducing resolution or bitrate for regions of non-interest. In VR panoramic videos, tile-based and field-of-view priority strategies are introduced, encoding tiles within the user's field of view at a high bitrate and gradually reducing the bitrate outside the field of view according to a pyramid strategy based on distance from the center. These methods can achieve certain bandwidth savings and maintain subjective quality under experimental conditions, but the regions of interest are often determined by empirical thresholds or static rules, making it difficult to adapt to dynamic user interests and cross-content scene migration, and prone to misjudgment when scene changes or sudden changes in interactive behavior.

[0004] For field-of-view prediction, existing research has evolved from rule-based head motion extrapolation to using recurrent neural networks, attention mechanisms, and Transformer structures to learn spatiotemporal dependencies and predict viewport positions within short future time windows. Single-model prediction performs well on homogeneous data, but when applied to different user groups, video types, and interaction modes, bias accumulation and overfitting often occur, resulting in insufficient generalization ability across users and content. While explicit ensembles can alleviate this, they significantly increase computational and deployment costs, making real-time inference at the edge difficult to implement. Furthermore, most methods only output a single viewport point or region, failing to adequately characterize uncertainty and affecting the stability and robustness of downstream tile bitrate allocation.

[0005] Regarding adaptive bitrate, traditional ABR (Adaptive Rate Response) makes decisions based on network metrics such as throughput, buffering, and packet loss, along with fixed weights, making it difficult to reflect users' differentiated preferences for image quality, fluctuations, and stuttering. Recent methods have introduced reinforcement learning to learn policies in complex network environments, but their reward functions are mostly based on linear weighting of objective metrics or simple scoring, failing to explicitly model the retention and expression of user preference information in the state-action mapping. This makes them prone to policy drift or catastrophic forgetting when preferences change or when no preference combinations are seen, leading to a mismatch between personalized experience and resource allocation.

[0006] At the systems engineering and deployment level, cloud-edge-device collaboration is the mainstream path to reduce latency and improve concurrency and elasticity. Existing practices typically implement viewport prediction, bitrate decision-making, and QoE feedback links separately: lightweight detection is performed on the device side, edge nodes are scheduled using a general ABR, and models and strategies are trained offline in the cloud. Due to the lack of closed loops and common goal optimization between links, the problem of inconsistent training distribution and online distribution is prominent; when there are bandwidth jitters and random packet loss in the network, the policy adjustment at the link end lags, erroneously triggering bitrate oscillations and buffer oscillations, thereby amplifying subjective quality fluctuations. Faced with multi-user concurrency and content diversity, the trade-off between robustness, scalability, and real-time performance in existing systems is still not ideal.

[0007] From the perspective of algorithmic robustness, immersive video links are affected by both content perturbations and network perturbations. Existing methods are mostly evaluated under natural noise or simple packet loss models, lacking a systematic adversarial robustness training and evaluation framework. This makes it difficult to guarantee stable viewport predictions and robust bitrate sequences under conditions of high packet loss, sudden congestion, or abnormal trajectory inputs. The lack of targeted constraints on perturbation-sensitive links causes end-to-end QoE to degrade significantly under extreme network conditions.

[0008] In terms of data and evaluation, existing work often uses single or similarly distributed datasets, which do not adequately cover user head movement trajectories, scene motion patterns, and interactive behaviors. Evaluation metrics are also mostly focused on single-point metrics, such as the intersection-union ratio of viewport prediction or the average throughput utilization of bitrate strategies. There is a lack of a unified indicator system and process constraints for the comprehensive stability and consistency across preferences, scenes, and network conditions, which leads to discrepancies between offline metrics and online experiences.

[0009] In summary, existing technologies suffer from several common shortcomings in key areas: viewport prediction relies heavily on single models or high-overhead explicit ensembles, resulting in insufficient generalization capabilities and uncertainty characterization across users and content; bitrate selection fails to explicitly preserve and utilize user QoE preference information through representation learning, making stable generalization difficult under preference switching and unseen preference configurations; there is a lack of adversarial robust training and online robustness constraints for network perturbations and abnormal inputs, making bitrate oscillations and experience degradation prone to occur under jitter and packet loss environments; the cloud-edge-device link lacks closed-loop optimization driven by common goals and lightweight and efficient edge inference design, making it difficult to achieve a balance between real-time performance, concurrency, and accuracy; and attention zone partitioning and resource allocation still heavily rely on experience or static rules, making it difficult to adapt to dynamic user interests and complex scene changes.

[0010] In summary, existing technologies suffer from several shortcomings, including insufficient generalization of viewport prediction and uncertainty modeling, lack of learnable representations and adaptive decision-making mechanisms for QoE preferences, insufficient robustness to network disturbances and abnormal inputs, lack of closed-loop and low-latency implementation for cloud-edge-device collaboration, and insufficient adaptability of attention area and bitrate allocation to dynamic interests and complex scenarios. Summary of the Invention

[0011] To address the shortcomings of existing technologies, such as insufficient generalization of viewport prediction and uncertainty modeling, lack of learnable representation and adaptive decision-making mechanisms for QoE preferences, insufficient robustness to network disturbances and abnormal inputs, lack of closed-loop and low-latency implementation for cloud-edge-device collaboration, and insufficient adaptability of interest area and bitrate allocation to dynamic interests and complex scenarios, the technical solution provided by this invention is as follows:

[0012] An immersive video-aware transmission method based on implicit ensemble viewport prediction and representation learning bitrate decision-making includes:

[0013] The steps include: acquiring immersive video data containing user head motion trajectories and field-of-view heatmap labels; cleaning, synchronizing, and standardizing the original trajectories; mapping the trajectories to equidistant bar coordinates and slicing them according to time windows; and generating a multimodal feature sample dataset.

[0014] The steps involve establishing a viewport prediction model with a multi-input-output structure based on a sample dataset, embedding the user's historical viewpoint sequence into a vectorized form, extracting spatiotemporal features using a multi-head attention mechanism, compressing the high-dimensional sequence through a feature distillation module, and fusing the outputs of multiple sub-models using an implicit ensemble strategy to generate future viewport prediction results.

[0015] Based on the viewport prediction results, user experience preference weights, and network state information, the reinforcement learning decision engine is input. A reward function with mutual information constraints is constructed to maintain the transmission of preference features. A bitrate allocation table for video tiles is dynamically generated. High bitrate is allocated to tiles within the viewport, and the bitrate of tiles outside the viewport is gradually reduced according to the distance from the center. The real-time transmission strategy parameters are output.

[0016] The steps include injecting bandwidth jitter and packet loss perturbation models during the training phase, optimizing the decision network by combining gradient-direction-based adversarial example generation methods, updating the policy parameter set, and outputting the policy model weights.

[0017] In a collaborative environment consisting of terminals, edge nodes, and the cloud, a viewport prediction model with a multi-input-output structure is deployed. The terminal performs lightweight viewport prediction in real time and uploads the results. Edge nodes make bitrate decisions based on the prediction and network status and issue transmission parameters. The cloud updates the strategy periodically and synchronizes it to the edge nodes. The end-to-end adaptive video transmission is maintained through closed-loop feedback.

[0018] Furthermore, a preferred implementation method is provided, in which the original head motion trajectory is projected into equidistant cylindrical coordinates during data construction and feature processing, and then one-dimensional convolution and max pooling feature compression are performed, and sample fragments are generated through a time sliding window.

[0019] Furthermore, a preferred implementation is provided, wherein the viewport prediction model adopts a two-layer structure consisting of an encoder and a decoder. The encoder is used to extract global spatiotemporal dependency features of the user's viewpoint sequence, and the decoder is used to generate coordinate prediction results of the future viewport range. Cross-frame feature fusion is achieved through a multi-head attention mechanism and combined with an implicit integration strategy to output the global predicted viewport.

[0020] Furthermore, a preferred implementation method is provided, in which the reinforcement learning decision engine introduces a mutual information-driven reward function during the training phase, using user preference weights, bandwidth fluctuations and buffer states as input variables, to achieve dynamic transfer between different preferences while maintaining a balance between image quality and playback smoothness.

[0021] Furthermore, a preferred implementation method is provided, in which a bitrate allocation strategy is used to manage video tiles in layers. Tiles within the viewport are transmitted at a high bitrate to ensure center sharpness, while tiles outside the viewport are divided into multiple ring zones according to their distance from the center of the viewport, each using a decreasing bitrate level.

[0022] Furthermore, a preferred implementation method is provided, which constructs a random perturbation environment of bandwidth jitter and packet loss during the adversarial robust training process, and adopts a perturbation sample generation method based on gradient direction.

[0023] Based on the same inventive concept, this invention also provides an immersive video sensing transmission device based on implicit integrated viewport prediction and representation learning bitrate decision, comprising:

[0024] This module acquires immersive video data containing user head motion trajectories and field-of-view heatmap labels, cleans, synchronizes, and standardizes the original trajectories, maps the trajectories to equidistant bar coordinates and slices them according to time windows, and generates a multimodal feature sample dataset.

[0025] A viewport prediction model with a multi-input-output structure is established based on a sample dataset. The user's historical viewpoint sequence is vectorized and embedded, spatiotemporal features are extracted using a multi-head attention mechanism, high-dimensional sequences are compressed through a feature distillation module, and the outputs of multiple sub-models are fused using an implicit ensemble strategy to generate the future viewport prediction results.

[0026] Based on viewport prediction results, user experience preference weights, and network state information, a reinforcement learning decision engine is input. A reward function with mutual information constraints is constructed to maintain the transmission of preference features. A bitrate allocation table for video tiles is dynamically generated. High bitrate is allocated to tiles within the viewport, and the bitrate of tiles outside the viewport is gradually reduced according to the distance from the center. A module that outputs real-time transmission strategy parameters is also provided.

[0027] A module that injects bandwidth jitter and packet loss perturbation models during the training phase, and combines gradient-direction-based adversarial example generation methods to optimize the decision network, update the policy parameter set, and output the policy model weights.

[0028] A multi-input-output viewport prediction model is deployed in a collaborative environment consisting of terminals, edge nodes, and the cloud. The terminal performs lightweight viewport prediction in real time and uploads the results. Edge nodes make bitrate decisions based on the prediction and network status and issue transmission parameters. The cloud updates the strategy periodically and synchronizes it to the edge nodes. This module maintains end-to-end adaptive video transmission through closed-loop feedback.

[0029] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program, wherein when the computer program is read by a computer, the computer executes the method described thereon.

[0030] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, wherein when the processor reads a computer program stored in the storage medium, the computer executes the method described thereon.

[0031] Based on the same inventive concept, the present invention also provides a computer program product, which, when executed, implements the method described.

[0032] Compared with the prior art, the advantages of the technical solution provided by the present invention are as follows:

[0033] The implicitly integrated MTIO-Transformer viewport prediction architecture enhances generalization capabilities across users and content. This is achieved by setting up multiple input / output heads on a shared backbone to form implicit sub-model ensembles. This eliminates the single-model bias accumulation problem without significantly increasing computational cost, improving prediction accuracy for unknown viewing modes by approximately three to nine percentage points and reducing prediction bias compared to the recurrent network baseline by about four percentage points. This results in more stable viewport hit rates and continuity in downstream bitrate allocation.

[0034] The feature distillation and sequence compression module significantly reduces edge-side inference overhead. This is achieved by using one-dimensional convolution and max pooling to reduce the dimensionality of temporal features and compressing redundant sequence lengths by approximately 40%. Combined with TensorRT inference optimization, the model's memory footprint is approximately 28 megabytes, and inference latency is less than 36 milliseconds. Compared to explicit integration or deep Transformer solutions, this is easier to implement at edge nodes and maintains higher viewport prediction accuracy with the same latency budget.

[0035] Multi-head attention and long-term dependency modeling mitigate short-term jitter in viewport trajectories. This approach involves explicitly learning long-range spatiotemporal dependencies using an encoder-decoder structure and context-aligning with rapid head turns and dramatic scene changes. This reduces the frequency and magnitude of temporal jumps in prediction points. Compared to rule-based extrapolation or single-recurrent networks, this significantly reduces tile bitrate oscillations caused by prediction jitter and smooths the buffer curve.

[0036] Reinforcement learning bitrate decision-making based on representation learning achieves adaptive matching of QoE preferences. This approach introduces a mutual information term into the reward function to maximize information retention in the mapping from user preferences to policy actions and enhances preference feature propagation through residual connections. This results in a QoE gain of approximately 3 to 15 percentage points across eight preference configurations and improves training convergence speed by about 35%. Compared to traditional ABR or RL policies without modeled preferences, it maintains stable performance under preference switching and unseen preference combinations and avoids catastrophic forgetting.

[0037] The hierarchical tiled bitrate allocation within and outside the viewport significantly saves bandwidth while maintaining subjective quality. This is achieved by prioritizing high bitrate allocation to tiles within the viewport and employing a pyramid-shaped bitrate reduction strategy based on distance from the center for tiles outside the viewport. Compared to uniform coding or regional strategies based solely on static saliency, this approach achieves at least 30% bandwidth savings while maintaining sharpness in high-speed motion and complex texture scenes, even with more accurate viewport hits.

[0038] Adversarial robust training significantly enhances network stability under perturbations. This is achieved by injecting a stochastic model of bandwidth jitter and packet loss into the training environment and combining it with gradient-based adversarial example generation to cover anomalous trajectories and extreme network states. This maintains a robust output with QoE fluctuations not exceeding 5% even in test scenarios with up to 20% packet loss. Compared to strategies trained solely on natural perturbations, this effectively suppresses bitrate oscillations and rebuffering caused by sudden congestion.

[0039] The closed-loop architecture of cloud-edge-device collaboration reduces end-to-end latency and minimizes distribution drift. This is achieved by moving lightweight viewport prediction to the terminal and edge, residing bitrate decisions at edge nodes, and placing adversarial training and policy iteration in the cloud to form a feedback loop. Online distribution is fed back to the training end via edge sampling to narrow the offline-online gap and control end-to-end latency to the order of 40 milliseconds. Compared to centralized decision-making in the cloud, this reduces control loop lag and improves scalability under multi-user concurrency.

[0040] The QoE identifier and reward reconstruction deliver a smoother session experience. This is achieved by using mutual information estimation to guide the trade-off between image quality fluctuations and stuttering in the reward calculation, and by using a logarithmic penalty for prediction errors to suppress over-excitement. This significantly reduces the frequency and magnitude of bitrate switching and decreases the incidence of short-term rebuffering in environments with bandwidth jitter. Compared to heuristic rewards that linearly weight throughput and buffering, it achieves higher subjective coherence at the same average bitrate.

[0041] The dataset construction and strong-constraint evaluation improve offline-online consistency. This approach involves incorporating user head motion trajectory data from various sources, along with segments containing extreme network fluctuations, into a stratified sampling process across training, validation, and test sets. Joint performance metrics such as viewport overlap, bandwidth savings, and end-to-end latency are used as optimization objectives to constrain performance. This reduces the problem of overly similar and homogeneous data at the source and makes offline metrics more consistent with the online experience. Compared to optimization of a single dataset or single metric, this approach is more reusable across content, users, and networks.

[0042] Visual monitoring and policy adjustment, along with engineered deployment, promote maintainability and operational optimization. This approach uses WebGL heatmaps and QoE dashboards to display viewport hits and bandwidth usage in real time, and issues thresholds and constraints via gRPC to enable online fine-tuning of policies. This allows for rapid rollback and refined operations during peak or abnormal periods. Compared to black-box policies, this approach can maintain a balance between service quality and cost with small parameter adjustments without retraining.

[0043] It is suitable for optimizing intelligent content-aware video transmission in high-bandwidth, low-latency scenarios such as cloud gaming and immersive VR videos. Attached Figure Description

[0044] Figure 1To construct the overall architecture of an immersive video transmission system for MTIO-Transformer viewport prediction and representation learning bitrate decision-making;

[0045] Figure 2 Viewport prediction model diagram;

[0046] Figure 3 This is a diagram of the bitrate decision model;

[0047] Figure 4 A console front-end framework;

[0048] Figure 5 This is a system workflow diagram. Detailed Implementation

[0049] To make the advantages and benefits of the technical solution provided by the present invention clearer, the technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings, specifically:

[0050] Implementation Method 1: This implementation method provides an immersive video perception transmission method based on implicit integrated viewport prediction and representation learning bitrate decision, including:

[0051] The steps include: acquiring immersive video data containing user head motion trajectories and field-of-view heatmap labels; cleaning, synchronizing, and standardizing the original trajectories; mapping the trajectories to equidistant bar coordinates and slicing them according to time windows; and generating a multimodal feature sample dataset.

[0052] The steps involve establishing a viewport prediction model with a multi-input-output structure based on a sample dataset, embedding the user's historical viewpoint sequence into a vectorized form, extracting spatiotemporal features using a multi-head attention mechanism, compressing the high-dimensional sequence through a feature distillation module, and fusing the outputs of multiple sub-models using an implicit ensemble strategy to generate future viewport prediction results.

[0053] Based on the viewport prediction results, user experience preference weights, and network state information, the reinforcement learning decision engine is input. A reward function with mutual information constraints is constructed to maintain the transmission of preference features. A bitrate allocation table for video tiles is dynamically generated. High bitrate is allocated to tiles within the viewport, and the bitrate of tiles outside the viewport is gradually reduced according to the distance from the center. The real-time transmission strategy parameters are output.

[0054] The steps include injecting bandwidth jitter and packet loss perturbation models during the training phase, optimizing the decision network by combining gradient-direction-based adversarial example generation methods, updating the policy parameter set, and outputting the policy model weights.

[0055] In a collaborative environment consisting of terminals, edge nodes, and the cloud, a viewport prediction model with a multi-input-output structure is deployed. The terminal performs lightweight viewport prediction in real time and uploads the results. Edge nodes make bitrate decisions based on the prediction and network status and issue transmission parameters. The cloud updates the strategy periodically and synchronizes it to the edge nodes. The end-to-end adaptive video transmission is maintained through closed-loop feedback.

[0056] During data construction and feature processing, the original head motion trajectory is projected into equidistant cylindrical coordinates and then subjected to one-dimensional convolution and max pooling feature compression. Sample fragments are generated through a time sliding window.

[0057] The viewport prediction model adopts a two-layer structure consisting of an encoder and a decoder. The encoder is used to extract global spatiotemporal dependency features of the user's viewpoint sequence, and the decoder is used to generate coordinate prediction results of the future viewport range. Cross-frame feature fusion is achieved through a multi-head attention mechanism and combined with an implicit ensemble strategy to output the global predicted viewport.

[0058] The reinforcement learning decision engine introduces a mutual information-driven reward function during the training phase, using user preference weights, bandwidth fluctuations, and buffer states as input variables to achieve dynamic transfer between different preferences while maintaining a balance between image quality and playback smoothness.

[0059] The bitrate allocation strategy manages video tiles in layers. Tiles within the viewport are transmitted at a high bitrate to ensure center sharpness, while tiles outside the viewport are divided into multiple ring zones based on their distance from the center of the viewport, each using a decreasing bitrate level.

[0060] During adversarial robust training, a random perturbation environment with bandwidth jitter and packet loss is constructed, and a perturbation sample generation method based on gradient direction is adopted.

[0061] An immersive video sensing transmission device based on implicit integrated viewport prediction and representation learning bitrate decision is also provided, comprising:

[0062] This module acquires immersive video data containing user head motion trajectories and field-of-view heatmap labels, cleans, synchronizes, and standardizes the original trajectories, maps the trajectories to equidistant bar coordinates and slices them according to time windows, and generates a multimodal feature sample dataset.

[0063] A viewport prediction model with a multi-input-output structure is established based on a sample dataset. The user's historical viewpoint sequence is vectorized and embedded, spatiotemporal features are extracted using a multi-head attention mechanism, high-dimensional sequences are compressed through a feature distillation module, and the outputs of multiple sub-models are fused using an implicit ensemble strategy to generate the future viewport prediction results.

[0064] Based on viewport prediction results, user experience preference weights, and network state information, a reinforcement learning decision engine is input. A reward function with mutual information constraints is constructed to maintain the transmission of preference features. A bitrate allocation table for video tiles is dynamically generated. High bitrate is allocated to tiles within the viewport, and the bitrate of tiles outside the viewport is gradually reduced according to the distance from the center. A module that outputs real-time transmission strategy parameters is also provided.

[0065] A module that injects bandwidth jitter and packet loss perturbation models during the training phase, and combines gradient-direction-based adversarial example generation methods to optimize the decision network, update the policy parameter set, and output the policy model weights.

[0066] A multi-input-output viewport prediction model is deployed in a collaborative environment consisting of terminals, edge nodes, and the cloud. The terminal performs lightweight viewport prediction in real time and uploads the results. Edge nodes make bitrate decisions based on the prediction and network status and issue transmission parameters. The cloud updates the strategy periodically and synchronizes it to the edge nodes. This module maintains end-to-end adaptive video transmission through closed-loop feedback.

[0067] A computer storage medium is also provided for storing a computer program, which, when read by the computer, executes the method.

[0068] A computer is also provided, including a processor and a storage medium, wherein the computer executes the method when the processor reads a computer program stored in the storage medium.

[0069] A computer program product is also provided, which, when executed, implements the method described.

[0070] Implementation Method Two: This implementation method is a further detailed description of the technical solution provided in Implementation Method One, specifically:

[0071] An immersive video perception transmission method based on implicit integration of viewport prediction and representation learning for bitrate decision-making aims to address the problems of insufficient viewport prediction generalization ability, difficulty in adapting bitrate allocation strategies to dynamic user preferences, and poor transmission stability under network disturbances in existing immersive video systems. The entire method comprises five consecutive steps: data construction and feature extraction, viewport prediction modeling, bitrate decision generation, adversarial robustness optimization, and cloud-edge-device collaborative deployment. These steps are logically interconnected, with the output of one step directly serving as the input to the next, achieving a complete closed-loop adaptive video transmission process.

[0072] First, a dataset is constructed and features are extracted. This step aims to provide representative and diverse training samples for subsequent model training. The system selects a publicly available immersive video dataset containing user head motion trajectories and viewpoint heatmaps, and cleans, synchronizes, and standardizes the original trajectories. Specifically, the original 3D trajectory data is projected onto an equidistant cylindrical coordinate system to eliminate errors caused by differences in user head movement amplitude and sampling frequency, and sliced ​​in one-second windows to generate continuous time segment samples. Each sample contains the user's viewpoint coordinate sequence within a specific time period, the corresponding video frame content, and external variables such as environmental bandwidth and packet loss rate. The data is divided into training, validation, and test sets in a 7:2:1 ratio, with high packet loss and high jitter network scene samples forcibly introduced into the test set to verify the system's robustness under extreme network conditions. The standardized sample data obtained through this process will serve as direct input to the subsequent viewpoint prediction model.

[0073] Next, viewport prediction modeling is performed. The goal of this stage is to predict the user's area of ​​interest within a short future time window, thus providing a spatial basis for differentiated encoding. The model adopts a Transformer architecture with a multi-input / output structure, extracting spatiotemporal correlation features through a multi-head attention mechanism. At the encoding end, the system embeds the input trajectory sequence, mapping the horizontal and vertical angle information of several historical frames into high-dimensional feature vectors. Based on this, the model introduces a feature distillation module, compressing the input sequence length and feature dimension through one-dimensional convolution and max pooling operations, reducing computational complexity by about 40%. The innovation of this model lies in its implicit ensemble strategy, which sets up multiple input / output heads on a shared backbone structure. Each sub-model independently performs future viewport prediction, and finally, the sub-model outputs are merged through a weighted average to generate the final viewport prediction result. This structure effectively suppresses the accumulation of bias in a single model without significantly increasing the number of parameters, improving the prediction generalization ability across users and scenes. The prediction result is output in the form of the viewport center point and range coordinates within the next second, serving as the basic input for the next stage of bitrate allocation.

[0074] The third step is bitrate decision generation and optimization. This step is based on the combination of representation learning and reinforcement learning to construct an adaptive dynamic bitrate allocation strategy. The system receives viewport prediction results and real-time network state parameters, such as available bandwidth, buffer length, and historical packet loss rate, and uses user preference weights as additional input. User preference weights reflect the relative importance that users place on image clarity, playback smoothness, and image stability. The model introduces mutual information terms into the reward function to maximize the retention of user preference information in the state-to-action mapping, enabling the reinforcement learning agent to understand and adapt to differences in preferences. The core process of bitrate decision-making includes three steps: first, determining the importance level of video tiles based on viewport results; second, calculating the target bitrate allocation value for each tile based on user preference weights and the current network state; and finally, adjusting the encoding parameters in real time through a dynamic update mechanism to cope with bandwidth changes. The system prioritizes allocating high bitrates to tiles within the viewport to ensure clarity, and uses a pyramid-shaped bitrate reduction strategy based on distance from the center for tiles outside the viewport to significantly reduce overall bandwidth overhead while maintaining subjective quality. The output of this step is a complete video tile bitrate mapping table, which is used by the transmission scheduling module.

[0075] The fourth step is adversarial robustness optimization training. Considering the significant impact of network state fluctuations and sudden packet loss on system performance in immersive video transmission, this invention introduces an adversarial perturbation mechanism during model training to enhance robustness. The system first establishes a bandwidth jitter model based on a normal distribution and a packet loss model based on a Bernoulli distribution. Simulated perturbations are dynamically injected into the training data, enabling the model to learn stable strategies under high volatility conditions. Furthermore, to simulate potential abnormal trajectory inputs or sudden errors, a gradient-direction-based perturbation generation method is used to slightly perturb the input samples, generating adversarial examples for training. In this way, the model is continuously exposed to adverse environments during the learning process, prompting it to learn more stable decision boundaries. In the reinforcement learning framework, the Actor module is responsible for generating actions (i.e., bitrate allocation schemes), and the Critic module evaluates the quality of the actions and guides updates; the two form a closed-loop feedback through alternating optimization. After adversarial robustness training, the system can still maintain video playback continuity in scenarios with up to 20% packet loss and 15 Mbps bandwidth jitter, with QoE fluctuations of less than 5%, thus ensuring stability under extreme network conditions.

[0076] The fifth step is the collaborative deployment and operation of cloud, edge, and terminal devices. To balance real-time performance and system scalability, this invention designs a layered deployment architecture. Terminal devices embed a lightweight viewport prediction model, which is optimized for inference using the TensorRT engine, with a memory footprint of less than 30 megabytes and a single prediction latency of no more than 36 milliseconds. Edge nodes deploy bitrate decision agents, utilizing the Flask framework to provide gRPC communication interfaces to receive viewport results and network status information uploaded by the terminals, generating and distributing video transmission parameters in real time. A complete training and policy update environment is deployed in the cloud, continuously optimizing model parameters and reward function weights through a containerized adversarial training module, and periodically pushing updated results to edge nodes, forming a closed-loop adaptive optimization mechanism. The system front-end is developed using a visual console based on Vue3 and WebGL to display real-time viewport heatmaps, bandwidth utilization, and QoE metric curves. It also allows operators to dynamically adjust bitrate thresholds, preference weights, and latency constraint parameters through a graphical interface, enabling online policy fine-tuning and rapid rollback in case of anomalies. The terminal, edge, and cloud maintain low-latency communication through a unified protocol interface to ensure real-time data synchronization and feedback at each stage.

[0077] Through the orderly connection of the above five steps, this invention constructs an intelligent content perception and optimization system for immersive video transmission. The system can automatically learn the viewing behavior patterns and experience preferences of different users, and generate optimal transmission strategies in real time under different network environments, achieving high-quality display in the core video area and bandwidth savings in non-core areas. Compared with traditional rule-based or single-model-based transmission schemes, this method significantly improves viewport prediction accuracy, bitrate decision flexibility, and network adaptability, achieving the technical effect of improving transmission efficiency and system stability while ensuring subjective experience. It can be widely applied to high-bandwidth, low-latency service scenarios such as cloud gaming, VR video streaming, and immersive remote interaction.

[0078] Implementation Method 3, in conjunction with Appendix Figure 1-5 This embodiment describes the technical solution provided above in further detail through specific examples. Specifically:

[0079] In the field of spatiotemporal feature extraction from video content, deep learning technology has become a core support. Convolutional Neural Networks (CNNs), with their powerful spatial feature extraction capabilities, play a crucial role in image and video key region recognition tasks. Krizhevsky's AlexNet model pioneered the widespread application of deep learning in computer vision, laying the foundation for subsequent video content analysis techniques. He's ResNet model, by introducing residual connections, effectively solved the gradient vanishing problem during deep network training, significantly improving the accuracy of region segmentation in video frame segmentation tasks, making the recognition of different regions in videos more precise.

[0080] In the processing of temporal features, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) have been widely used. The LSTM model proposed by Hochreiter and Schmidhuber can effectively capture the dynamic dependencies of video sequences and accurately track changes in user attention over time, providing technical support for analyzing users' dynamic interests during video viewing.

[0081] For cloud gaming and VR scenarios, key area recognition needs to balance accuracy and real-time performance. In international research, Tekin et al. proposed an attention-based VR field-of-view prediction model. This model, by analyzing user head movement data, achieves over 85% accuracy in predicting the central region of the user's field of view. However, its computational complexity is high, which may be limited by hardware performance in practical applications. Domestically, Zhang et al. developed a lightweight object detection algorithm suitable for cloud gaming, achieving a real-time processing frame rate of 30fps on a GPU platform and a recognition accuracy of 92% for game characters. However, its robustness in complex dynamic scenes still has room for improvement.

[0082] Quality of User Experience (QoE) modeling is a crucial step in balancing video transmission efficiency and user experience assurance. Traditional QoE evaluation methods often rely on objective metrics such as PSNR and SSIM, but these metrics often fail to accurately reflect users' subjective feelings. In recent years, subjective QoE modeling based on machine learning has become a research hotspot. Balachandran et al. constructed the first regression model that correlates video content features with user subjective ratings. This model reveals the influence of features such as video motion intensity and regional complexity on QoE, providing important insights for subsequent QoE modeling. Domestically, Huawei Technologies Co., Ltd. (2021) proposed a hybrid QoE model that integrates objective metrics and user behavior data. In cloud gaming scenarios, the prediction error is reduced by 23% compared to traditional methods, and it better reflects the real user experience in actual scenarios.

[0083] In terms of transmission optimization algorithms, Adaptive Bit Rate (ABR) technology has been widely used in the streaming media field. Akhtar et al. (2020) proposed an ABR algorithm based on deep reinforcement learning, which achieved a 15% improvement in bandwidth utilization and a 20% reduction in buffering time by dynamically adjusting the bit rate, effectively optimizing the efficiency of video transmission. Domestically, the intelligent transmission scheduling system developed by the Institute of Computing Technology, Chinese Academy of Sciences, combined network state prediction and content priority ranking, achieving end-to-end latency control within 40ms in VR video transmission. However, the stability of the system in multi-user concurrent scenarios still needs further verification.

[0084] Content-aware differentiated transmission is the core innovation of this project, and related research has made some progress. In terms of spatial differentiated transmission, Jiao et al. (2019) proposed a region coding scheme based on visual saliency. This scheme uses high bit rate transmission for salient regions of the image and reduces the resolution for non-salient regions. While achieving a 30% bandwidth saving, it maintains no significant decrease in subjective quality, verifying the effectiveness of spatial differentiated transmission in saving bandwidth.

[0085] In the time dimension, Müller et al. (2021) designed a dynamic update frequency mechanism to adjust the transmission interval of video frames according to changes in user attention, achieving an 18% bandwidth saving in VR roaming scenarios, providing a reference for the optimization of transmission strategies in the time dimension.

[0086] Domestic research focuses more on adapting to actual business needs. Tencent Technology (2023) deployed a block-based differentiated transmission system in its cloud gaming platform, dividing the game screen into character, background, and interaction areas, and using different encoding parameters for different areas. Actual testing showed a 25% reduction in bandwidth consumption and a 12% increase in user satisfaction. However, most existing solutions rely on manually defined area priorities, lack adaptability to dynamic user interests, and have limitations in cross-scenario applicability, leaving considerable room for improvement.

[0087] In this implementation, FRAMES, a chunk-based immersive video streaming system based on ensemble and representation learning, is proposed to address the challenge of user diversity and improve generalization. To capture viewing pattern diversity, a viewport prediction model is developed, employing an efficient multi-viewport trajectory input / output (MTIO) architecture based on implicit ensemble learning (EL). The MTIO architecture implicitly trains multiple sub-models by building multiple input-output heads, incurring low computational cost. Each sub-model makes predictions independently, and their predictions are ensembled to produce a well-calibrated predicted viewport, thereby reducing prediction bias and resulting in stronger generalization. Furthermore, the backbone of the prediction model is designed based on the MTIO Transformer, which utilizes an attention mechanism to effectively learn long-term dependencies. This enables the model to more accurately predict viewport movement trends, further improving prediction accuracy.

[0088] To adapt to diverse user QoE preferences, an advanced representation learning (Repl.) technique is used to train the DRL bitrate selection model. Specifically, by incorporating mutual information into the reward function of the training model, the model is encouraged to mine useful hidden representations from the user's QoE preferences. This enables the model to capture fundamental features of user preferences, such as the priority of bitrate quality and playback smoothness. This gives the model the ability to dynamically select bitrates based on user QoE preferences, achieving strong generalization capabilities even when encountering preferences not seen during training. Furthermore, since directly computing mutual information is difficult, an efficient neural network model is designed to estimate the mutual information term used for reward calculation.

[0089] The plan includes:

[0090] (1) Multimodal spatiotemporal modeling

[0091] The project employs the MTIO-Transformer deep network architecture for viewport prediction. It extracts long-range spatiotemporal feature dependencies through a multi-head attention mechanism and combines LSTM to analyze the dynamic changes in pitch and yaw angles in the user's head motion trajectory. It innovatively introduces a feature distillation module (1D convolutional layer + max pooling layer) to compress 40% of redundant feature computation, and finally outputs an integrated prediction result v_{t+j}^e = \frac{1}{M}\sum_{i=1}^M v_{t+j}^i, effectively eliminating the prediction bias problem of single models.

[0092] (2) Represents learning-driven bitrate decision

[0093] A mutual information optimization framework is designed to solve the QoE preference adaptation problem: The DRL agent receives user-defined QoE preference weights w=(λ1,λ2,λ3) and environmental state s_c (including network bandwidth and buffer data). The QoE identifier neural network estimates the mutual information term I(w;a_c,s_c) and constructs the reward function rew_c = (1-α)QoE_c(w) - α·log MSE(w;Q_δ(s_c,a_c)). In particular, the preference feature propagation is enhanced by residual connections, which improves the training convergence speed by 35% and supports real-time bitrate decision-making under 8 dynamic preference configurations.

[0094] (3) Resistance robustness training

[0095] A network disturbance simulation and adversarial training mechanism was established: a normal distribution bandwidth jitter model ...

[0096] (4) Cross-platform monitoring system

[0097] Develop a visualization console based on Vue3 and WebGL, integrating real-time viewport heatmap rendering, bandwidth consumption monitoring dashboard, and QoE score curve display functions; the backend provides a gRPC interface through the Flask framework, supporting the transmission of strategy parameters (such as bitrate threshold / latency constraints) in JSON format; the deployment scheme includes: running a TensorRT-accelerated MTIO model (inference latency <36ms) on edge nodes, a cloud-based Docker containerized adversarial training environment, and lightweight WebGL trajectory annotation on the client, forming an end-to-end deployable system.

[0098] This implementation method constructs an immersive video transmission system based on MTIO-Transformer viewport prediction and representation learning bitrate decision-making. The overall architecture is as follows: Figure 1 As shown. The training data comes from publicly available immersive video datasets (Wu2017 / Jin2022) and head motion trajectory logs, and a noise injection pool is constructed using a bandwidth perturbation model. The core innovation is the implicit integration of viewport prediction and mutual information-driven bitrate selection. The specific implementation steps are as follows:

[0099] 1. Dataset Construction and Processing:

[0100] We collected the Wu2017 (48 users × 8 videos) and Jin2022 (60 users × 24 videos) datasets, requiring the inclusion of user head motion trajectories (horizontal / vertical coordinates) and eye-tracking heatmap labels, with a total duration exceeding 100 hours. Scenes with sudden viewpoint changes and high-speed motion were specifically annotated. During the data cleaning stage, the raw trajectories were converted to equidistant cylindrical projection coordinates and processed in 1-second window slices. Feature extraction employed a multi-head trajectory input architecture: historical viewpoint trajectories {\hat{v}_{th}^i,...,\hat{v}t^i}{i=1}^M were projected into 512-dimensional embedding vectors. The datasets were divided into training / validation / test sets at 70% / 20% / 10%, with the test set mandatory to include 20% high packet loss scenarios (bandwidth jitter \mathcal{N}(50,15) Mbps, packet loss rate B(0.05)). Processed data was stored in the HDFS distributed system.

[0101] 2. Model Training and Evaluation:

[0102] The viewport prediction model adopts the MTIO-Transformer architecture (structure as follows) Figure 2 As shown): The encoder-decoder structure (N_{block}=2 layers) extracts spatiotemporal features, the multi-head attention mechanism (N_{ah}=8) learns long-range dependencies, and the feature distillation module (1D convolution + max pooling) compresses the sequence length by 50%. The loss function is defined as the period-aware mean squared error.

[0103] The bitrate decision model is based on a representation learning framework (process as follows) Figure 3 As shown): The DRL agent inputs QoE preferences w=(\lambda_1,\lambda_2,\lambda_3) and environmental state s_c (bandwidth / buffer / historical QoE), and the QoE identifier outputs mutual information reward rew_c = (1-\alpha)QoE_c(w) - \alpha \log MSE(w;Q_\delta(s_c,a_c)). Adversarial training injects FGSM perturbation trajectories \hat{v}_{adv} = v + \epsilon \cdot sign(\nabla_v J) to improve robustness. Evaluation metrics: viewport prediction mIoU≥0.85, bandwidth saving rate≥30%, end-to-end latency≤40ms (see Table 1 for details).

[0104]

[0105] 3. Platform Construction:

[0106] The system deployment adopts a cloud-edge-device collaborative architecture: the client integrates a lightweight MTIO model (TensorRT accelerated, 28.15MB memory) to render viewport heatmaps in real time; the edge nodes deploy a DRL agent decision engine, receiving policy parameters (JSON-encapsulated bitrate thresholds / latency constraints) via a gRPC interface provided by Flask; and the cloud runs an adversarial training environment (Docker containerized). The front-end console is developed based on Vue3 + WebGL, implementing three main functions: ① Dynamic visualization of viewpoint trajectory ② Real-time bandwidth / QoE dashboard ③ Dynamic adjustment panel for policy parameters. Workflow: User uploads video → Edge nodes predict viewport → DRL agent allocates bitrate → Cloud optimizes global policy → Terminal displays QoE feedback.

[0107] The above description of several specific embodiments further details the technical solution provided by the present invention in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above-described specific embodiments are not intended to limit the present invention. Any reasonable modifications and improvements to the present invention, combinations of embodiments, and equivalent substitutions based on the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An immersive video perceptual transmission method based on implicit integrated viewport prediction and representation learning rate decision, characterized in that, include: The steps include: acquiring immersive video data containing user head motion trajectories and field-of-view heatmap labels; cleaning, synchronizing, and standardizing the original trajectories; mapping the trajectories to equidistant bar coordinates and slicing them according to time windows; and generating a multimodal feature sample dataset. The steps involve establishing a viewport prediction model with a multi-input-output structure based on a sample dataset, embedding the user's historical viewpoint sequence into a vectorized form, extracting spatiotemporal features using a multi-head attention mechanism, compressing the high-dimensional sequence through a feature distillation module, and fusing the outputs of multiple sub-models using an implicit ensemble strategy to generate future viewport prediction results. During the data construction and feature processing, the original head motion trajectory is projected into equidistant cylindrical coordinates and then subjected to one-dimensional convolution and max pooling feature compression. Sample fragments are generated through a time sliding window. The viewport prediction model adopts a two-layer structure consisting of an encoder and a decoder. The encoder is used to extract global spatiotemporal dependency features of the user's viewpoint sequence, and the decoder is used to generate coordinate prediction results of the future viewport range. Cross-frame feature fusion is achieved through a multi-head attention mechanism and combined with an implicit ensemble strategy to output the global predicted viewport. Based on the viewport prediction results, user experience preference weights, and network state information, the reinforcement learning decision engine is input. A reward function with mutual information constraints is constructed to maintain the transmission of preference features. A bitrate allocation table for video tiles is dynamically generated. High bitrate is allocated to tiles within the viewport, and the bitrate of tiles outside the viewport is gradually reduced according to the distance from the center. The real-time transmission strategy parameters are output. The reinforcement learning decision engine introduces a mutual information-driven reward function during the training phase, using user preference weights, bandwidth fluctuations, and buffer states as input variables to achieve dynamic transfer between different preferences while maintaining a balance between image quality and playback smoothness. The bitrate allocation strategy manages video tiles in layers. Tiles within the viewport are transmitted at a high bitrate to ensure center sharpness, while tiles outside the viewport are divided into multiple ring zones according to their distance from the center of the viewport, each using a decreasing bitrate level. The steps include injecting bandwidth jitter and packet loss perturbation models during the training phase, optimizing the decision network by combining gradient-direction-based adversarial example generation methods, updating the policy parameter set, and outputting the policy model weights. In a collaborative environment consisting of terminals, edge nodes, and the cloud, a viewport prediction model with a multi-input-output structure is deployed. The terminal performs lightweight viewport prediction in real time and uploads the results. Edge nodes make bitrate decisions based on the prediction and network status and issue transmission parameters. The cloud updates the strategy periodically and synchronizes it to the edge nodes. The end-to-end adaptive video transmission is maintained through closed-loop feedback.

2. The immersive video perception transmission method based on implicit integrated viewport prediction and representation learning bitrate decision according to claim 1, characterized in that, During data construction and feature processing, the original head motion trajectory is projected into equidistant cylindrical coordinates and then subjected to one-dimensional convolution and max pooling feature compression. Sample fragments are generated through a time sliding window.

3. The immersive video perception transmission method based on implicit integrated viewport prediction and representation learning bitrate decision according to claim 1, characterized in that, The viewport prediction model adopts a two-layer structure consisting of an encoder and a decoder. The encoder is used to extract global spatiotemporal dependency features of the user's viewpoint sequence, and the decoder is used to generate coordinate prediction results of the future viewport range. Cross-frame feature fusion is achieved through a multi-head attention mechanism and combined with an implicit ensemble strategy to output the global predicted viewport.

4. The immersive video perception transmission method based on implicit integrated viewport prediction and representation learning bitrate decision-making according to claim 1, characterized in that, The reinforcement learning decision engine introduces a mutual information-driven reward function during the training phase, using user preference weights, bandwidth fluctuations, and buffer states as input variables to achieve dynamic transfer between different preferences while maintaining a balance between image quality and playback smoothness.

5. The immersive video perception transmission method based on implicit integrated viewport prediction and representation learning bitrate decision according to claim 1, characterized in that the bitrate... The allocation strategy manages video tiles in layers. Tiles within the viewport are transmitted at a high bit rate to ensure center sharpness, while tiles outside the viewport are divided into multiple ring zones based on their distance from the center of the viewport, each using a decreasing bit rate level.

6. The immersive video perception transmission method based on implicit integrated viewport prediction and representation learning bitrate decision according to claim 1, characterized in that, During adversarial robust training, a random perturbation environment with bandwidth jitter and packet loss is constructed, and a perturbation sample generation method based on gradient direction is adopted.

7. An immersive video sensing transmission device based on implicit integrated viewport prediction and representation learning bitrate decision, characterized in that, include: This module acquires immersive video data containing user head motion trajectories and field-of-view heatmap labels, cleans, synchronizes, and standardizes the original trajectories, maps the trajectories to equidistant bar coordinates and slices them according to time windows, and generates a multimodal feature sample dataset. A viewport prediction model with a multi-input-output structure is established based on a sample dataset. The user's historical viewpoint sequence is vectorized and embedded, spatiotemporal features are extracted using a multi-head attention mechanism, high-dimensional sequences are compressed through a feature distillation module, and the outputs of multiple sub-models are fused using an implicit ensemble strategy to generate the future viewport prediction results. During the data construction and feature processing, the original head motion trajectory is projected into equidistant cylindrical coordinates and then subjected to one-dimensional convolution and max pooling feature compression. Sample fragments are generated through a time sliding window. The viewport prediction model adopts a two-layer structure consisting of an encoder and a decoder. The encoder is used to extract global spatiotemporal dependency features of the user's viewpoint sequence, and the decoder is used to generate coordinate prediction results of the future viewport range. Cross-frame feature fusion is achieved through a multi-head attention mechanism and combined with an implicit ensemble strategy to output the global predicted viewport. Based on viewport prediction results, user experience preference weights, and network state information, a reinforcement learning decision engine is input. A reward function with mutual information constraints is constructed to maintain the transmission of preference features. A bitrate allocation table for video tiles is dynamically generated. High bitrate is allocated to tiles within the viewport, and the bitrate of tiles outside the viewport is gradually reduced according to the distance from the center. A module that outputs real-time transmission strategy parameters is also provided. The reinforcement learning decision engine introduces a mutual information-driven reward function during the training phase, using user preference weights, bandwidth fluctuations, and buffer states as input variables to achieve dynamic transfer between different preferences while maintaining a balance between image quality and playback smoothness. The bitrate allocation strategy manages video tiles in layers. Tiles within the viewport are transmitted at a high bitrate to ensure center sharpness, while tiles outside the viewport are divided into multiple ring zones according to their distance from the center of the viewport, each using a decreasing bitrate level. A module that injects bandwidth jitter and packet loss perturbation models during the training phase, and combines gradient-direction-based adversarial example generation methods to optimize the decision network, update the policy parameter set, and output the policy model weights. A multi-input-output viewport prediction model is deployed in a collaborative environment consisting of terminals, edge nodes, and the cloud. The terminal performs lightweight viewport prediction in real time and uploads the results. Edge nodes make bitrate decisions based on the prediction and network status and issue transmission parameters. The cloud updates the strategy periodically and synchronizes it to the edge nodes. This module maintains end-to-end adaptive video transmission through closed-loop feedback.

8. A computer storage medium for storing computer programs, characterized in that, When the computer program is read by the computer, the computer executes the method of claim 1.

9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method of claim 1.

10. A computer program product, as a computer program, is characterized by: When the computer program is executed, it implements the method of claim 1.

Images