Adaptive code rate allocation method based on spatio-temporal graph neural network

Abstract

The application provides an adaptive code rate allocation method based on a space-time graph neural network. The method comprises the following steps: S1: panoramic tiling step; S2: dynamic space-time graph construction step; S3: space-time graph neural network inference step; S4: bandwidth weight generation step; S5: agent local collaborative decision step; S6: global verification and unified fine-tuning step; and S7: encoding and transmission step. The method provided by the application can automatically converge the bandwidth allocation without additional control delay, and can still preferentially allocate resources to the area that a user is about to see in the case of bandwidth limitation, so as to ensure that the picture is clear, stable and low-delay in the instant of turning head.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, specifically to an adaptive bitrate allocation method based on a spatiotemporal graph neural network. Background Technology

[0002] With the development of Virtual Reality (VR) technology, real-time VR live streaming based on cloud rendering has become an important application form for realizing multi-person synchronous immersive experiences. Currently, mainstream cloud VR systems use cloud servers to complete the real-time rendering, encoding, and transmission of 360° panoramic video, while user terminals are only responsible for decoding and display, which can significantly reduce the computing and power consumption burden on local devices.

[0003] To maintain image clarity and smooth interaction under limited network bandwidth, mainstream solutions generally employ spherical tile encoding technology. This technology first divides the panoramic video into six faces using cube map projection, then further divides each face into multiple tiles according to fixed rows and columns, encoding and configuring the bitrate for each tile independently. When bandwidth is limited, the system prioritizes allocating higher bitrates to tiles within the user's current viewport, while using lower bitrates for areas outside the viewport to improve bandwidth utilization.

[0004] However, in real-world VR live streaming scenarios, users often make large, rapid head movements. Given the limited network bandwidth and the instantaneous nature of head movements, if the system cannot predict the user's upcoming viewing direction and dynamically adjust the bitrate, "field of view blurring" and "cross-tile tearing" can easily occur, reducing visual continuity and immersive experience. Existing solutions to this problem include prediction methods based on historical head movement trajectories or saliency maps, centralized allocation methods based on global heatmaps, and traditional bitrate allocation methods based on content complexity or RD models. However, all of these methods have limitations.

[0005] Firstly, in prediction methods based on historical head movement trajectories or saliency maps, the saliency map is the result of an algorithm that calculates the visual attention region in an image or video. The system predicts the user's attention region and improves the tile bitrate of that region by using historical head movement trajectories or saliency maps. However, this type of data can only predict "where the user will ultimately look," and cannot capture the path swept by the gaze during head turning. Blur and discontinuities are still prone to occur during head turning.

[0006] Secondly, in the centralized allocation method based on a global heatmap, the global heatmap is generated by the system by statistically analyzing the viewing directions of all users, reflecting the overall level of attention, and the system allocates bandwidth accordingly. However, this method is more suitable for on-demand or high-latency scenarios, not for live streaming, as it cannot reflect changes in the current gaze of an individual user, nor can it make fine adjustments on a 200ms timescale.

[0007] Finally, in traditional bitrate allocation methods based on content complexity or RD models, the bitrate is adjusted according to the tile texture complexity or content motion, without considering whether the user will watch the area, resulting in low bandwidth utilization efficiency.

[0008] In addition, the above methods also have common problems: compensation can only be performed "after the head has turned", lacking forward-looking configuration; prediction relies solely on the head motion trajectory and cannot simultaneously consider the visual axis scanning path and cross-tile motion correlation; centralized scheduling relies on frequent communication, resulting in high system latency; bandwidth allocation is mainly global and cannot be refined to the tile level, which easily leads to uneven local quality.

[0009] Therefore, there is a need to provide a technical solution that can solve the above-mentioned problems in the prior art. Summary of the Invention

[0010] In view of this, this application provides an adaptive bit rate allocation method based on a spatiotemporal graph neural network to solve the above-mentioned technical problems in the prior art.

[0011] According to one aspect of this application, an adaptive code rate allocation method based on a spatiotemporal graph neural network is provided, comprising the following steps:

[0012] S1: The panoramic tileization step includes unfolding the VR video into a cube, dividing it into a set of spherical tiles, and establishing an adjacency table for each tile and its surrounding tiles; creating a tile agent for each tile, which stores the following information: tile ID, neighboring tile IDs, current actual bitrate, current complexity index, and current bitrate quota;

[0013] S2: Dynamic spatiotemporal graph construction steps, including constructing a spatiotemporal graph that changes over time based on the user's real-time head posture and the content of the VR video. The spatiotemporal graph includes tile nodes, orientation edges, motion edges, and virtual nodes. Tile nodes represent tiles, orientation edges represent the relationship of the gaze sweep, motion edges represent the motion continuity relationship across tile content, and virtual nodes represent tiles that may enter the field of view.

[0014] S3: Spatiotemporal graph neural network inference steps, including inputting the dynamic spatiotemporal graph into the spatiotemporal graph neural network to obtain the attention probability and confidence of each tile;

[0015] S4: Bandwidth weight generation step, including generating the bandwidth weight of each tile based on the attention probability of each tile;

[0016] S5: Local cooperative decision-making steps of the agents, including each tile agent calculating the effective required bit rate of the tile based on the tile's bandwidth weight and texture complexity;

[0017] S6: Global verification and unified fine-tuning steps, including verifying whether the total bitrate exceeds the global bandwidth limit based on the effective required bitrate of each tile; if so, adjusting the effective required bitrate of each tile to determine the final bitrate of each tile.

[0018] S7: Encoding and transmission steps: Encode the VR video according to the final bitrate of each tile, and send the encoded video stream to the client device.

[0019] According to a preferred embodiment of this application, in step S2, the user's real-time head posture is represented by a head posture sequence, which describes the user's head turning trajectory over a recent period of time.

[0020] According to a preferred embodiment of this application, step S2 includes:

[0021] Based on the head posture sequence, determine the tiles that the user's line of sight may sweep over in the future, and establish orientation edges from the current center tile of the field of vision to the tiles that the user's line of sight may sweep over in the future.

[0022] Based on the significant motion characteristics between adjacent tiles, establish motion edges between adjacent tiles;

[0023] The system aggregates the tiles that the user's gaze may sweep over within a certain period of time in the future, forming virtual nodes, and feeds the aggregation results back to each tile's intelligent agent.

[0024] According to a preferred embodiment of this application, step S3 includes: injecting the head pose sequence and the motion salient features of each tile into each tile agent.

[0025] According to a preferred embodiment of this application, step S3 further includes:

[0026] The information propagation and feature aggregation steps include, for oriented edges, only propagating posture information related to the user's line of sight; for motion edges, only propagating information related to content motion.

[0027] The delayed fusion step includes aggregating the attitude flow and motion flow separately, and then fusing them at the end of the window to generate a fused feature vector, which represents the overall attention intensity of the tile in the future.

[0028] The virtual node aggregation and input steps include injecting the fused feature vector into the virtual node and outputting the attention probability and confidence of each tile.

[0029] According to a preferred embodiment of this application, step S4 includes:

[0030] The bandwidth control parameters for each tile are determined based on the attention probability of each tile.

[0031] Bandwidth control parameters are transmitted to the tile agent via video stream.

[0032] According to a preferred embodiment of this application, the bandwidth control parameters include bandwidth weights, where w is the bandwidth weight of the i-th tile. i Determined by the following nonlinear piecewise function:

[0033] , where p i Let T1 and T2 be the attention probability of the i-th tile, and T1 and T2 be the low threshold and high threshold, respectively. β is the scaling factor.

[0034] According to a preferred embodiment of this application, step S5 includes:

[0035] Multiply the bandwidth weight of each tile by the texture complexity to obtain the effective demand value;

[0036] Calculate the total demand value of tiles in the neighborhood. If the total demand value is higher than the upper limit allocated by the system, maintain the effective demand code rate of the tile with the highest effective demand value and reduce the effective demand code rate of other tiles.

[0037] According to a preferred embodiment of this application, in step S6, if the total bit rate exceeds the global bandwidth limit, the bit rate of tiles with low confidence and high effective demand bit rate is preferentially reduced, and boundary smoothing is performed around the tiles whose bit rate is reduced.

[0038] According to a preferred embodiment of this application, step S7 includes: sending the encoded video stream to the client device using a differential transmission mechanism.

[0039] According to another aspect of this application, a VR system is provided, including a cloud server and a client device, the VR system being configured to perform the methods described above.

[0040] The embodiments of this application can achieve the following technical effects:

[0041] 1. More accurate predictions

[0042] The method in this application establishes a dynamic spatiotemporal map in the cloud, integrating user head posture changes and screen motion information to predict tiles that may enter the field of view within a certain period of time (e.g., approximately 200 milliseconds). This prediction method is more accurate than traditional methods that rely on saliency maps or simple trajectories, and can identify the area that the user is about to see in advance.

[0043] 2. More efficient bandwidth allocation:

[0044] The method in this application allocates bandwidth priority to different tiles based on the predicted level of interest. Through a nonlinear mapping function and a differential transmission mechanism, high-interest areas automatically receive higher bit rates, while bandwidth in non-interest areas is automatically reduced, minimizing bandwidth waste. Furthermore, no additional control signaling is required, resulting in lower latency.

[0045] 3. Faster and more stable decision-making:

[0046] On the client side, each tile has an independent intelligent agent that quickly determines the bitrate through local and neighborhood collaboration. This mechanism eliminates the need for centralized scheduling, completes bandwidth allocation within milliseconds, and maintains a stable total bandwidth through global checks, ultimately ensuring a clear, continuous, and uninterrupted image the instant the user turns their head. Attached Figure Description

[0047] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application, but do not constitute a limitation on the technical solutions of this application.

[0048] Figure 1 A flowchart illustrating a method of an exemplary embodiment of this application is shown;

[0049] Figure 2 A structural diagram of the VR system provided in an exemplary embodiment of this application is shown;

[0050] Figure 3 A structural block diagram of an electronic device provided by an exemplary embodiment of this application is shown. Detailed Implementation

[0051] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. The descriptions of the exemplary embodiments are merely illustrative and are not intended to limit the scope of this application or its application or use. This application can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to make the application thorough and complete, and to fully express the scope of this application to those skilled in the art.

[0052] Unless explicitly stated otherwise, an element may be one or more unless otherwise specified. The terms “multiple / several” mean two or more, the term “based on” should be interpreted as “at least partially based on,” and the terms “and / or” and “at least one of…” cover any one of the listed items and all possible combinations thereof. Furthermore, expressions such as “first,” “second,” etc., are for descriptive purposes only and do not indicate or imply their relative importance or implicitly specify the number of technical features indicated.

[0053] This application relates to a cloud-based VR spherical tile adaptive bitrate allocation method based on spatiotemporal graph neural network (ST-GNN) and multi-agent collaboration, which is used to predict changes in the user's field of view in advance and perform intelligent bandwidth allocation under bandwidth-limited conditions, thereby maintaining image clarity and continuity in real-time VR live streaming scenarios.

[0054] In cloud-based VR live streaming, a 360° panoramic view is broken down into many spherical tiles. Each tile has a fixed adjacency relationship with its surrounding tiles, which naturally forms a "graph" structure. At the same time, as time goes by, the user's perspective and the content of the image are constantly changing, so a model that can handle both spatial relationships (connections between tiles) and temporal changes (frame sequences) is needed.

[0055] Spatiotemporal graph neural networks are designed for this purpose. They capture the correlations between tiles in the spatial dimension and track their dynamic changes in the temporal dimension, thus predicting which tiles will enter the user's field of view and attract attention in the future, for example, about 200ms.

[0056] Meanwhile, in this embodiment, an independent agent is designed for each tile, which can independently make bitrate decisions based on its own local information (such as prediction weights, texture complexity, and the state of neighboring tiles). Multiple agents coordinate quickly through local communication, thereby collaboratively completing the overall bandwidth allocation and jointly achieving the bitrate target without the need for centralized control.

[0057] This application's embodiments utilize a dynamic spatiotemporal graph neural network to predict user head movement trends and image motion characteristics in advance, identifying tiles that may enter the time domain within a certain period (e.g., approximately 200ms). These predictions are then converted into corresponding bandwidth allocation weights and sent in real-time along with the video stream to the agent corresponding to each tile. Each agent, based on the received weights and its own situation, adjusts the bitrate in parallel within its local neighborhood, deciding which tile maintains a high bitrate and which appropriately lowers it. This approach allows bandwidth allocation to converge automatically without additional latency control. Even with limited bandwidth, it prioritizes resources for the area the user is about to see, ensuring a clear, stable, and low-latency image when the user turns their head.

[0058] refer to Figure 1 The diagram illustrates a flowchart of a method according to an exemplary embodiment of this application. Figure 1 As shown, the method according to an exemplary embodiment of this application includes: S1: panoramic tile generation step; S2: dynamic spatiotemporal graph construction step; S3: spatiotemporal graph neural network inference step; S4: bandwidth weight generation step; S5: agent local collaborative decision-making step; S6: global verification and unified fine-tuning step; S7: encoding and transmission step.

[0059] Figure 2 A structural diagram of the VR system provided in an exemplary embodiment of this application is shown. Figure 2 The VR system includes cloud devices and client devices. The cloud devices can transmit encoded VR videos to client devices via a network (such as the Internet). The cloud devices can be, for example, server devices, and the client devices can be, for example, VR glasses or other VR devices.

[0060] The following will combine Figure 1 and Figure 2 The specific description includes steps S1 to S7 of the method according to the embodiments of this application.

[0061] S1: Panoramic Tile Conversion Steps

[0062] At the start of the cloud-based VR live streaming session, the cloud device first unfolds the 360° panoramic video into a cube, dividing the image into several rectangular tiles according to fixed row and column dimensions. These tiles together form a set of tiles for the entire sphere.

[0063] To enable subsequent prediction and bandwidth allocation processes to be based on spatial relationships, cloud devices establish adjacency relationships between each tile and its surrounding tiles. This adjacency information forms a spherical tile adjacency table, which is used to construct the spatial structure of the graph neural network.

[0064] Next, the cloud device creates a lightweight tile agent for each tile. Each agent is essentially an independent decision-making unit, primarily storing the following information:

[0065] - The unique index ID of this tile;

[0066] - Its adjacency list (records the IDs of adjacent tiles);

[0067] - The current actual bitrate level;

[0068] - Current complexity metrics (statistics compiled by the encoder for each frame);

[0069] - Current bitrate quota.

[0070] S2: Dynamic Spatiotemporal Graph Construction:

[0071] After tile initialization is complete, the system will construct a spatiotemporal graph that changes over time based on the user's real-time behavior and screen content, in order to predict which tiles are more likely to be seen by the user in the future (e.g., about 200ms).

[0072] In a dynamic spatiotemporal graph, nodes represent tiles, and edges represent the relationships between tiles. The graph structure integrates the following two types of input information:

[0073] - Head pose sequence: describes the user's head turning trajectory over a recent period of time (e.g., approximately 300 ms);

[0074] - Tile motion saliency map: Extracted by the encoder or optical flow module in each frame, reflecting the motion amplitude, direction and trend of the tile content.

[0075] These two types of information come from the head pose sensor and the video encoding module, respectively. The system updates them in real time during each calculation cycle and dynamically generates a spatiotemporal map based on this information. The construction process of the dynamic spatiotemporal map mainly consists of the following three steps:

[0076] 1. Determine the user's gaze path: Based on recent head posture changes over a short period (e.g., approximately 300 ms), calculate the area the user's gaze may traverse within a future period (e.g., approximately 200 ms). For these tiles that are "about to be scanned by the gaze," establish directional edges from the current center tile of the field of view to them, representing the path the gaze will take.

[0077] 2. Identify cross-tile motion relationships: Compare the salient features of motion between adjacent tiles. If two tiles move in similar directions and with similar amplitudes, it indicates that an object in the image may be moving across these two tiles. In this case, establish a motion edge between them to represent this visual continuity across tiles.

[0078] 3. Add a forward-looking virtual node: The cloud device creates a virtual node to aggregate information from all potential tiles that may enter the field of view. This virtual node aggregates this information on the layer surface to form a global understanding of the future field of view, and feeds the aggregation results back to each tile agent, so that each node (tile) can know its relative importance in the future field of view.

[0079] In the dynamic spatiotemporal graph structure formed in step S2:

[0080] - Nodes represent tiles;

[0081] - The direction the eye moves towards represents the relationship being scanned;

[0082] -Movement edges represent the continuity of content across tiles;

[0083] - Virtual nodes are responsible for aggregating potentially interested regions.

[0084] This dynamic spatiotemporal graph structure is then fed into a spatiotemporal graph neural network, which learns the spatial and temporal patterns of change and outputs the attention probability and confidence for each tile. The attention probability represents the likelihood that the tile will be seen by the user within the next 200ms. The confidence represents the network's certainty about the prediction result.

[0085] S3: Spatiotemporal graph neural network inference:

[0086] After the dynamic spatiotemporal graph is constructed, the system inputs it into a spatiotemporal graph neural network to predict which tiles are most likely to enter the user's field of view within a future time period (e.g., approximately 200 ms). The goal of the spatiotemporal graph neural network is to combine spatial structure and temporal change trends to calculate the importance of each tile in the short term, providing a basis for subsequent bandwidth allocation.

[0087] At this stage, the spatiotemporal graph neural network injects the head pose sequence and tile motion salient features collected in step S2 into the internal state of each node. Each tile node maintains two types of internal memory:

[0088] - Pose Memory: Records changes in the user's head posture over time to reflect the trend of eye movement;

[0089] - Motion Memory: Records the motion characteristics of the tile's own content to reflect dynamic information in the image.

[0090] The reasoning process in step S3 includes the following steps:

[0091] 1. Information dissemination and feature aggregation:

[0092] 1) For the directional edge, only the posture information related to the user's line of sight is propagated, so that the tiles that the "view axis is about to sweep over" will influence each other;

[0093] 2) For motion edges, only information related to content motion is propagated to ensure that continuous motion across tiles is captured correctly;

[0094] 3) Unrelated edges do not propagate messages to avoid redundant calculations and interference.

[0095] After receiving messages from neighboring nodes, each node combines its own historical state with a short-term convolution operation with a fixed window (e.g., about 200 ms) over time. This step is equivalent to smoothing and aggregating the most recent frames over time, allowing the network to focus more on trends rather than instantaneous fluctuations.

[0096] 2. Delayed fusion:

[0097] After temporal convolution, the network aggregates the pose flow and motion flow separately, and then fuses them at the end of the window. Fusion is performed at the node level, preserving the differences between tiles. The fusion method involves feature concatenation and linear mapping. Feature concatenation combines the pose and motion flow features into a joint vector feature. Then, a linear mapping layer is used to obtain the fused feature. In cloud VR scenarios, user head-turning signals (pose changes) typically precede changes in screen content, so a delayed fusion strategy is needed. The two feature flows are allowed to accumulate independently within a time window (e.g., approximately 200 ms), and then aligned and fused at the end of the window. This delayed fusion mechanism aligns the "user head-turning signal" and the "screen motion signal" in time, avoiding prediction jitter caused by the asynchrony of the two types of information. The fused feature vector represents the overall attention intensity of the tile in the near future.

[0098] 3. Virtual Node Aggregation and Result Output

[0099] The fused node feature vectors are fed into the lookahead virtual nodes. The virtual nodes aggregate information from all potential tiles that "might enter the field of view," assess their importance from a global perspective, and feed the results back to each node. Ultimately, the network outputs two results for each tile: attention probability and confidence score.

[0100] The following details the calculation methods for the probability of attention and the confidence score. Based on the comprehensive node feature vector obtained after the above fusion, the system outputs a value between 0 and 1 through a linear transformation and a sigmoid function, which serves as the probability of attention. The specific calculation process is as follows:

[0101] ,

[0102] , in,

[0103] It is a node-fused feature vector that combines attitude flow and motion flow;

[0104] It is a weight vector, which consists of parameters trained in advance;

[0105] It is a bias;

[0106] It is a score, which comprehensively represents the "potential strength" of user attention to the tile;

[0107] It is a temperature parameter that controls the smoothness of the sigmoid function; the larger the value, the smoother the output.

[0108] This yields the probability of interest, which is then mapped to the 0-1 interval using the sigmoid function.

[0109] Confidence level reflects the "certainty" of a model's predictions. It can be obtained by analyzing the stability of the probability distribution of interest. In practice, it can be calculated using the entropy of the Bernoulli distribution.

[0110] ,

[0111] If the model predicts a value close to 0 or 1 (very certain), the confidence level is high; if the prediction is close to 0.5 (uncertain), the confidence level is low.

[0112] The probability and confidence scores are used as inputs to step S4 to generate bandwidth weights and guide bitrate configuration, thereby pre-allocating resources before the user turns away.

[0113] S4: Bandwidth Weight Generation

[0114] The attention probability of each tile obtained in step S3 is converted into executable bandwidth control parameters and sent to each tile agent in real time via video stream to guide subsequent bitrate adjustments.

[0115] Cloud devices calculate the corresponding bandwidth weight based on the attention probability of each tile. To prioritize the clarity of critical regions under limited bandwidth conditions, the system employs a nonlinear piecewise mapping function, which significantly increases the weight of high-interest regions.

[0116] , in:

[0117] Let i be the attention probability of the i-th tile;

[0118] These are the low threshold and the high threshold, respectively.

[0119] , This is the scaling factor, used to control the shape of the curve.

[0120] This non-linear allocation method can improve the model's sensitivity to key visual regions while keeping the total bandwidth constant, making the sharpness allocation more consistent with human subjective perception. The calculated bandwidth weights are updated in real time.

[0121] To reduce control latency, cloud devices do not send separate signaling messages. Instead, they encapsulate weight changes (differential information) directly in the video stream, such as in an RTP extension header, and transmit it along with the video stream. When the client's tile agent receives the video stream, it automatically parses the corresponding bandwidth weights and immediately adjusts the current bitrate based on its own texture complexity and other information. In this way, bandwidth allocation can be updated in milliseconds without centralized control. The system can complete bitrate switching before the user turns their head, ensuring a consistently clear and continuous picture.

[0122] S5: Local Cooperative Decision-Making by Agents

[0123] After receiving the bandwidth weights from S4, each tile's agent will independently complete a small-scale bandwidth allocation within its local neighborhood based on these weights and its own image complexity.

[0124] Specifically, each tile agent performs the following operations:

[0125] 1. Calculate effective demand

[0126] The agent multiplies its bandwidth weight by the texture complexity to obtain an "effective demand value." If the image is rich in detail or involves rapid motion, the effective demand for that tile will be higher, even with the same weight. This ensures that complex areas receive more bandwidth, preventing the image from becoming blurry.

[0127] 2. Neighborhood negotiation

[0128] Each tile only undergoes a simple "negotiation" process with its neighboring tiles. The system allocates a local bandwidth limit (quota) to each neighborhood, which is automatically derived from the global bandwidth allocation. If the total demand of all tiles in the neighborhood exceeds the quota, only the tile with the "highest effective demand" maintains its current bitrate, while the remaining tiles automatically drop one level to free up bandwidth. This process requires only one round of comparison and does not require repeated communication, resulting in extremely low latency.

[0129] 3. Determine local results

[0130] After the negotiation is completed, each agent will immediately determine its final bitrate level (e.g., high, medium, low) and output the result.

[0131] Through this "local collaboration within the neighborhood" mechanism, the system can complete bandwidth convergence within milliseconds. Moreover, the bitrate adjustment of each tile does not depend on central scheduling and does not cause long waiting times. As a result, within a time window of approximately 200 milliseconds before the user turns their head, tiles in the relevant areas of the view automatically maintain a high bitrate, while edge or non-interested areas automatically reduce the bitrate level, thereby achieving optimal bandwidth utilization.

[0132] S6: Global Validation and Unified Fine-tuning

[0133] After each tile agent completes its local collaborative decision-making, the system aggregates the results of all tiles in the cloud and checks whether the overall bitrate meets the session's bandwidth limit. Step S6 specifically includes:

[0134] 1. Summary and Verification

[0135] The cloud-based device collects the current bitrate of all tiles and calculates whether the total bitrate exceeds the bandwidth limit set for the session. If the total bitrate does not exceed the limit, the encoding stage begins directly; if it does exceed the limit, it indicates that the bitrate configuration of some tiles is too high and needs to be adjusted uniformly.

[0136] 2. Unified fine-tuning

[0137] When exceeding the limit is detected, the cloud device prioritizes reducing the bitrate of tiles with "low confidence but high bitrate consumption." These tiles are typically located at the edge of the viewport or in areas with unstable prediction. The cloud device will uniformly reduce their bitrate by, for example, half a stop to save bandwidth while keeping the main viewport unaffected.

[0138] 3. Smooth boundary

[0139] To avoid abrupt changes in image quality between tiles, cloud devices perform a slight "border smoothing" around the tiles that are downgraded. This prevents excessive bitrate differences between adjacent tiles, resulting in a more natural image transition and no obvious stitching artifacts.

[0140] After global verification and fine-tuning, the system ensures that the total bitrate of all tiles strictly meets the bandwidth limit, while ensuring that the image remains visually continuous and smooth, thus preparing for the final encoded output.

[0141] S7: Encoding and Transmission

[0142] After completing global verification and fine-tuning, the cloud device sends the final determined bitrate to the encoder. Based on the final bitrate of each tile, the encoder generates corresponding encoding parameters (such as quantization step size, layer selection, etc.), thereby outputting bitstreams of different resolutions for different tiles within the same frame.

[0143] To reduce redundant data, cloud devices employ a differential transmission mechanism, sending only the portions that have changed compared to the previous frame, while tiles with unchanged bitrates retain their original encoding settings. This significantly reduces signaling and bandwidth overhead.

[0144] In the final generated video stream, the bitrate of each tile is matched to its "level of attention":

[0145] - The tiles that are about to enter the field of view are kept in high quality (high bit rate).

[0146] - Automatically reduce bitrate in edge and non-interested areas;

[0147] - Total bandwidth is always kept within the preset limit.

[0148] When the user turns their head, the client device can immediately display the pre-optimized image, with a clear and smooth transition of vision, without any blurring or gaps.

[0149] The following description, using a VR basketball game livestream scenario as an example, illustrates the implementation of this application in order to provide a better understanding of its technical solution.

[0150] In the embodiments of this application, the cloud device is responsible for panoramic tile division, dynamic spatiotemporal graph construction, spatiotemporal graph neural network inference, and the generation and encapsulation of bandwidth weights, as well as transmitting, in real time, a bitstream carrying weight information, such as an RTP stream, to the client device via the network. The client device parses the weight information in the RTP extension header through a tile agent, performs neighborhood collaborative decision-making locally, generates a global quantization parameter table, and finally completes video rendering and display.

[0151] The overall goal of the VR system in this application is to achieve a clear, continuous, and low-latency video experience in scenarios where users are rapidly turning their heads, through forward prediction and multi-agent collaboration, under bandwidth-constrained conditions. The following provides a detailed description of each step of the method in this embodiment.

[0152] S1: Panoramic tiled layout:

[0153] At the start of a cloud-based VR session, the system unfolds the 360° panoramic video into a cube and divides it into several spherical tiles with fixed rows and columns. The system establishes adjacency relationships and unique indexes for each tile to form a topological structure that can be used for graph computation.

[0154] Subsequently, a lightweight agent is created for each tile to handle local bandwidth decisions and bitrate management. Each agent maintains its tile index, adjacency list, texture complexity metric, current tier, and local quota.

[0155] During the initialization phase, the system calculates the local bandwidth quota for each tile neighborhood based on the global bandwidth limit of the session, enabling subsequent bandwidth adjustments to converge quickly locally without the need for centralized scheduling.

[0156] S2: Dynamic Spatiotemporal Graph Construction:

[0157] During video playback, the cloud device updates the dynamic spatiotemporal graph every 33 milliseconds. Nodes in the graph correspond to spherical tiles, and edges represent spatial or kinematic relationships between tiles. The inputs for this step include:

[0158] (1) The user's head movement trajectory in the last 300 milliseconds;

[0159] (2) The significant motion characteristics of the current frame tile (such as the direction and amplitude of motion).

[0160] The system establishes "facing edges" based on head movement trajectories, representing the area that the user's gaze is about to sweep across; it establishes "movement edges" based on the consistency of movement of adjacent tiles, reflecting the continuity of the target's movement across tiles in the image; at the same time, it uses "look-ahead nodes" to aggregate potential high-attention tiles, providing a global view for subsequent inference.

[0161] S3: Spatiotemporal graph neural network inference:

[0162] The system inputs the aforementioned dynamic spatiotemporal graph into the spatiotemporal graph neural network model. Each tile node maintains two types of memory:

[0163] - Posture Flow Memory: Records the trend of changes in the user's head posture over time;

[0164] - Motion flow memory: Records the motion characteristics of the tile content.

[0165] During message transmission, attitude information is propagated only along the orientation edge, and motion information is propagated only along the motion edge to avoid redundant interference.

[0166] Subsequently, the system performs temporal aggregation and delayed fusion of the two types of memories within a fixed time window of approximately 200 milliseconds. Since the user's head movements typically precede changes in the image content, this delayed fusion mechanism can temporally align posture and motion information, improving prediction stability.

[0167] Finally, the model outputs the attention probability and confidence score of each tile, which is used to determine which tiles are most likely to enter the field of view in the next 200 milliseconds.

[0168] To ensure that the model has stable predictive capabilities when running online, the spatiotemporal graph neural network model in this embodiment is trained once using an offline dataset before deployment.

[0169] During the training phase, real VR viewing data is used to collect user head pose sequences and corresponding frame motion saliency maps, constructing dynamic spatiotemporal graph samples. The "whether the user enters the field of view within the next 200 ms" is used as the supervision signal. The model employs supervised learning for parameter optimization, without relying on the reward function and experience pool of reinforcement learning. After training, the model parameters are fixed (frozen) and deployed to the cloud. In actual operation, only forward inference is performed, and no further online updates are performed. This design significantly reduces system latency and computational burden, ensuring the real-time performance and stability of the prediction results.

[0170] S4: Bandwidth weight generation:

[0171] The cloud-based device generates bandwidth weights based on the attention probability output by the ST-GNN and the area ratio of the tiles in the viewport. To ensure that high-attention areas receive more bandwidth while preventing individual tiles from excessively consuming resources, the system uses a non-linear piecewise mapping function to compress the original demand values ​​into normalized bandwidth weights.

[0172] Weight changes are embedded differentially in the RTP extension header of the video stream and sent to the terminal in real time. This eliminates the need for additional signaling; the client's tile agent can instantly parse the weights and adjust the bitrate accordingly upon receiving the video, significantly reducing communication latency.

[0173] S5: Local Cooperative Decision-Making by Agents

[0174] Each tile agent on the terminal, after receiving the weights, calculates the "effective demand" based on its texture complexity. Within its neighborhood, agents perform a rapid local coordination: if the total demand in the neighborhood does not exceed the local quota, the current bitrate is maintained; if it exceeds the quota, only the tile with the highest effective demand is retained at its current level, and the remaining tiles are automatically downgraded by one level. The entire process requires no central scheduling and can be completed through a single comparison between neighbors, with latency typically within a few milliseconds.

[0175] S6: Global Validation and Unified Fine-tuning

[0176] The cloud system periodically summarizes the local decision results of each tile and verifies whether the total bitrate exceeds the global bandwidth limit. If the total bitrate exceeds the limit, the system prioritizes tiles with lower confidence and higher bandwidth usage for downgrading, and performs boundary smoothing on adjacent areas to prevent sudden changes in image quality.

[0177] After verification, the bitrate allocation of all tiles will strictly comply with the session bandwidth constraints, while maintaining the continuity and smoothness of the image.

[0178] S7: Encoding and Transmission

[0179] The encoder generates a bitstream based on the final determined bitrate level and sends only the part that is different from the previous frame (differential transmission), further reducing the transmission load.

[0180] In the final generated video stream, tiles within the viewport remain in high quality, while non-viewport areas are automatically downgraded. When watching the live stream, even if users turn their heads quickly, they can still see a clear, pre-optimized image with natural transitions and no obvious blurring or breaks.

[0181] In VR basketball live streaming scenarios, the method of this invention can complete the bitrate adjustment about 200 milliseconds before the user turns their head, so that the relevant areas of the field of view always maintain a high bitrate and the edge areas transition smoothly.

[0182] In addition to the above-described technical solutions, the present invention also provides an electronic device comprising one or more processors and a memory for storing executable instructions. The one or more processors are configured to implement the above-described method via executable instructions. The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to perform the above-described method. The electronic device can be a cloud device, such as a server device, or a client device, such as VR glasses.

[0183] The following sections of this specification will be combined with Figure 3 Illustrative examples are provided to describe the aforementioned electronic devices and computer-readable storage media.

[0184] Figure 3 An example configuration of an electronic device 300 that can be used to implement the methods described in this invention is shown. The technical solutions of this invention can also be implemented wholly or at least partially by the electronic device 300 or similar devices / systems. The electronic device 300 can be of various different types. Examples of the electronic device 300 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices, wearable devices, entertainment devices, televisions or other display devices, automotive computers, etc.

[0185] Electronic device 300 may include at least one processor 302, memory 304, multiple communication interfaces 309, display device 301, other input / output (I / O) devices 310, and one or more mass storage devices 303 capable of communicating with each other via bus 311 or other suitable connection.

[0186] Processor 302 may be one or more processing units, and all processing units may include one or more computing units or multiple cores. Processor 302 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 302 may be configured to retrieve and execute computer-readable instructions stored in memory 304, mass storage device 303, or other computer-readable media, such as program code of operating system 305, application program 306, or other program 307.

[0187] Memory 304 and mass storage device 303 are examples of computer-readable storage media for storing instructions that are executed by processor 302 to perform the various functions described above. For example, memory 304 can generally include both volatile and non-volatile memory. Furthermore, mass storage device 303 can generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks, storage arrays, network-attached storage, storage area networks, etc. Memory 304 and mass storage device 303 can both be collectively referred to as memory or computer-readable storage media in this invention, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which can be executed by processor 302 as a specific machine configured to perform the operations and functions described in the examples of this invention.

[0188] Multiple programs can be stored on mass storage device 303. These programs include operating system 305, one or more application programs 306, other programs 307, and program data 308, and they can be loaded into memory 304 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: the methods provided by the present invention (including any suitable steps of the method) and / or other embodiments described in the present invention.

[0189] Although shown in the figure as stored in the memory 304 of the electronic device 300, the modular operating system 305, application program 306, other programs 307, and program data 308, or portions thereof, can be implemented using any form of computer-readable medium accessible by the electronic device 300. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer. A communication medium includes a medium through which communication signals, such as computer-readable instructions, data structures, program modules, or other data, are transmitted from one system to another. Communication media can include guided transmission media and wireless media capable of propagating energy waves. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals, for example, in a wireless medium.

[0190] For example, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other known media or those developed hereafter capable of storing computer-readable information / data for use by a computer system.

[0191] One or more communication interfaces 309 are used to exchange data with other devices via means such as a network or direct connection. This communication interface can be one or more of the following: any type of network interface, wired or wireless (such as WLAN) interface, Wi-MAX interface, Ethernet interface, USB interface, cellular network interface, Bluetooth interface, NFC interface, etc. Communication interface 309 can facilitate communication across various network and protocol types, including wired and wireless networks, the Internet, etc. Communication interface 309 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0192] In some examples, a display device 301, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 310 may be devices that receive user input and provide output to the user, and may include touch / gesture input devices, cameras, keyboards, remote controls, mice, audio input / output devices, etc.

[0193] The technical solutions described in this invention can be supported by various configurations of the electronic device 300, and are not limited to the specific examples of the technical solutions described in this invention. The foregoing description and illustrations of this invention are not restrictive. It will be apparent to those skilled in the art that this invention is not limited to the details of the exemplary embodiments described above, and that this invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the scope of protection claimed by this invention is defined by the claims rather than the foregoing description, and all variations falling within the meaning and scope of equivalents of the claims are covered within the scope of protection of this invention.

Claims

1. An adaptive code rate allocation method based on a spatiotemporal graph neural network, characterized in that, Includes the following steps: S1: Panoramic tileization step, including unfolding the VR video into a cube, cutting it into a set of spherical tiles, and establishing an adjacency list of each tile and its surrounding tiles; A tile agent is created for each tile, and the tile agent stores the following information: tile ID, neighboring tile IDs, current actual bitrate, current complexity index, and current bitrate quota; S2: Dynamic spatiotemporal graph construction steps, including constructing a spatiotemporal graph that changes over time based on the user's real-time head posture and the content of the VR video. The spatiotemporal graph includes tile nodes, orientation edges, motion edges, and virtual nodes. Tile nodes represent tiles, orientation edges represent the relationship of the gaze sweep, motion edges represent the motion continuity relationship across tile content, and virtual nodes represent tiles that may enter the field of view. S3: Spatiotemporal graph neural network inference steps, including inputting the dynamic spatiotemporal graph into the spatiotemporal graph neural network to obtain the attention probability and confidence of each tile; S4: Bandwidth weight generation step, including generating the bandwidth weight of each tile based on the attention probability of each tile; S5: Local cooperative decision-making steps of the agents, including each tile agent calculating the effective required bit rate of the tile based on the tile's bandwidth weight and texture complexity; S6: Global verification and unified fine-tuning steps, including verifying whether the total bitrate exceeds the global bandwidth limit based on the effective required bitrate of each tile; if so, adjusting the effective required bitrate of each tile to determine the final bitrate of each tile. S7: Encoding and transmission steps: Encode the VR video according to the final bitrate of each tile, and send the encoded video stream to the client device.

2. The method according to claim 1, characterized in that, In step S2, the user's real-time head posture is represented by a head posture sequence, which describes the user's head turning trajectory over a recent period of time.

3. The method according to claim 2, characterized in that, Step S2 includes: Based on the head posture sequence, determine the tiles that the user's line of sight may sweep over in the future, and establish orientation edges from the current center tile of the field of vision to the tiles that the user's line of sight may sweep over in the future. Based on the significant motion characteristics between adjacent tiles, establish motion edges between adjacent tiles; The system aggregates the tiles that the user's gaze may sweep over within a certain period of time in the future, forming virtual nodes, and feeds the aggregation results back to each tile's intelligent agent.

4. The method according to claim 2, characterized in that, Step S3 includes: injecting the head pose sequence and the motion salient features of each tile into each tile agent.

5. The method according to claim 4, characterized in that, Step S3 also includes: The information propagation and feature aggregation steps include, for oriented edges, only propagating posture information related to the user's line of sight; for motion edges, only propagating information related to content motion. The delayed fusion step includes aggregating the attitude flow and motion flow separately, and then fusing them at the end of the window to generate a fused feature vector, which represents the overall attention intensity of the tile in the future. The virtual node aggregation and input steps include injecting the fused feature vector into the virtual node and outputting the attention probability and confidence of each tile.

6. The method according to claim 5, characterized in that, Step S4 includes: The bandwidth control parameters for each tile are determined based on the attention probability of each tile. Bandwidth control parameters are transmitted to the tile agent via video stream.

7. The method according to claim 6, characterized in that, The bandwidth control parameters include bandwidth weights, where w is the bandwidth weight of the i-th tile. i Determined by the following nonlinear piecewise function: Where pi is the attention probability of the i-th tile, and T1 and T2 are the low threshold and high threshold, respectively. β is the scaling factor.

8. The method according to claim 1, characterized in that, Step S5 includes: Multiply the bandwidth weight of each tile by the texture complexity to obtain the effective demand value; Calculate the total demand value of tiles in the neighborhood. If the total demand value is higher than the upper limit allocated by the system, maintain the effective demand code rate of the tile with the highest effective demand value and reduce the effective demand code rate of other tiles.

9. The method according to claim 1, characterized in that, In step S6, if the total bitrate exceeds the global bandwidth limit, the bitrate of tiles with low confidence and high effective demand bitrate is reduced first, and boundary smoothing is performed around the tiles whose bitrate is reduced.

10. The method according to claim 1, characterized in that, Step S7 includes: sending the encoded video stream to the client device using a differential transmission mechanism.

11. A VR system, comprising a cloud server and client devices, characterized in that, The VR system is configured to perform the method according to any one of claims 1 to 10.

Images