360-degree video stream adaptive transmission method based on saliency perception and viewport prediction
By introducing saliency perception and viewport prediction into 360-degree video streaming, a joint utility evaluation model is constructed, which solves the problem of high bandwidth consumption in viewport adaptive transmission and achieves clarity assurance and bandwidth saving in high saliency areas.
Patent Information
- Application Number
- CN202511865441.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-02-24
AI Technical Summary
Existing 360-degree video streaming methods suffer from coarse-grained bitrate allocation and perceptual redundancy in viewport adaptive schemes, resulting in high bandwidth consumption and poor user experience, especially since the differences in visual importance of different areas within the viewport are not fully considered.
An adaptive transmission method based on saliency perception and viewport prediction is adopted. By combining the spatial location weight of slices, saliency factor and quality benefit index, a joint utility evaluation model is constructed to refine the transmission priority ranking and bit rate decision, so as to ensure the clarity of high saliency areas and reduce bandwidth requirements.
Under limited bandwidth conditions, it significantly reduces transmission bandwidth consumption while ensuring clarity for users in highly salient areas, achieving an optimized balance between transmission quality and subjective experience.
Smart Images
Figure CN121567894A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of virtual reality, video streaming transmission and image processing technology, and more specifically, to a 360-degree video stream adaptive transmission method based on saliency perception and viewport prediction. Background Technology
[0002] With the popularization of virtual reality (VR) technology, 360-degree panoramic videos have attracted much attention due to their immersive interactive experience. However, in order to provide an immersive experience, 360-degree videos require extremely high resolution (such as 8K or even higher), and their bitrate is usually 4 to 6 times that of planar videos of the same quality, which poses a serious challenge to existing network bandwidth.
[0003] To reduce bandwidth consumption, current mainstream technologies employ a tile-based viewport adaptive transmission scheme. This scheme divides the panoramic video into tiles and, based on predictions of the user's future viewport, allocates high bitrates to tiles within the predicted viewport and low bitrates to tiles outside the viewport.
[0004] However, existing viewport adaptive solutions still suffer from "coarse granularity" and "perceptual redundancy" in bitrate allocation: typically, the system uniformly sets all slices within the predicted viewport to high resolution. But in reality, even within the user's viewport, the visual importance of different areas varies significantly. For example, flat background areas like the sky and grass within the viewport, or edge areas swept by the user's peripheral vision, have negligible impact on the user's subjective visual experience (QoE), even if their image quality (resolution or bitrate) is reduced. If these "inefficient" slices within the viewport could be proactively downgraded for transmission based on slice distortion effects (such as PSNR) and positional probabilities, bandwidth costs could be significantly reduced.
[0005] However, relying solely on objective metrics (such as PSNR or geometric location) for "viewport slice downgrading" carries a significant risk of misjudgment. For example, some slices located at the viewport edge, or slices with low texture complexity resulting in low PSNR calculation weights, may contain salient objects that users are highly interested in (such as suddenly appearing people, text, or moving targets). If these slices are judged as "low-value" and forcibly downgraded based solely on the objective utility formula, once the user's attention is drawn to these salient contents, obvious blurring or blockiness will be observed, severely disrupting the immersive experience.
[0006] To address the aforementioned issues, this invention proposes an adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction. This method, while utilizing position probability and distortion impact (PSNR) for aggressive bitrate allocation to save bandwidth, innovatively introduces a visual saliency mechanism as a correction factor. By incorporating saliency weights into the transmission decision model, this invention can accurately identify slices within the viewport that, although having low geometric weights or low objective distortion gains, contain content of high user interest, and prioritize their transmission to prevent them from being incorrectly downgraded to low resolution. This invention fully exploits the bandwidth compression potential of non-saliency areas within the viewport while effectively ensuring the clarity of the user's focus, achieving an optimal balance between bandwidth saving and subjective immersive experience. Summary of the Invention
[0007] This invention provides an adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction. This method constructs a joint utility evaluation model by combining information such as the spatial location weight of slices, saliency factors, and quality benefit indicators, and performs refined transmission priority ranking and bitrate decision-making for slices, thereby significantly reducing the bandwidth required for transmission while ensuring the user's viewing experience in high-saliency areas.
[0008] To achieve the above objectives, the present invention adopts the following technical solution.
[0009] An adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction includes the following steps:
[0010] 1) The server acquires 360-degree panoramic video, performs equidistant cylindrical projection (ERP) and slices it, and generates encoded slices with different bitrate gradients.
[0011] 2) The server performs saliency detection on the 360-degree panoramic video described in 1), generates a corresponding saliency map, calculates the saliency factor of each slice, and calculates the quality gain index of each slice at different bitrates.
[0012] 3) The client obtains a video description file, which includes slice division information, the significance factor, and the quality benefit index;
[0013] 4) The client collects head posture data in real time during video playback and uses a viewport prediction model to predict the viewport center position of the user within a preset time period in the future.
[0014] 5) Based on the viewport center position predicted in 4), the client calculates the spatial position weight of each slice relative to the viewport center;
[0015] 6) The client uses a joint utility formula to calculate the transmission priority score of the slice, which is constructed based on spatial location weights, significance factors, and quality benefit indicators;
[0016] 7) Based on the transmission priority score in 6), the client selects the optimal bitrate for each slice and initiates a transmission request to the server, under the constraint of the currently available network bandwidth.
[0017] Preferably, in step 1), the server acquires a 360-degree panoramic video, performs equidistant cylindrical projection (ERP) and slice division on it, and generates encoded slices with different bitrate gradients, including:
[0018] The spherical coordinates of the 360-degree panoramic video are obtained by using the equidistant cylindrical projection (ERP) algorithm. Mapped to two-dimensional Cartesian coordinates :
[0019]
[0020] in, Representing longitude and latitude in a spherical coordinate system, This represents the horizontal and vertical pixel coordinates on the projected two-dimensional planar image. and These represent the width and height resolutions of the ERP projected planar image, respectively.
[0021] On the ERP projection plane, the video frames are spatially discretized into 12 rows and 12 columns of rectangular grid slices. For the frame located in the row... , No. The geometric coverage area of a slice of a column on the projection plane is represented by the following formula:
[0022]
[0023] in, Slice The set of pixel coordinates on a two-dimensional plane and These represent the row index and column index of the slice, respectively. and These represent the width and height in pixels for a single slice, respectively.
[0024] In terms of time, the video stream is divided into fixed-duration 1-second video segments. For each slice within each video segment, multi-bitrate encoding is performed using the video coding standard. Let the preset set of bitrate levels be... , This represents the total number of bitrate levels, and satisfies the following conditions: For any slice Server generated The system generates encoded data streams of different quality versions, calculates the average bitrate and video quality metrics for each version, and encapsulates this metadata into a Media Description File (MPD) for client download decisions.
[0025] Preferably, in step 2), the server performs saliency detection on the 360-degree panoramic video, generates a corresponding saliency map, calculates the saliency factor for each slice, and calculates the quality gain index for each slice at different bitrates, including:
[0026] The Itti-Koch visual attention model is used to extract features from video frames. First, a Gaussian pyramid is constructed from the input image to extract three low-level visual features: color, intensity, and orientation. Then, a center-surround difference operation is used to generate multi-scale feature maps, which are further fused to obtain three saliency maps: a color saliency map, an intensity saliency map, and an orientation saliency map. Finally, these three saliency maps are fused using a linear combination method to generate the final pixel-level total saliency map. :
[0027]
[0028] in, This represents the generated total saliency map, where the value of each pixel represents the degree to which that location attracts visual attention; This represents a color saliency map calculated based on the red-green and blue-yellow complementary colors. This represents a saliency map calculated based on luminance contrast. This represents the directional saliency map calculated based on the Gabor filter's response at different angles.
[0029] For index The slice, its significance factor Defined as the mean of the saliency values of all pixels within the area covered by this slice:
[0030]
[0031] in, Indicates the first The significance factor of each slice, with values normalized to [value range missing]. ; Indicates the first The pixel coordinate region of each slice on the saliency map; and These represent the width and height in pixels for a single slice, respectively. Representing coordinates The saliency value of the pixel at that location.
[0032] Calculate slices In bitrate The mean square error (MSE) between the reconstructed image and the original undamaged image is then obtained. As an objective indicator for evaluating the quality and benefits of slices:
[0033]
[0034] in, This indicates the color depth (bit depth) of the video. Indicates the slice at bitrate The mean square error of pixels below.
[0035] Preferably, in step 3), the client obtains a video description file, which includes slice division information, the saliency factor, and the quality gain index, including:
[0036] The client retrieves the media description file for the current video segment via an HTTP request. This file is parsed, global segmentation parameters are extracted, and then applied to each segment... The client then parses out the custom metadata it contains. The client then reassembles the parsed data into a standardized mathematical structure.
[0037]
[0038] in, Indicates the first A tuple of attributes for each slice. This represents the significance factor of the slice. This indicates how high the fidelity of the slice is compared to the original undamaged slice.
[0039] Preferably, in step 4), the client collects head posture data in real time during video playback and uses a viewport prediction model to predict the user's viewport center position within a preset time period in the future, including:
[0040] The client collects user head posture data at a fixed sampling frequency, mainly including yaw and pitch angles. Due to the spherical topology of 360-degree video, there is a numerical break in the yaw angle at 0 degrees and 360 degrees. To ensure the accuracy of the subsequent linear regression model, the periodic jumps in angle must be eliminated, and the spherical coordinates must be transformed into continuous linear coordinates. Let... The raw yaw angle collected at each moment is Its corresponding continuous yaw angle The recursive calculation formula is:
[0041]
[0042] in, express The raw yaw angle collected by the time sensor represents the raw yaw angle at the previous sampling time. This represents the original yaw angle at the previous sampling time. This represents the cumulative number of rotation cycles at any given moment.
[0043] Select the most recent Continuous coordinate sequence of sampling points As input, a linear prediction model about time is established. Estimate the slope of the line using the least squares method:
[0044]
[0045] in, Indicates the timestamp of historical sampling points. This represents the continuous yaw angle corresponding to the historical sampling points. This indicates the length of the historical window used for prediction.
[0046] Based on the calculated regression parameters and Predict the future preset duration The continuous coordinates of the viewport center are then mapped back to the standard spherical coordinate system to obtain the final predicted yaw angle. :
[0047]
[0048] in, Indicates the predicted future yaw angle of the viewport center. Represents the continuous yaw angle at the current moment. This indicates that a modulo operation is performed on 360 to ensure that the result falls within the range of 360. Within the range.
[0049] Similar to yaw angle, the client uses the least squares method to calculate the nearest... Elevation angle sequence of sampling points Perform linear fitting.
[0050] Because the physical range of pitch angle values is usually limited by There is no periodic crossing problem, so a prediction model can be directly established:
[0051]
[0052] in, This represents a linear prediction function of the pitch angle with respect to time. This represents the rate of change of the pitch angle. This represents the intercept of the fitted line. and This represents the timestamps and elevation angle data of historical sampling points.
[0053] Based on the calculation and Predict the future preset duration Pitch angle of the rear viewport center :
[0054]
[0055] in, Indicates the predicted future yaw angle of the viewport center. This represents the continuous yaw angle at the current moment.
[0056] Based on the above calculations, the client determines the user's future rendering time. spherical coordinates of the viewport center .
[0057] Preferably, in step 5), the client calculates the spatial position weight of each slice relative to the viewport center based on the predicted viewport center position, including:
[0058] Since a video slice is 1 second long and contains multiple frames, the predicted viewport coordinates at the midpoint of this time interval are selected as the average viewport center for that slice's time interval. According to the ERP projection rules, determine the slice index where the average viewport center is located. ,in Indicates row index, Indicates column index.
[0059] Then, the distance between any slice and the center slice on the 2D grid is calculated using the Chebyshev distance, and corrections are made for the horizontal wraparound characteristics of 360-degree video.
[0060]
[0061] in, To account for the minimum horizontal distance after wrapping around the left and right boundaries, and For the row and column indices of the current slice, and For the row and column indices of the centered slice, The total number of columns to divide the slice. For the position located at the line, number Spatial index distance between the slice of a column and the center slice.
[0062] Based on the calculated distance, the slice is divided into three discrete priority levels and assigned corresponding spatial location weights:
[0063]
[0064] in, For slices Spatial location weights.
[0065] Preferably, in step 6), the client calculates the transmission priority score of the slice using a joint utility formula, which is constructed based on spatial location weights, significance factors, and quality benefit indicators, including:
[0066]
[0067] in, Slice In bitrate Transmission priority score under each gear level Slice Spatial location weights, Slice The significance factor, This represents the significance adjustment coefficient, used to adjust the ratio between the geometric weight and the significance weight. Slice In bitrate The quality and benefit indicators below.
[0068] Preferably, in step 6), the client, based on the transmission priority score and within the constraint of currently available network bandwidth, selects the optimal bitrate for each slice and initiates a transmission request to the server, including:
[0069] A greedy algorithm is used for iterative decision-making. First, the initial bitrate of all slices in the current video segment is set to the lowest level. Then, for each slice that has not yet reached the highest bitrate level... Assuming its bitrate is reduced from the current level Upgrade to the next level Calculate the marginal utility density brought about by this lifting operation. :
[0070]
[0071] in, The joint utility score calculated in step 6) is... This represents the average bit rate for the corresponding gear.
[0072] Select from all candidate slices slice with the largest value As the optimal upgrade target for this iteration, it is also checked whether the total bitrate of all slices exceeds the current estimated available network bandwidth if the above upgrade operation is performed. If the constraint is met, the slice is confirmed to be upgraded. The bitrate level has been increased to Update the total bitrate consumption and continue to the next iteration; If the constraints are not met, or if all slices have reached the highest bitrate, then the iteration stops.
[0073] Based on the final determined bitrate of each slice, the client constructs an HTTP request list and requests the server to download the corresponding media data segments.
[0074] This invention proposes an adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction. This method innovatively introduces a visual saliency mechanism as a correction factor into the traditional viewport adaptive transmission framework, constructing a three-dimensional evaluation system of "geometric position + visual content + encoding quality." Through the joint decision-making of discretized spatial grading and a greedy algorithm, it effectively solves the transmission quality bottleneck problem of 360-degree panoramic video in network bandwidth-constrained scenarios, maximizing the user's subjective experience (QoE) under limited bandwidth conditions. Attached Figure Description
[0075] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly explained below. The accompanying drawings described herein are merely some embodiments of the present invention.
[0076] Figure 1 This is a step diagram of the present invention. Detailed Implementation
[0077] To clarify the technical issues, technical solutions, implementation process, and performance demonstration, the present invention will be further described in detail below with reference to embodiments. Note. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The following will refer to the accompanying drawings. Various exemplary embodiments, features, and aspects of this disclosure are described in detail. The same reference numerals in the accompanying drawings denote elements that have the same or similar functions.
[0078] Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0079] This invention provides an adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction. This method constructs a joint utility evaluation model by combining information such as the spatial location weight of slices, saliency factors, and quality benefit indicators, and performs refined transmission priority ranking and bitrate decision-making for slices, thereby significantly reducing the bandwidth required for transmission while ensuring the user's viewing experience in high-saliency areas.
[0080] To achieve the above objectives, the present invention adopts the following technical solution.
[0081] An adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction includes the following steps:
[0082] 1) In step S1 of Figure 1, a 360-degree panoramic video is acquired, and it is subjected to equidistant cylindrical projection (ERP) and sliced to generate encoded slices with different bitrate gradients.
[0083] 2) In step S2 of Figure 1, saliency detection is performed on the 360-degree panoramic video described in step S1 to generate a corresponding saliency map. Based on this, the saliency factor of each slice is calculated, and the quality gain index of each slice at different bitrates is calculated.
[0084] 3) In step S3 of Figure 1, a video description file is obtained, which includes slice division information, the significance factor, and the quality benefit index;
[0085] 4) In step S4 of Figure 1, head posture data is collected in real time during video playback, and the viewport prediction model is used to predict the viewport center position of the user within a preset time period in the future.
[0086] 5) In step S5 of Figure 1, the spatial position weight of each slice relative to the viewport center is calculated based on the viewport center position predicted in step S4.
[0087] 6) In step S6 of Figure 1, the transmission priority score of the slice is calculated using the joint utility formula, which is constructed based on spatial location weights, significance factors, and quality benefit indicators.
[0088] 7) In step S7 of Figure 1, based on the transmission priority score in step S6, under the constraint of the currently available network bandwidth, the optimal bitrate level is selected for each slice and a transmission request is initiated to the server.
[0089] Preferably, in step 1), the server acquires a 360-degree panoramic video, performs equidistant cylindrical projection (ERP) and slice division on it, and generates encoded slices with different bitrate gradients, including:
[0090] The spherical coordinates of the 360-degree panoramic video are obtained by using the equidistant cylindrical projection (ERP) algorithm. Mapped to two-dimensional Cartesian coordinates :
[0091]
[0092] in, Representing longitude and latitude in a spherical coordinate system, This represents the horizontal and vertical pixel coordinates on the projected two-dimensional planar image. and These represent the width and height resolutions of the ERP projected planar image, respectively.
[0093] On the ERP projection plane, the video frames are spatially discretized into 12 rows and 12 columns of rectangular grid slices. For the frame located in the row... , No. The geometric coverage area of a slice of a column on the projection plane is represented by the following formula:
[0094]
[0095] in, Slice The set of pixel coordinates on a two-dimensional plane and These represent the row index and column index of the slice, respectively. and These represent the width and height in pixels for a single slice, respectively.
[0096] In terms of time, the video stream is divided into fixed-duration 1-second video segments. For each slice within each video segment, multi-bitrate encoding is performed using the video coding standard. Let the preset set of bitrate levels be... , This represents the total number of bitrate levels, and satisfies the following conditions: For any slice Server generated The system generates encoded data streams of different quality versions, calculates the average bitrate and video quality metrics for each version, and encapsulates this metadata into a Media Description File (MPD) for client download decisions.
[0097] Preferably, in step 2), the server performs saliency detection on the 360-degree panoramic video, generates a corresponding saliency map, calculates the saliency factor for each slice, and calculates the quality gain index for each slice at different bitrates, including:
[0098] The Itti-Koch visual attention model is used to extract features from video frames. First, a Gaussian pyramid is constructed from the input image to extract three low-level visual features: color, intensity, and orientation. Then, a center-surround difference operation is used to generate multi-scale feature maps, which are further fused to obtain three saliency maps: a color saliency map, an intensity saliency map, and an orientation saliency map. Finally, these three saliency maps are fused using a linear combination method to generate the final pixel-level total saliency map. :
[0099]
[0100] in, This represents the generated total saliency map, where the value of each pixel represents the degree to which that location attracts visual attention; This represents a color saliency map calculated based on the red-green and blue-yellow complementary colors. This represents a saliency map calculated based on luminance contrast. This represents the directional saliency map calculated based on the Gabor filter's response at different angles.
[0101] For index The slice, its significance factor Defined as the mean of the saliency values of all pixels within the area covered by this slice:
[0102]
[0103] in, Indicates the first The significance factor of each slice, with values normalized to [value range missing]. ; Indicates the first The pixel coordinate region of each slice on the saliency map; and These represent the width and height in pixels for a single slice, respectively. Representing coordinates The saliency value of the pixel at that location.
[0104] Calculate slices In bitrate The mean square error (MSE) between the reconstructed image and the original undamaged image is then obtained. As an objective indicator for evaluating the quality and benefits of slices:
[0105]
[0106] in, This indicates the color depth (bit depth) of the video. Indicates the slice at bitrate The mean square error of pixels below.
[0107] Preferably, in step 3), the client obtains a video description file, which includes slice division information, the saliency factor, and the quality gain index, including:
[0108] The client retrieves the media description file for the current video segment via an HTTP request. This file is parsed, global segmentation parameters are extracted, and then applied to each segment... The client then parses out the custom metadata it contains. The client then reassembles the parsed data into a standardized mathematical structure.
[0109]
[0110] in, Indicates the first A tuple of attributes for each slice. This represents the significance factor of the slice. This indicates how high the fidelity of the slice is compared to the original undamaged slice.
[0111] Preferably, in step 4), the client collects head posture data in real time during video playback and uses a viewport prediction model to predict the user's viewport center position within a preset time period in the future, including:
[0112] The client collects user head posture data at a fixed sampling frequency, mainly including yaw and pitch angles. Due to the spherical topology of 360-degree video, there is a numerical break in the yaw angle at 0 degrees and 360 degrees. To ensure the accuracy of the subsequent linear regression model, the periodic jumps in angle must be eliminated, and the spherical coordinates must be transformed into continuous linear coordinates. Let... The raw yaw angle collected at each moment is Its corresponding continuous yaw angle The recursive calculation formula is:
[0113]
[0114] in, express The raw yaw angle collected by the time sensor represents the raw yaw angle at the previous sampling time. This represents the original yaw angle at the previous sampling time. This represents the cumulative number of rotation cycles at any given moment.
[0115] Select the most recent Continuous coordinate sequence of sampling points As input, a linear prediction model about time is established. Estimate the slope of the line using the least squares method:
[0116]
[0117] in, Indicates the timestamp of historical sampling points. This represents the continuous yaw angle corresponding to the historical sampling points. This indicates the length of the historical window used for prediction.
[0118] Based on the calculated regression parameters and Predict the future preset duration The continuous coordinates of the viewport center are then mapped back to the standard spherical coordinate system to obtain the final predicted yaw angle. :
[0119]
[0120] in, Indicates the predicted future yaw angle of the viewport center. Represents the continuous yaw angle at the current moment. This indicates that a modulo operation is performed on 360 to ensure that the result falls within the range of 360. Within the range.
[0121] Similar to yaw angle, the client uses the least squares method to calculate the nearest... Elevation angle sequence of sampling points Perform linear fitting.
[0122] Because the physical range of pitch angle values is usually limited by There is no periodic crossing problem, so a prediction model can be directly established:
[0123]
[0124] in, This represents a linear prediction function of the pitch angle with respect to time. This represents the rate of change of the pitch angle. This represents the intercept of the fitted line. and This represents the timestamps and elevation angle data of historical sampling points.
[0125] Based on the calculation and Predict the future preset duration Pitch angle of the rear viewport center :
[0126]
[0127] in, Indicates the predicted future yaw angle of the viewport center. This represents the continuous yaw angle at the current moment.
[0128] Based on the above calculations, the client determines the user's future rendering time. spherical coordinates of the viewport center .
[0129] Preferably, in step 5), the client calculates the spatial position weight of each slice relative to the viewport center based on the predicted viewport center position, including:
[0130] Since a video slice is 1 second long and contains multiple frames, the predicted viewport coordinates at the midpoint of this time interval are selected as the average viewport center for that slice's time interval. According to the ERP projection rules, determine the slice index where the average viewport center is located. ,in Indicates row index, Indicates column index.
[0131] Then, the distance between any slice and the center slice on the 2D grid is calculated using the Chebyshev distance, and corrections are made for the horizontal wraparound characteristics of 360-degree video.
[0132]
[0133] in, To account for the minimum horizontal distance after wrapping around the left and right boundaries, and For the row and column indices of the current slice, and For the row and column indices of the centered slice, The total number of columns to divide the slice. For the position located at the line, number Spatial index distance between the slice of a column and the center slice.
[0134] Based on the calculated distance, the slice is divided into three discrete priority levels and assigned corresponding spatial location weights:
[0135]
[0136] in, For slices Spatial location weights.
[0137] Preferably, in step 6), the client calculates the transmission priority score of the slice using a joint utility formula, which is constructed based on spatial location weights, significance factors, and quality benefit indicators, including:
[0138]
[0139] in, Slice In bitrate Transmission priority score under each gear level Slice Spatial location weights, Slice The significance factor, This represents the significance adjustment coefficient, used to adjust the ratio between the geometric weight and the significance weight. Slice In bitrate The quality and benefit indicators below.
[0140] Preferably, in step 6), the client, based on the transmission priority score and within the constraint of currently available network bandwidth, selects the optimal bitrate for each slice and initiates a transmission request to the server, including:
[0141] A greedy algorithm is used for iterative decision-making. First, the initial bitrate of all slices in the current video segment is set to the lowest level. Then, for each slice that has not yet reached the highest bitrate level... Assuming its bitrate is reduced from the current level Upgrade to the next level Calculate the marginal utility density brought about by this lifting operation. :
[0142]
[0143] in, The joint utility score calculated in step 6) is... This represents the average bit rate for the corresponding gear.
[0144] Select from all candidate slices slice with the largest value As the optimal upgrade target for this iteration, it is also checked whether the total bitrate of all slices exceeds the current estimated available network bandwidth if the above upgrade operation is performed. If the constraint is met, the slice is confirmed to be upgraded. The bitrate level has been increased to Update the total bitrate consumption and continue to the next iteration; If the constraints are not met, or if all slices have reached the highest bitrate, then the iteration stops.
[0145] Based on the final determined bitrate of each slice, the client constructs an HTTP request list and requests the server to download the corresponding media data segments.
[0146] This invention proposes an adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction. This method innovatively introduces a visual saliency mechanism as a correction factor into the traditional viewport adaptive transmission framework, constructing a three-dimensional evaluation system of "geometric position + visual content + encoding quality." Through the joint decision-making of discretized spatial grading and a greedy algorithm, it effectively solves the transmission quality bottleneck problem of 360-degree panoramic video in network bandwidth-constrained scenarios, maximizing the user's subjective experience (QoE) under limited bandwidth conditions.
Claims
1. A method for adaptive transmission of 360-degree video streams based on saliency perception and viewport prediction, characterized in that, Includes the following steps: 1) The server acquires 360-degree panoramic video, performs equidistant cylindrical projection (ERP) and slices it, and generates encoded slices with different bitrate gradients. 2) The server performs saliency detection on the 360-degree panoramic video, generates a corresponding saliency map, calculates the saliency factor of each slice, and calculates the quality gain index of each slice at different bitrates. 3) The client obtains a video description file, which includes slice division information, the significance factor, and the quality benefit index; 4) The client collects head posture data in real time during video playback and uses a viewport prediction model to predict the viewport center position of the user within a preset time period in the future. 5) The client calculates the spatial position weight of each slice relative to the viewport center based on the predicted viewport center position; 6) The client uses a joint utility formula to calculate the transmission priority score of the slice, which is constructed based on spatial location weights, significance factors, and quality benefit indicators; 7) Based on the transmission priority score, and under the constraint of the currently available network bandwidth, the client selects the optimal bitrate for each slice and initiates a transmission request to the server.
2. The 360-degree video stream adaptive transmission method based on saliency perception and viewport prediction according to claim 1, characterized in that, The server performs saliency detection on the 360-degree panoramic video, generates a corresponding saliency map, calculates the saliency factor for each slice, and calculates the quality gain index for each slice at different bitrates. This includes: extracting pixel-level saliency probability maps of video frames using the Itti-Koch saliency detection model, calculating the mean of the saliency probabilities of all pixels within the slice region, and using the normalized result as the saliency factor for that slice; calculating the peak signal-to-noise ratio (PSNR) of the slice at each coding bitrate, and using it as the quality gain index.
3. The 360-degree video stream adaptive transmission method based on saliency perception and viewport prediction according to claim 1, characterized in that, The aforementioned calculation of the precise viewport center at the future rendering time based on the latest head pose data within a preset time window before video frame rendering includes: the viewport prediction model adopts a linear regression algorithm, with the input being the Euler angle coordinates of the past L sampling points and the output being the viewport center coordinates after a future time interval ΔT.
4. The adaptive transmission method for 360-degree video streams based on saliency perception and viewport prediction according to claim 1, characterized in that, The client uses a joint utility formula to calculate the transmission priority score of the slice. The joint utility formula is constructed based on spatial location weight, significance factor and quality benefit index. Specifically, the joint utility formula is as follows: first, the spatial location weight is calculated by adding the product of significance factor and significance adjustment coefficient to obtain the corrected location weight; then, the corrected location weight is multiplied by the quality benefit index to obtain the transmission priority score.
5. The 360-degree video stream adaptive transmission method based on saliency perception and viewport prediction according to claim 1, characterized in that, The client, based on the transmission priority score and under the constraint of the currently available network bandwidth, selects the optimal bitrate tier for each slice and initiates a transmission request to the server. This includes: using a greedy algorithm to select the optimal bitrate tier for each slice. Specifically, the steps include: using a greedy algorithm, while attempting to increase the tier of high-priority slices in ascending bitrate order to maximize the overall utility, detecting whether the cumulative transmission delay caused by the increase operation leads to the failure of subsequent slices in the request queue. If the failure occurs, the subsequent slices are either downgraded in bitrate or removed from the queue.