Adaptive streaming media method based on content saliency

By calculating the significance value of the video block and combining network conditions and player status, bit rate decisions are made using a linear user experience quality model, which solves the limitations of the existing adaptive bit rate algorithm in terms of resource utilization and video service quality. It has achieved more efficient utilization of streaming media resources and improved user experience quality.

CN114501016BActive Publication Date: 2024-11-12GUANGXI YOUNAI INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210272934.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-18
Publication Date
2024-11-12
Estimated Expiration
2042-03-18

AI Technical Summary

Technical Problem

The existing adaptive bit rate algorithm has limitations in resource utilization and video service quality, and cannot meet the needs of high-quality streaming media services, ignoring the impact of video content on user experience quality.

Method used

Adaptive streaming media method based on content significance is adopted to calculate the significance value of the video block through the significance map of the video block, and combine network conditions and player status, and use a linear user experience quality model to make bit rate decisions and optimize bits. Rate allocation.

Benefits of technology

It improves the utilization rate of streaming media resources and user experience quality, realizes more scientific and efficient bit rate allocation, and meets the needs of high-quality streaming media services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114501016B_ABST
    Figure CN114501016B_ABST
Patent Text Reader

Abstract

The present invention discloses an adaptive streaming media method based on content saliency. On the basis of considering network conditions and player status, the influence of the saliency of different video contents on user QoE is considered, and linear iterative clustering and graph-based transformation are used to extract local and global saliency features to construct a saliency model. By modeling the optimization target QoE, under the framework of model predictive control, the bit rate decision is made by comprehensively considering the available information to maximize the optimization target. The present invention can realize bit rate allocation based on content saliency, and further improve the utilization rate of streaming media resources and the quality of user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of streaming media transmission, and in particular to an adaptive streaming media method based on content saliency. Background Art

[0002] In recent years, video streaming has become one of the main types of Internet traffic. At the same time, users' demand for high video service quality is also growing rapidly, for example: (1) Users hope to be able to watch videos smoothly. (2) Users are no longer satisfied with low-definition videos, but look forward to the emergence of more high-definition video services. Although content providers strive to provide satisfactory video quality, it is still a huge challenge to achieve high-quality streaming services under various network conditions and terminal devices. The International Organization for Standardization MPEG proposed adaptive streaming technology, in which the client player uses an adaptive bitrate (ABR) algorithm to dynamically select the bitrate of future video blocks according to network conditions to maximize the user experience quality (QoE).

[0003] At present, ABR algorithms usually select bitrates for video chunks based on network conditions and player status, for example, strategies based on predicted network throughput, based on player buffer occupancy, and comprehensive consideration of both factors. However, they all ignore the impact of video content on user QoE. Since salient areas in video content can attract more visual attention, more buffer and bitrate resources need to be allocated to improve user experience quality. Therefore, existing adaptive bitrate algorithms still have certain limitations in resource allocation and maximizing QoE, and cannot meet the current deployment and development requirements of high-quality streaming media services. Therefore, a more scientific and efficient adaptive streaming media method is urgently needed. Summary of the invention

[0004] The present invention aims to solve the problems of low resource utilization and low video service quality existing in the current adaptive bit rate algorithm, and provides an adaptive streaming media method based on content significance.

[0005] To solve the above problems, the present invention is achieved through the following technical solutions:

[0006] The adaptive streaming media method based on content saliency includes the following steps:

[0007] Step 1: Cut the source video file in the video server into video blocks with fixed duration, and transcode each video block into different bit rate levels;

[0008] Step 2: The video server transmits the video blocks of each bit rate of the source video file to the content distribution network for storage, and samples a predetermined number of video frames from the video block with the highest bit rate to obtain a set of sampled video frames of the video block;

[0009] Step 3, the content distribution network calculates a local saliency map of each sampled video frame of the video block in pixels;

[0010] Step 4, the content distribution network calculates a global saliency map of each sampled video frame of the video block in pixels;

[0011] Step 5: The content distribution network linearly fuses the local saliency map in pixels of each sampled video frame of the video block obtained in step 3 and the global saliency map in pixels of each sampled video frame of the video block obtained in step 4 to obtain a saliency map in pixels of each sampled video frame of the video block;

[0012] Step 6: The content distribution network calculates the saliency of the video block based on the saliency maps of all sampled video frames of the video block obtained in step 5 in units of pixels;

[0013] Step 7: Use a linear user experience quality model that includes video quality, quality smoothness, and freeze time that comprehensively considers the saliency and bit rate of video blocks as the optimization target of the adaptive bitrate algorithm, and model the bitrate decision as an optimization problem based on predictive control of the linear user experience quality model, making decisions on the bitrate of video blocks while considering the predicted future information.

[0014] The specific process of step 3 above is as follows:

[0015] Step 3.1, extracting a feature map of the current sampled video frame in RGB color space, CIELab color space and LBP space in pixels;

[0016] Step 3.2: Use the linear iterative clustering algorithm to perform superpixel segmentation on the current sampled video frame to obtain the superpixel region r i ;

[0017] Step 3.3: Select all superpixel regions r i The superpixel region located at the edge of the current sampled video frame constitutes the background superpixel region r j ;

[0018] Step 3.4: Calculate each superpixel region r i Background saliency SM b (r i ):

[0019]

[0020] Step 3.5: Select all superpixel regions r i Medium background saliency SM b (ri ) The superpixel area greater than the set threshold constitutes the foreground superpixel area r p ;

[0021] Step 3.6: Calculate each superpixel region r i The prospect significance of SM f (r i ):

[0022]

[0023] Step 3.7: Set the superpixel region r i Background saliency SM b (r i ) and foreground saliency SM f (r i ) are fused to obtain the super pixel area r i Local saliency SM l (r i ):

[0024]

[0025] Step 3.8: Based on the superpixel regions r of the current sampled video frame i Local saliency SM l (r i ), and obtain the local saliency map SM of the current sampled video frame in pixels. l (x, y);

[0026] In the above formulas, SM b (r i ) is the super pixel area r i Background saliency, SM f (r i ) is the super pixel area r i The prospect significance of SM l (r i ) is the super pixel area r i The local significance of k b is the number of background superpixel areas; k f is the number of foreground superpixel areas; D RGB (r i , r j ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the RGB color space i and the background superpixel region r j The Euclidean distance, D CIELab (r i , r j) is the superpixel region r calculated based on the feature map of the current sampled video frame in the CIELab color space i and the background superpixel region r j The Euclidean distance, D LBP (r i , r j ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the LBP space i and the background superpixel region r j The Euclidean distance of i is the superpixel region r i The spatial distance between the center coordinates of and the center coordinates of the current sampled video frame; is the color space distance weight set, is the set LBP space distance weight; α is the set balance weight between background and foreground.

[0027] The specific process of step 4 above is as follows:

[0028] Step 4.1, extracting feature maps of the current sampled video frame in CBY color space, CRG color space, brightness space and direction space;

[0029] Step 4.2: The feature map in the CBY color space, the feature map in the CRG color space, the feature map in the brightness space, and the feature map in the direction space are added and merged pixel by pixel to obtain a global saliency map SM of the current sampled video frame in pixels. g (x, y).

[0030] The fusion formula of step 5 above is:

[0031] SM(x, y) = γ*SM l (x,y)+(1-γ)*SM g (x, y)

[0032] Where SM(x, y) is the saliency map of the current sampled video frame in pixels, SM l (x, y) is the local saliency map of the current sampled video frame in pixels, SM g (x, y) is the global saliency map of the current sampled video frame in pixels, and γ is the balance weight between the set local saliency and the global saliency.

[0033] The specific process of step 6 above is as follows:

[0034] Step 6.1, calculating the saliency average value of all pixels in the saliency map of each sampled video frame of the video block in units of pixels, to obtain the saliency value of each sampled video frame of the video block;

[0035] Step 6.2: Calculate the average of the saliency values ​​of all sampled video frames of the video block to obtain the saliency value of the video block.

[0036] In the above step 7, the optimization problem P based on the linear user experience quality model predictive control is:

[0037] P:max(Q1(b t )-σ*Q2(b t )-τ*Q3(b t ))

[0038] in:

[0039]

[0040]

[0041]

[0042] Where b t is the bit rate of the tth video block to be solved; Q1(b t ) is the reward function of video quality, Q2(b t ) is the penalty function of quality smoothness, Q3(b t ) is the penalty function of the freeze time; σ is the penalty weight of quality smoothness, τ is the penalty weight of the freeze time; T is the number of future video blocks set, H is the number of historical video blocks set; S t is the saliency of the t-th video block; min(S t ) is the maximum saliency among all video blocks of the source video, max(S t ) is the minimum saliency of all video blocks in the source video; μ is the set normalized weight, ξ is the set normalized deviation; VMAF(b t ) is the VMAF metric of the t-th video block, VMAF(b t-1 ) is the VMAF metric of the t-1th video block; C is the duration of the video block; L t-1 The buffer occupancy of the video player before downloading the tth video chunk; is the average download speed of the hth video chunk.

[0043] Compared with the prior art, the present invention considers the impact of the saliency of different video contents on user QoE on the basis of considering network conditions and player status in traditional adaptive methods. The present invention uses extracted local features and global features to construct a saliency model of video content and calculates the saliency values ​​of different video blocks. According to the collected saliency information of network conditions, player status and video content, an adaptive bitrate decision model is established based on a model predictive control framework. The present invention can realize bitrate allocation based on content saliency, further improving the utilization rate of streaming media resources and the quality of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is an application scenario diagram of the present invention.

[0045] Figure 2 The overall flow chart of the adaptive streaming media method based on content saliency.

[0046] Figure 3 Flowchart for saliency of video chunks.

[0047] Figure 4 Flowchart of the ABR algorithm. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific examples.

[0049] Figure 1 The application scenario diagram of the present invention is mainly composed of a video server, a content distribution network (CDN) and a video player. The video server cuts the source video file into multiple video blocks and transcodes them into different bit rates (representing different clarity) for storage. The content distribution network obtains the video blocks of each bit rate of the source video file from the video server and calculates the significance value of each video block. The video player determines the bit rate of each video block based on the ABR controller, and requests the content distribution network to download the video blocks of the corresponding bit rate through the Internet.

[0050] An adaptive streaming method based on content saliency, such as Figure 2 As shown, the steps include:

[0051] Step 1: Use FFmpeg to cut the source video file in the video server into video blocks of fixed duration (e.g., 4 seconds), and transcode each video block into different bit rate levels (e.g., 750 kbps, 1200 kbps, and 1850 kbps). Different bit rate levels correspond to different definitions (750 kbps, 1200 kbps, and 1850 kbps correspond to low definition, standard definition, and high definition, respectively).

[0052] Step 2: The video server transmits the video blocks of each bit rate of the source video file to the content distribution network for storage, and samples a certain number (eg, 20) of video frames from the video block with the highest bit rate to obtain a set of sampled video frames for each video block.

[0053] The video block with the highest bit rate after the source video is cut will be used to calculate the significance of each video block of the source video. Figure 3 Flowchart for saliency of video chunks.

[0054] Step 3: The content distribution network calculates the local saliency map SM of each sampled video frame of the video block in pixels. l (x, y).

[0055] Step 3.1, extract the feature map of the current sampled video frame in RGB color space, CIELab color space and LBP space in pixels.

[0056] Due to the influence of the feature information of the image on the unconscious selective attention mechanism, including static features (color, contour, texture and brightness, etc.) and motion features (motion direction, speed, etc.), the saliency of the superpixel region is calculated based on the feature map on the typical RGB color space, the feature map on the CIELab color space and the feature map on the LBP space. The feature map on the RGB color space, the feature map on the CIELab color space and the feature map on the LBP space are the feature maps commonly used in image processing. Among them, the feature map on the RGB color space represents the RGB color value of each pixel of the current sampled video frame, the feature map on the CIELab color space represents the CIELab color value of each pixel of the current sampled video frame, and the feature map on the LBP space represents the LBP value of each pixel of the current sampled video frame.

[0057] Step 3.2: Use the SLIC algorithm to perform superpixel segmentation on the current sampled video frame to obtain the superpixel region r i ; Where i∈{1, 2, ..., k}, k is the number of superpixel regions.

[0058] Step 3.3: Select all superpixel regions r i The superpixel region located at the edge of the current sampled video frame constitutes the background superpixel region r j ; where j∈{1, 2, ..., k b}, k b is the number of background superpixel regions.

[0059] Step 3.4: Calculate each superpixel region r iBackground saliency SM b (r i ).

[0060]

[0061] Where SM b (r i ) is the super pixel area r i The background significance of D RGB (r i , r j ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the RGB color space i and the background superpixel region r j The Euclidean distance, D CIELab (r i , r j ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the CIELab color space i and the background superpixel region r j The Euclidean distance, DLBP(r i , r j ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the LBP space i and the background superpixel region r j The Euclidean distance of i is the superpixel region r i The spatial distance between the center coordinates of and the center coordinates of the current sampled video frame; b is the number of background superpixel regions.

[0062] Step 3.5: Select all superpixel regions r i Medium background saliency SM b (r i ) The superpixel area greater than the set threshold constitutes the foreground superpixel area r p ; p∈{1, 2, ..., k f}, k f is the number of foreground superpixel regions.

[0063] Step 3.6: Calculate each superpixel region r i The prospect significance of SM f (r i ).

[0064]

[0065] Where SM f (r i ) is the super pixel area r iThe prospect significance of RGB (r i , r p ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the RGB color space i and the foreground superpixel region r p The Euclidean distance, D CIELab (r i , r p ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the CIELab color space i and the foreground superpixel region r p The Euclidean distance, D LBP (r i , r p ) is the superpixel region r calculated based on the feature map of the current sampled video frame in the LBP space i and the foreground superpixel region r p The Euclidean distance of is the color space distance weight set, is the set LBP space distance weight; k f is the number of foreground superpixel regions.

[0066] Step 3.7: Set the superpixel region r i Background saliency SM b (r i ) and foreground saliency SM f (r i ) are fused to obtain the super pixel area r i Local saliency SM l (r i ):

[0067]

[0068] Where SM l (r i ) is the super pixel area r i The local significance of SM b (r i ) is the super pixel area r i Background saliency, SM f (r i ) is the super pixel area r i The foreground saliency of α is the balance weight between the background and foreground. The background saliency can better highlight the object area, and the foreground saliency can better suppress the background noise.

[0069] Step 3.8: Based on the superpixel regions r of the current sampled video frame iLocal saliency SM l (r i ), and obtain the local saliency map SM of the current sampled video frame in pixels. l (x, y).

[0070] Since the superpixel region is obtained by segmenting the current sampled video frame, each superpixel region is composed of several pixels in the current sampled video frame. Therefore, when the local saliency of all superpixel regions is determined, the local saliency of each pixel of the current sampled video frame is also determined. At this time, the local saliency map of the current sampled video frame in pixels can be obtained.

[0071] Step 4: The content distribution network calculates the global saliency map SM of each sampled video frame of the video block in pixels. g (x, y).

[0072] Step 4.1, extract feature maps of the current sampled video frame in CBY color space, CRG color space, brightness (L) space and orientation (OT) space.

[0073] The feature maps on the CBY color space, the feature maps on the CRG color space, the feature maps on the brightness space, and the feature maps on the direction space are commonly used feature maps in image processing. Among them, the feature map on the CBY color space represents the CBY color value of each pixel of the current sampled video frame, the feature map on the CRG color space represents the CRG color value of each pixel of the current sampled video frame, the feature map on the brightness space represents the brightness value of each pixel of the current sampled video frame, and the feature map on the direction space represents the direction value of each pixel of the current sampled video frame.

[0074] Step 4.2: The feature map in the CBY color space, the feature map in the CRG color space, the feature map in the brightness space, and the feature map in the direction space are added and merged pixel by pixel to obtain a global saliency map SM of the current sampled video frame in pixels. g (x, y).

[0075] Step 5: The content distribution network converts the local saliency map SM of each sampled video frame of the video block obtained in step 3 into a pixel-based local saliency map SM l (x, y) and the global saliency map SM in pixels for each sampled video frame of the video block obtained in step 4 g (x, y) are linearly fused to obtain a saliency map SM(x, y) in pixels for each sampled video frame of the video block;

[0076] SM(x, y) = γ*SM l(x,y)+(1-γ)*SM g (x, y)

[0077] Where SM l (x, y) is the local saliency map of the current sampled video frame in pixels, SM g (x, y) is the global saliency map of the current sampled video frame in pixels, and γ is the balance weight between the set local saliency and the global saliency.

[0078] Step 6: The content distribution network calculates the saliency of the video block based on the saliency maps of all sampled video frames of the video block obtained in step 5 in units of pixels.

[0079] Step 6.1, calculating the saliency average value of all pixels in the saliency map of each sampled video frame of the video block in units of pixels, to obtain the saliency value of each sampled video frame of the video block;

[0080] Step 6.2: Calculate the average of the saliency values ​​of all sampled video frames of the video block to obtain the saliency value of the video block.

[0081] Step 7: Use the linear QoE (user experience quality) model that takes into account the video quality, quality smoothness and freeze time of the video block saliency and bit rate as the optimization target of the ABR (adaptive bit rate) algorithm, and model the bit rate decision as an optimization problem based on the user experience quality model predictive control, and make decisions on the bit rate of the video block while considering the predicted future information. Figure 4 .

[0082] The optimization goal of the ABR algorithm is to maximize user QoE, and video quality, quality smoothness, freeze time and video significance have an important impact on user QoE. For this reason, the present invention uses a linear QoE model based on the above factors as the optimization goal of the algorithm.

[0083] (1) Video quality

[0084] In the present invention, video quality is mainly determined by the saliency of the video block and the VMAF (Video Multimethod Assessment Fusion) metric based on the bit rate. If the saliency of the video block is higher, the required video quality is higher; otherwise, the required video quality is lower. If the bit rate of the video block is higher, the larger the VMAF metric is, the higher the video quality is; otherwise, the video quality is lower. In order to keep the bit rate allocation consistent with the saliency of the video content, video blocks with high saliency will be allocated more bit rate resources. To this end, the video quality model is as follows:

[0085]

[0086] In the formula, Q1(b t ) is the reward function for video quality; T is the number of future video blocks set; max(S t ) is the maximum saliency among all video blocks of the currently playing source video, min(S t ) is the minimum saliency among all video blocks of the currently playing source video, S t is the saliency of the t-th video block; VMAF(b t ) is the VMAF metric of the tth video block, b t is the bit rate of the t-th video block. μ is the set normalization weight, and ξ is the set normalization deviation. The role of μ and ξ is to convert the saliency S of the t-th video block t Mapped to the range of [ξ, μ+ξ]. In this embodiment, μ and ξ are set to 2 and 1 respectively.

[0087] (2) Quality smoothness

[0088] The time-varying nature of network bandwidth may cause fluctuations in video quality, which will reduce video quality smoothness and user QoE. In order to minimize the quality difference between different video blocks, a quality switching penalty function is defined to avoid frequent video quality changes. To this end, quality smoothness is modeled as follows:

[0089]

[0090] In the formula, Q2(b t ) is the penalty function of quality smoothness; T is the number of future video blocks set; VMAF(b t ) is the VMAF metric of the tth video block, b t is the bit rate of the tth video block, VMAF(b t-1 ) is the VMAF metric of the t-1th video block, b t-1 is the bit rate of the t-1th video block.

[0091] (3) Stall time

[0092] Continuously selecting high bitrates may exceed the network bandwidth capacity, causing video freezes and reducing user QoE, so video freezes should be avoided when making bitrate decisions. To this end, the video freeze time is modeled as:

[0093]

[0094] In the formula, Q3(b t ) is the penalty function for the freeze time; T is the number of set video blocks; max(a, b) means taking the larger value of a and b; C is the duration of the video block; b tis the bit rate of the tth video block. t-1 is the buffer occupancy rate of the video player before downloading the tth video block. Since the network bandwidth information cannot be obtained in advance, the present invention predicts the future average network bandwidth based on the harmonic mean of the download speeds of the past H video blocks. is the average download speed of the hth video block, and H is the number of set historical video blocks.

[0095] (4) QoE Model

[0096] The goal of the ABR algorithm is to maximize the overall QoE under time-varying network bandwidth. This problem is expressed as an optimization objective P, which comprehensively considers video quality, quality smoothness, and freeze time to maximize QoE. To this end, the optimization problem P is:

[0097] P:max(Q1(b t )-σ*Q2(b t )-τ*Q3(b t ))

[0098] In the formula, Q1(b t ) is the reward function of video quality, Q2(b t ) is the penalty function of quality smoothness, Q3(b t ) is the penalty function of the jamming time, b t is the bit rate of the t-th video block; σ is the penalty weight for quality smoothness, and τ is the penalty weight for freeze time.

[0099] The optimization target P achieves a trade-off between video quality, quality smoothness, and video stuttering time. If σ and τ are small, the video player will tend to choose a higher bitrate, however, this will result in greater video quality changes and stuttering time. Conversely, if σ and τ are large, the video player will maintain a lower bitrate unchanged to avoid frequent quality changes and video stuttering. At the same time, the bitrate allocation strategy will be consistent with the prominence of the video chunks, allocating more bandwidth resources to video content with higher prominence to maximize user QoE.

[0100] (5) Optimization method

[0101] The bit rate of each video block is output by solving the optimization problem P. Considering that the network bandwidth information cannot be obtained in advance, it is necessary to predict the future bandwidth in a short time. The present invention uses a model predictive control framework to optimize within a limited future time horizon based on the predicted bandwidth information to make a bit rate decision, that is, to transform the optimization problem P into a deterministic optimization problem with T future time horizons. When requesting to download video block t, the average network bandwidth of the next T video blocks is first predicted, and the bit rate of the current video block t is selected by maximizing the overall QoE of the next T video blocks. After the video block is downloaded, the player buffer size is updated, the download information is recorded, and T future time horizons are moved forward. The process is repeated until all video blocks are transmitted. By adopting a certain time horizon to make a bit rate decision, the bit rate resource allocation based on significance between different video blocks can be achieved. If the significance of the next video block (for example, video block t+1) is low, and video block t+2 has a higher significance, video block t+1 can be requested at a lower bit rate and video block t+2 can be requested at a higher bit rate by considering the video blocks in the future horizon.

[0102] The present invention considers the impact of the saliency of different video contents on user QoE based on network conditions and player status, and uses linear iterative clustering and graph-based transformation to extract local and global saliency features to build a saliency model. By modeling the optimization target QoE, under the framework of model predictive control, the bit rate decision is made by comprehensively considering the available information to maximize the optimization target. The present invention can realize bit rate allocation based on content saliency, further improving the utilization rate of streaming media resources and the quality of user experience.

Claims

1. Adaptive streaming media method based on content saliency, characterized by: The steps include: Step 1: Cut the source video file in the video server into video blocks with fixed duration, and transcode each video block into different bit rate levels; Step 2: The video server transmits the video blocks of each bit rate of the source video file to the content distribution network for storage, and samples a predetermined number of video frames from the video block with the highest bit rate to obtain a set of sampled video frames of the video block; Step 3, the content distribution network calculates a local saliency map of each sampled video frame of the video block in pixels; Right now: Step 3.1, extracting a feature map of the current sampled video frame in RGB color space, CIELab color space and LBP space in pixels; Step 3.2: Use the linear iterative clustering algorithm to perform superpixel segmentation on the current sampled video frame to obtain the superpixel region. ; Step 3.3: Select all superpixel regions The superpixel region located at the edge of the current sampled video frame constitutes the background superpixel region ; Step 3.4: Calculate each superpixel area Background salience : Step 3.5: Select all superpixel regions Medium background saliency The superpixel area larger than the set threshold constitutes the foreground superpixel area ; Step 3.6: Calculate each superpixel area Prospect Significance : Step 3.7: Superpixel area Background salience Fusion with foreground saliency to obtain superpixel region The local significance : Step 3.8: Based on each superpixel area of ​​the current sampled video frame The local significance , get the local saliency map of the current sampled video frame in pixels ; In the formula, Super pixel area The background significance of Super pixel area The prospect of Super pixel area The local significance of is the number of background superpixel areas; is the number of foreground superpixel areas; is the superpixel region calculated based on the feature map of the current sampled video frame in the RGB color space Superpixel area with background The Euclidean distance of is the superpixel region calculated based on the feature map of the current sampled video frame in the CIELab color space Superpixel area with background The Euclidean distance of is the superpixel region calculated based on the feature map of the current sampled video frame in the LBP space Superpixel area with background The Euclidean distance of is the superpixel region calculated based on the feature map of the current sampled video frame in the RGB color space Superpixel area with background The Euclidean distance of is the superpixel region calculated based on the feature map of the current sampled video frame in the CIELab color space Superpixel area with background The Euclidean distance of is the superpixel region calculated based on the feature map of the current sampled video frame in the LBP space Superpixel area with background The Euclidean distance of Super pixel area The spatial distance between the center coordinates of and the center coordinates of the current sampled video frame; is the color space distance weight set, is the set LBP space distance weight; The balance weight between the background and foreground is set; Step 4, the content distribution network calculates a global saliency map of each sampled video frame of the video block in pixels; Step 5: The content distribution network linearly fuses the local saliency map in pixels of each sampled video frame of the video block obtained in step 3 and the global saliency map in pixels of each sampled video frame of the video block obtained in step 4 to obtain a saliency map in pixels of each sampled video frame of the video block; Step 6: The content distribution network calculates the saliency of the video block based on the saliency maps of all sampled video frames of the video block obtained in step 5 in units of pixels; Step 7: Use a linear user experience quality model that comprehensively considers the saliency and bit rate of the video block, video quality, quality smoothness, and freeze time as the optimization target of the adaptive bitrate algorithm, and model the bitrate decision as an optimization problem based on the linear user experience quality model predictive control, and make a decision on the bitrate of the video block while considering the predicted future information; Optimization problem of predictive control based on linear user experience quality model for: in: In the formula, To be solved The bit rate of each video chunk; is the reward function for video quality, is the penalty function for quality smoothness, is the penalty function for the jamming time; is the penalty weight for quality smoothness, is the penalty weight of the jam time; is the number of future video blocks to be set, The number of historical video blocks set; For the The saliency of each video block; is the maximum saliency among all video blocks of the source video, is the minimum saliency among all video blocks of the source video; is the normalized weight set, is the normalized deviation set; For the Video blocks measure, For the Video blocks measure; is the duration of the video block; For the The buffer occupancy of the video player before downloading a video chunk; is the average download speed of the th video chunk.

2. The method for adaptive streaming media based on content saliency according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 4.1: Extract the current sampled video frame Color space, Feature maps in color space, brightness space and orientation space; Step 4.2: Feature maps in color space, The feature map in color space, the feature map in brightness space, and the feature map in direction space are combined pixel by pixel to obtain the global saliency map of the current sampled video frame in pixels. .

3. The adaptive streaming media method based on content saliency according to claim 1, characterized in that: The fusion formula of step 5 is: Where is the saliency map of the current sampled video frame in pixels, is the local saliency map of the current sampled video frame in pixels, is the global saliency map of the current sampled video frame in pixels, is the balance weight between the local significance and the global significance.

4. The adaptive streaming media method based on content saliency according to claim 1, characterized in that: The specific process of step 6 is as follows: Step 6.1, calculating the saliency average value of all pixels in the saliency map of each sampled video frame of the video block in units of pixels, to obtain the saliency value of each sampled video frame of the video block; Step 6.2: Calculate the average of the saliency values ​​of all sampled video frames of the video block to obtain the saliency value of the video block.