Code rate self-adaptive real-time video transmission method based on QoE (Quality of Experience) and resource consumption perception

Abstract

The invention discloses a code rate self-adaptive real-time video transmission method based on QoE (Quality of Experience) and resource consumption perception, belongs to the technical field of deep reinforcement learning and streaming media, and aims to solve the problem of balance between user experience quality and resource consumption in a live video stream. According to a traditional self-adaptive live streaming media method, a user can select the most suitable bit rate to achieve optimal rate self-adaption through predefined multi-bit-rate coding source video content, the method adopts a Dueling DDQN algorithm, the optimal rate self-adaption is achieved by analyzing user QoE feedback and resource consumption in a system state and learning an optimal coding step, and the optimal rate self-adaption is achieved. Therefore, the overall performance is maximized. According to the method, the effectiveness of the method is verified by using different types of video contents and real bandwidth data for comparison experiments, and the result shows that the method is superior to other baseline methods in overall performance.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning and streaming media technology, and in particular to a real-time video transmission method with adaptive bit rate based on QoE and resource consumption perception. Background Art

[0002] The global live video streaming market is expanding rapidly, with increasing audiences for live events from sports to gaming, and rising demand for seamless playback and high-definition video quality. However, the expansion of streaming services often exceeds the capabilities of current network infrastructure, resulting in high latency and video quality degradation, affecting user satisfaction and service provider revenue.

[0003] Adaptive bitrate (ABR) streaming technology optimizes the playback experience of live and on-demand video by dynamically adjusting video quality to adapt to network conditions. ABR technology detects the user's bandwidth and device capabilities, selects the appropriate video quality, and ensures a balance between video quality and continuous playback, reducing playback interruptions. This method relies on multiple pre-encoded video versions with different bit rates and resolutions. The client switches between these versions based on network conditions to maintain a smooth viewing experience.

[0004] While ABR effectively reduces rebuffering and improves the quality of experience (QoE) of users, its application in live broadcasting faces unique challenges. Originally designed for on-demand video, ABR is difficult to handle in live broadcasting due to the real-time requirements and unpredictable network fluctuations. Traditional ABR systems rely on fixed encoding parameters such as bitrate and resolution, which are determined by encoding ladders. However, fixed encoding ladders in live broadcasting events are difficult to cope with sudden drops in bandwidth or surges in viewers, resulting in reduced streaming quality and user dissatisfaction. In addition, existing ABR encoding ladder designs make it difficult to effectively manage the resource consumption required for live broadcasting.

[0005] In a live broadcast environment, the immediacy of content delivery is critical, and the ABR system must respond quickly to network fluctuations to reduce latency and maintain real-time performance. Unlike on-demand streaming, which can pre-buffer content, live broadcasts require more active and rapid adaptation strategies. The delay of buffering or bitrate switching has a significant impact on live broadcasts, especially in sports events or game broadcasts. In addition, network changes may occur suddenly during live broadcasts, and the surge in viewers during popular live broadcasts makes this situation even more serious, requiring the encoding ladder design to not only adapt to gradual network changes, but also to cope with sudden bandwidth fluctuations.

[0006] To address these issues, we propose a QoE- and resource-consumption-aware coding gradient optimization method for real-time video streaming, which can dynamically select the best coding ladder in live broadcast according to network conditions and the number of viewers. Summary of the invention

[0007] Purpose of the invention: In view of the problems of frequent network fluctuations, drastic changes in the number of viewers and the inability of fixed coding steps to adapt to dynamic changes in the above-mentioned live broadcast, the present invention provides a real-time video transmission method with adaptive bit rate based on QoE and resource consumption perception.

[0008] Technical solution: A real-time video transmission method with adaptive bitrate based on QoE and resource consumption perception, characterized in that the method includes using a reinforcement learning-based adversarial dual-depth Q network to analyze user QoE feedback and resource consumption under system status, learning and determining the best coding ladder, and combining the current network bandwidth status and the change in the number of users. The cloud server processes the video stream in combination with the dynamically determined coding ladder, including transmitting the processed video stream in blocks and uploading it to the CDN server. When a user requests a video, the video block is pulled from the nearest CDN node;

[0009] The adversarial dual-depth Q network is used to learn the bit rate adaptation strategy in live broadcast and output the optimal strategy for coding ladder selection. The state in the network consists of the histogram bandwidth distribution of all viewers, the number of live viewers, and the video quality during the live broadcast. The state at time t is represented by S t , specifically:

[0010]

[0011] represents the percentage of users in each bandwidth interval at time t, N t Represents a dataset of user request counts in a historical period, and iterates the dataset to estimate the number of users entering the live broadcast room in time period t. Define VMAF t =(VMAF t , VMAF t+1 , ..., VMAF t+k-1 ) represents the video quality of the subsequent k video blocks during the live broadcast;

[0012] The control action in this network is to select the coding step of the video stream in the next time slot according to the current system state, which is expressed as:

[0013] a t =π(s t ) → EL t , E.L. t ∈EL

[0014] where π represents the strategy for selecting the coding ladder, a t represents the action corresponding to time t, EL t represents a set of encoding steps corresponding to time t, and EL represents a set of encoding steps in the streaming media system;

[0015] The reward in this network is that during live broadcast, the client's choice of bitrate is based on maximizing its expected viewing quality while taking into account the cost of bitrate and encoding time on the server. The following function is used to measure the reward within a time period:

[0016] r t (s t , a t )=QoE-α*B-β*E t

[0017] where r t (s t , a t ) means in s t Select action a in state t The reward obtained after the performance is quantized, QoE represents the quality of user experience, expressed as an average value, α is the weight of the average bit rate B, and β is the server-side encoding time E t The weight of

[0018] Based on the above-mentioned adversarial dual-depth Q network, the bitrate adaptation strategy in live broadcast is learned. The calculation of the Q value is not only based on a single Q value estimation, but also combines the value of the state V(s) and the advantage of the action A(s, a), which can be expressed as:

[0019] (s t , a t ;θ)=V(s t θ V )+A(s t ,a t θ A )-mean a′ A(s t , a′; θ A )

[0020] Here θ V and θ A Represent the parameters of the value flow and advantage flow respectively. According to the Bellman equation, the target network is used to predict the maximum Q value of the next state, combined with the immediate reward, to update the Q value of the current state-action pair, as shown below:

[0021]

[0022] Where γ is a discount factor used to measure the current immediate reward and future rewards. During training, the network parameters are adjusted by minimizing the difference between the predicted Q value and the target Q value:

[0023] L(θ)=E[(yQ(s t ,a t ;θ)) 2 ]

[0024] Where y is the target Q value in the above calculation;

[0025] Finally, we use the gradient descent method to update the network parameters:

[0026]

[0027] Where α represents the learning rate.

[0028] Furthermore, the quality of user experience (QoE) in the adversarial dual-depth Q network described in the method is expressed by the following formula:

[0029] QoE=B-w1*R-w2*ΔB

[0030] Where w1 represents the weight of R, w2 represents the weight of ΔB, R represents the average total rebuffering time of k video chunks in time slot t, and ΔB represents the average absolute change in client bitrate.

[0031] The average total rebuffering time R for k video chunks in time slot t is calculated as follows:

[0032]

[0033] The rebuffering time is calculated when the user's bandwidth is less than the currently selected bitrate, otherwise the rebuffering time is considered to be 0.

[0034] Furthermore, the method optimizes and adjusts the encoding gradient in the following manner:

[0035] The video stream is processed according to discrete time. After selecting the encoding ladder for time period t, the streaming server encodes the video content using the selected encoding ladder. The client selects the bit rate that best matches the current bandwidth from the encoding ladder. This process is repeated for each video block until the next time period t+1, when the agent receives the system state S observed during this period. t +1, then take action t +1 Select the encoding ladder of t+1 according to the current strategy, and finally get reward r after downloading all video blocks during time period t+1 t +1, and repeat the above process until the live broadcast session ends.

[0036] Furthermore, the application of this method includes processing of streaming media environment, video stream encoding and client bit rate adaptation, specifically:

[0037] In the streaming media environment, a set of encoding ladders in the streaming media system is represented as EL. In each time period t, the live streaming system will select an encoding ladder EL tTo encode live content, the bandwidth during a video session varies over time;

[0038] Video stream encoding: The cloud server receives the original video stream uploaded by the live source and first performs preprocessing, including format conversion, denoising, and resolution adjustment, to adapt to the subsequent encoding process. The cloud server creates video versions with different bit rates and resolutions based on the selected encoding ladder, and divides these versions into shorter time slices for transmission;

[0039] Client bitrate adaptation adopts a bitrate-based approach to client bitrate adaptation. During the live broadcast process, the specific process of client bitrate adaptation is as follows: first, select the bitrate that does not exceed the current network bandwidth from all available bitrates. If there is a bitrate that meets the current bandwidth conditions, select the highest bitrate, otherwise select the smallest effective bitrate; after a round of live broadcast, the adversarial dual-depth Q network model performs a weighted average of the average bitrate, average rebuffering time, and average bitrate change of all users to calculate the overall user experience quality QoE.

[0040] The cloud server performs data compression on the pre-processed video stream in the video stream encoding operation including inputting the encoder to reduce the required bandwidth.

[0041] Beneficial effects: Compared with the prior art, the method provided by the present invention not only considers traditional network parameters such as bandwidth, rebuffering and bit rate changes, but also considers the resource consumption of live broadcast services to design the coding ladder. By adaptively selecting the best coding ladder, the system implemented based on this method can more flexibly cope with the fluctuation of client network conditions during live broadcast, thereby optimizing the overall audience experience and resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a real-time video streaming system operation framework diagram;

[0043] Figure 2 It is the network structure of the reinforcement learning algorithm based on DuelingDDQN;

[0044] Figure 3 is a graph showing the convergence of the model constructed by the method of the present invention;

[0045] Figure 4 is the comparison of the reward, QoE, bandwidth consumption and encoding time of the present invention and the baseline method;

[0046] in, Figure 4 (a) shows the reward comparison of different methods under different weight modes. Figure 4 (b) shows the normalized QoE comparison of different methods under different weight modes. Figure 4(c) shows the bandwidth consumption comparison of different methods under different weight modes. Figure 4 (d) shows the encoding time comparison of different methods under different weight modes;

[0047] Figure 5 is the bitrate-quality comparison of different theme videos and the comparison of the overall reward of the present invention and the baseline method; wherein, Figure 5 (a) shows the changes in video quality under different themes. Figure 5 (b) Represents the total system reward for different video topics. DETAILED DESCRIPTION

[0048] In order to explain the technical solution of the present invention in detail, further elaboration is given below in conjunction with the accompanying drawings of the specification.

[0049] The present invention provides a real-time video transmission method based on QoE and resource consumption perception and bit rate adaptation, which is used to solve the balance problem between user experience quality (QoE) and resource consumption in live video streaming. The traditional adaptive live streaming method encodes the source video content through predefined multi-bitrates, so that users can choose the most suitable bitrate to achieve optimal rate adaptation. However, due to the dynamic nature of live broadcasting, the number of users fluctuates greatly, and the bandwidth status of each user is constantly changing, and the relationship between the coding ladder and the user QoE becomes complicated. In addition, the resource consumption in live streaming has a significant impact on operating costs because video encoding and transmission require a lot of resources. The method comprehensively considers the user experience quality QoE and the resource consumption of video encoding and streaming media to optimize the overall performance of live broadcasting. The method adopts the dueling dual deep Q network (Dueling DDQN algorithm) to learn and select the best coding ladder according to the video content characteristics, user bandwidth distribution and the number of viewers, and learns the best coding ladder by analyzing the user QoE feedback and the resource consumption under the system state, thereby maximizing the overall performance. The implementation of the method includes the following processes:

[0050] First, based on the consideration of discrete time systems, where time is assigned as t=1, 2, 3.... The duration of a time period may last for several minutes. The present invention formulates it as an RL problem (reinforcement learning problem), in which the agent (the intelligent agent in the reinforcement learning network) interacts with the streaming environment to learn the optimal strategy for the selection of the encoding ladder. Specifically, after selecting the encoding ladder for time period t, the streaming server encodes the video content using the selected encoding ladder, and the client selects the bit rate from the encoding ladder that best matches the current bandwidth. This process is repeated for each video block until the next time period t+1, when the agent receives the system state S observed during this period t +1, then take action t+1 Select the encoding ladder of t+1 according to the current strategy, and finally get reward r after downloading all video blocks during time period t+1 t +1. These processes repeat until the live session ends.

[0051] Streaming environment: The present invention represents a set of encoding ladders in the streaming system as EL. In each time period, the live streaming system selects an encoding ladder to encode the live content. The bandwidth during the video session varies with time. We will download the video chunk t i The average bandwidth is denoted as v i The coding step selected at time node t is represented by step EL t , is the video block t i The selected bit rate is denoted as b i , so the average bit rate

[0052] Video stream encoding: The server receives the original video stream uploaded by the content provider or live source, and first performs preprocessing, including format conversion, denoising, and resolution adjustment, to adapt to the subsequent encoding process. The preprocessed video stream is then input into the encoder for data compression to reduce the required bandwidth. The server then creates video versions with different bit rates and resolutions based on the selected encoding ladder, and divides these versions into shorter time slices, each of which contains a few seconds of video content. This segmentation allows the client to flexibly switch between different versions of the video stream as network conditions change. In addition, the server dynamically selects the most appropriate encoding ladder at the beginning of each time period based on changes in client bandwidth to ensure the continuity and quality of video playback.

[0053] Client bitrate adaptation: We use a bitrate-based approach to adapt the client bitrate. During live streaming, the specific process of client bitrate adaptation is as follows: First, select the bitrate that does not exceed the current network bandwidth from all available bitrates. If there is a bitrate that meets the current bandwidth conditions, select the highest bitrate, otherwise select the smallest effective bitrate. After a round of live streaming, the model performs a weighted average of the average bitrate, average rebuffering time, and average bitrate change of all users to calculate the overall user experience quality QoE.

[0054] The DuelingDDQN network structure used for learning the bit rate adaptation strategy in live broadcasting in the present invention is specifically introduced below.

[0055] State: The state consists of the histogram bandwidth distribution of all viewers, the number of live viewers, and the video quality during the live broadcast. The state at time t is represented by S t , specifically:

[0056]

[0057] Here the bandwidth is distributed between [0, 10Mb / s], divided into 20 intervals of 500kbps, such as [0, 0.5Mb / s], [0.5Mb / s, 1Mb / s], [1Mb / s, 2Mb / s], etc. BD quantifies the percentage of users in each bandwidth interval as a vector. For Nt, a dataset of user request counts from last month captured from domestic video websites is used, and the dataset is iterated to determine the number of users entering the live broadcast room in time period t. At the same time, video quality evaluation indicators for various live broadcast themes are collected. Furthermore, in actual application, a script can be developed to extract the average VMAF score of 10-second video blocks in a 2-hour live broadcast segment, which varies at different bit rates for different themes. Here, VMAF is a vector defined as VMAF t =(VMAF t , VMAF t+1 , ..., VMAF t+k-1 ), which represents the video quality of the subsequent k video blocks during the live broadcast process.

[0058] Action: The control action is to select the encoding step of the video stream of the next time slot according to the current system state, which is expressed as:

[0059] a t =π(s t ) → EL t ,EL t ∈EL

[0060] where π represents the strategy used to select the coding ladder.

[0061] Reward: In live streaming, the client's choice of bitrate is based on maximizing its expected viewing quality while taking into account the cost of bitrate and encoding time on the server. The following function is used to measure the reward over time:

[0062] r t s t ,a t )=QoE-α*B-β*E t

[0063] where r t s t ,a t ) means in s t Select action a in state t Here, QoE represents the quality of user experience, which we express as an average value, α is the weight of the average bitrate B, and β is the server-side encoding time E. t In addition, QoE can be expressed by the following formula:

[0064] QoE=B-w1*R-w2*ΔB

[0065] where R represents the average total rebuffering time of k video chunks in time slot t, and ΔB represents the average absolute change in client bitrate, specifically:

[0066]

[0067] The rebuffer time is typically calculated when the user's bandwidth is less than the currently selected bitrate, except in these cases the rebuffer time is considered to be 0.

[0068] In the DuelingDDQN used in the present invention to learn the bitrate adaptation strategy in live broadcast, the calculation of the Q value is not only based on a single Q value estimation, but also combines the value of the state (V(s)) and the advantage of the action (A(s, a)). The specific formula is as follows:

[0069] s t ,a t ;θ)=V(s t θ V )+A(s t ,a t θ A )-mean a′ A(s t , a′; θ A )

[0070] Here θ V and θ A Represent the parameters of the value flow and advantage flow respectively. According to the Bellman equation, the target network is used to predict the maximum Q value of the next state, combined with the immediate reward, to update the Q value of the current state-action pair:

[0071]

[0072] Where y is a discount factor used to weigh the current immediate reward and future rewards. During training, we adjust the network parameters by minimizing the difference between the predicted Q value and the target Q value:

[0073] L(θ)=E[(yQ(s t , a t ;θ)) 2 ]

[0074] Where y is the target Q value calculated according to the Bellman equation.

[0075] Finally, the network parameters are updated using the gradient descent method:

[0076]

[0077] Where α represents the learning rate.

[0078] Further introduction, combined with Figure 1 The system operation framework diagram is shown in Figure 1. When the live broadcast starts, the original video signal from the live broadcast is captured and sent to the encoder for preliminary encoding. The encoded video stream is then uploaded to the cloud server. At this stage, the cloud server not only serves as a storage and forwarding center for video streams, but also plays the role of intelligent decision-making. Based on the strategy trained by reinforcement learning, the server monitors the current network bandwidth status and changes in the number of users in real time, thereby selecting a set of the most appropriate bit rate combination ladders.

[0079] As the live broadcast progresses, the number of users entering the live broadcast room may fluctuate significantly. For example, the sudden entry of a large number of users may cause instantaneous network congestion. At this time, the reinforcement learning algorithm will dynamically adjust the bitrate ladder according to the predetermined reward mechanism, such as maximizing the user's viewing quality and reducing playback interruptions, and select a lower bitrate to ensure smooth transmission of the video stream. On the contrary, if the network condition is monitored to be good and the user base is stable, the algorithm may improve the video quality and select a higher bitrate ladder to provide a clearer picture.

[0080] The bitrate version selected by the cloud server will be transcoded, cut into small video chunks, and uploaded to the CDN server. When users request a video, they will pull the video chunk from the nearest CDN node. In this way, the entire live broadcast system can not only respond to real-time network and user changes, but also improve the user's overall viewing experience (QoE) through continuous learning and optimization.

[0081] For the server-side decision agent, a coding gradient optimization strategy based on QoE and resource consumption perception is adopted. The network structure of the reinforcement learning algorithm is as follows Figure 2 shown.

[0082] From left to right, the diagram starts with state inputs, including user bandwidth, number of users, and quality of each video chunk. These state metrics are fed into the feature extraction layer, where a neural network processes the data to extract key features for decision making. The data processed by the feature extraction layer is then divided into two parts: value and advantage. The value network aims to assess the overall expected utility of taking any possible action in a given state, while the advantage network assesses the relative advantage of each specific action compared to the average action value. Finally, the algorithm outputs a series of action values ​​Q, corresponding to different bitrate steps. Each action value Q(s t ,a t ) represents the expected return of selecting a specific bitrate step in the current state.

[0083] In the experiment, four groups of weights were used according to the optimization preferences for QoE, bandwidth consumption, and encoding consumption, namely Bandwidth Saving, Light Encoding, Balance Mode, and QoE Max. Bandwidth Saving aims to reduce bandwidth consumption, with weights of 0.05 and 30 respectively; Light Encoding aims to save computing resources for encoding, with weights of 0.03 and 50 respectively; Balance Mode aims to balance bandwidth saving and computing resource saving, with weights of 0.03 and 30 respectively; QoE Max represents a mode that maximizes user experience, focusing entirely on maximizing user experience without considering bandwidth saving or encoding time, with weights of 0 and 0 respectively.

[0084] The convergence of the corresponding models of the present invention under the four weight modes is further compared. Figure 3 It can be seen that after about 5000 rounds of training, the algorithm has achieved good convergence effect.

[0085] Next, we compared the average rewards of news topic videos under different weight settings. The selected benchmark methods include Rate, VMAF, CRF, PPO, and DQN. Among them, Rate represents bitrate optimization, VMAF represents optimization based on video quality assessment, and CRF is used for constant quality optimization. Figure 4 As shown in (a), our method achieves higher average rewards under different mode settings compared with the baseline method. Under different weights, the live broadcast system has different requirements for user QoE and resource consumption, such as maximizing user QoE, saving bandwidth costs for live video broadcasts, and reducing encoding computational costs. Through comparative analysis of the average rewards under different weights, our method achieves the best overall performance under all weight settings. Baseline algorithms such as Rate, VMAF, and CRF do not directly consider user QoE, bandwidth costs, and encoding computational costs when designing the encoding ladder. These baseline algorithms cannot adjust the encoding ladder according to different encoding modes, so the rewards under different encoding modes are fixed. Our method can dynamically adjust the encoding ladder according to different encoding modes, thereby obtaining higher rewards. In addition, compared with PPO and DQN, the use of the model of the present invention can learn more optimal strategies, thereby obtaining higher average rewards and improving system performance. Figure 4 (b) Figure 4 (c) and Figure 4 In (d), the performance characteristics of the method of the present invention in different encoding modes are demonstrated. Figure 4(b) shows that the method of the present invention can obtain the highest average user QoE in QoE Max mode, while in Bandwidth Saving mode, the average user QoE is the lowest, and in Balance mode, the average QoE is between the two. In addition, in Bandwidth Saving mode, the present invention can obtain the lowest user bandwidth cost, while in QoEMax mode, the bandwidth cost is the highest. In Light Encoding mode, the present invention can obtain the lowest encoding cost.

[0086] Next, we compared the live streaming performance of videos under different themes. In this example, we selected news, sports, and games as themes, and obtained complete live streaming clips from major domestic video websites. By obtaining video chunks of each theme every 10 seconds on the server and measuring the VMAF scores at different bit rates, we observed the changes in the video quality of these themes. Figure 5 As shown in (a), the VMAF score increases with the increase in bitrate, illustrating the direct impact of bitrate improvement on video quality. Specifically, the quality improvement of news videos is relatively gradual, indicating that high watchability can be maintained even at lower bitrates. Sports videos show rapid improvement in the medium bitrate range because they usually contain more fast-moving scenes, which require higher bitrates to maintain image quality. At the same time, game videos show significant quality improvement at high bitrates, reflecting the complexity and dynamic range of game graphics. Finally, we compare the system total reward for different video topics, such as Figure 5 As shown in (b), the results show that the proposed model performs best in all tested video subjects and is significantly better than other algorithms in terms of image quality and coding efficiency.

Claims

1. A real-time video transmission method with adaptive bit rate based on QoE and resource consumption perception, characterized in that: The method includes using a reinforcement learning-based adversarial dual-depth Q network to analyze user QoE feedback and resource consumption under system status, learning and determining the best encoding ladder, and combining the current network bandwidth status and the change in the number of users. The cloud server combines the dynamically determined encoding ladder to process the video stream, including transmitting the processed video stream in blocks and uploading it to the CDN server. When a user requests a video, the video block is pulled from the nearest CDN node; The adversarial dual-depth Q network is used to learn the bit rate adaptation strategy in live broadcast and output the optimal strategy for coding ladder selection. The state in the network consists of the histogram bandwidth distribution of all viewers, the number of live viewers, and the video quality during the live broadcast. The state at time t is represented by s t , specifically: represents the percentage of users in each bandwidth interval at time t, N t Represents a dataset of user request counts in a historical period, and iterates the dataset to estimate the number of users entering the live broadcast room in time period t. Define VMAF t =(VMAF t ,VMAF t+1 ,...,VMAF t+k-1 ) represents the video quality of the subsequent k video blocks during the live broadcast; The control action in this network is to select the coding step of the video stream in the next time slot according to the current system state, which is expressed as: to t =π(s t )→THE t ,THE t ∈EL where π represents the strategy for selecting the coding ladder, a t represents the action corresponding to time t, EL t represents a set of encoding steps corresponding to time t, and EL represents a set of encoding steps in the streaming media system; The reward in this network is that during live broadcast, the client's choice of bitrate is based on maximizing its expected viewing quality while taking into account the cost of bitrate and encoding time on the server. The following function is used to measure the reward within a time period: r t (s t ,a t )=QoE-α*B-β*E t where r t (s t ,a t ) means in s t Select action a in state t The reward obtained after the performance is quantized, QoE represents the quality of user experience, expressed as an average value, α is the weight of the average bit rate B, and β is the server-side encoding time E t The weight of Based on the above-mentioned adversarial dual-depth Q network, the bitrate adaptation strategy in live broadcast is learned. The calculation of the Q value is not only based on a single Q value estimation, but also combines the value of the state V(s) and the advantage of the action A(s,a), which can be expressed as: (s t ,a t ;θ)=V(s t ;θ V )+A(s t ,a t ;θ A )-mean a' A(s t ,a';θ A ) Here θ V and θ A Represent the parameters of the value flow and advantage flow respectively. According to the Bellman equation, the target network is used to predict the maximum Q value of the next state, combined with the immediate reward, to update the Q value of the current state-action pair, as shown below: Where γ is a discount factor used to measure the current immediate reward and future rewards; during training, the network parameters are adjusted by minimizing the difference between the predicted Q value and the target Q value: L(θ)=E[(yQ(s t ,a t ;i)) 2 Where y is the target Q value used to update the current state-action pair as mentioned above; Finally, we use the gradient descent method to update the network parameters: Where α represents the learning rate.

2. The real-time video transmission method based on QoE and resource consumption perception and bit rate adaptation according to claim 1 is characterized in that: The quality of experience (QoE) in the adversarial dual-depth Q network is expressed by the following formula: QoE=B-w1*R-w2*ΔB Where w1 represents the weight of R, w2 represents the weight of ΔB, R represents the average total rebuffering time of k video chunks in time slot t, and ΔB represents the average absolute change in client bitrate.

3. The real-time video transmission method based on QoE and resource consumption perception and bit rate adaptation according to claim 2 is characterized in that: The average total rebuffering time R for k video chunks in time slot t is calculated as follows: The rebuffering time is calculated when the user's bandwidth is less than the currently selected bitrate, otherwise the rebuffering time is considered to be 0.

4. The real-time video transmission method based on QoE and resource consumption perception and bit rate adaptation according to claim 1 is characterized in that: The method is to optimize and adjust the coding gradient in the following way: The video stream is processed according to discrete time. After selecting the encoding ladder for time period t, the streaming server encodes the video content using the selected encoding ladder. The client selects the bit rate that best matches the current bandwidth from the encoding ladder. This process is repeated for each video block until the next time period t+1, when the agent receives the system state S observed during this period. t +1, then take action t +1 Select the encoding ladder of t+1 according to the current strategy, and finally get reward r after downloading all video blocks during time period t+1 t +1, and repeat the above process until the live broadcast session ends.

5. The real-time video transmission method based on QoE and resource consumption perception and bit rate adaptation according to claim 1, characterized in that: The application of this method includes processing of streaming media environment, video stream encoding and client bit rate adaptation, specifically: In the streaming media environment, a set of encoding ladders in the streaming media system is represented as EL. In each time period t, the live streaming system will select an encoding ladder EL t To encode live content, the bandwidth during a video session varies over time; Video stream encoding: The cloud server receives the original video stream uploaded by the live source and first performs preprocessing, including format conversion, denoising, and resolution adjustment, to adapt to the subsequent encoding process. The cloud server creates video versions with different bit rates and resolutions based on the selected encoding ladder, and divides these versions into shorter time slices for transmission; Client bitrate adaptation adopts a bitrate-based approach to client bitrate adaptation. During the live broadcast process, the specific process of client bitrate adaptation is as follows: first, select the bitrate that does not exceed the current network bandwidth from all available bitrates. If there is a bitrate that meets the current bandwidth conditions, select the highest bitrate, otherwise select the smallest effective bitrate; after a round of live broadcast, the adversarial dual-depth Q network model performs a weighted average of the average bitrate, average rebuffering time, and average bitrate change of all users to calculate the overall user experience quality QoE.

6. The real-time video transmission method based on QoE and resource consumption perception and bit rate adaptation according to claim 5, characterized in that: The cloud server performs data compression on the pre-processed video stream in the video stream encoding operation including inputting the encoder to reduce the required bandwidth.

Images