Large-scale multi-party interaction real-time video stream transmission method

Abstract

The invention discloses a large-scale multi-party interaction real-time video stream transmission method, and belongs to a video communication and network optimization technology. The method aims at solving the problems of bandwidth allocation complexity, QoE coupling influence, suboptimal global QoE caused by local decision and the like in a large-scale multi-party interaction real-time video streaming media scene in the prior art. By introducing a new transmission scheme based on a double-layer Stackelberg game and an optimized embedding strategy, the method disclosed by the invention realizes efficient bandwidth allocation and adaptive bit rate control of large-scale multi-party interaction real-time video streams in a dynamic network environment, and remarkably improves the overall service performance and user experience.

Description

Technical Field

[0001] The invention belongs to the technical field of video communication and network optimization, and relates to a transmission method for large-scale multi-party interactive real-time video streams. Background Art

[0002] With the advent of the digital age, multi-party interactive real-time video streaming (MPI-RTVS) has been widely used in scenarios such as collaborative meetings, joint performances, and live PK. These application scenarios require multiple senders to share video content simultaneously and meet the different quality of experience (QoE) of many viewers. However, existing technologies still face some key challenges when dealing with these scenarios:

[0003] Complexity issue: As the number of viewers increases, especially in large-scale scenarios involving hundreds of viewers, existing methods find it difficult to effectively handle the optimization problems of upload and download bandwidth allocation and adaptive bitrate control. For example, deep reinforcement learning (DRL) and some small-scale optimization methods based on scalable video coding (SVC), although they perform well in one-to-one scenarios, often find it difficult to provide scalable solutions when faced with large-scale interactive scenarios of MPI-RTVS. In addition, most current adaptive bitrate (ABR) algorithms focus on the transmission of a single video stream, and implement bandwidth allocation and bitrate adjustment through fixed control rules (usually making decisions based on network status indices such as throughput and cache occupancy) or reinforcement learning (RL) methods. However, these methods show obvious limitations when dealing with large-scale MPI-RTVS scenarios and facing complex and dynamically changing network environments, and it is difficult to provide optimal service performance.

[0004] QoE coupling effect: In scenarios with multiple parties interacting, competition between senders not only affects their own QoE, but also has a significant impact on the QoE of all viewers. Existing transmission schemes usually assume that all participants are equal in resource allocation and ignore the coupling effect between senders, which leads to suboptimal service performance in large-scale application scenarios. In particular, most existing studies only focus on optimization problems in small-scale networks, often ignoring the complexity and scalability requirements in large-scale MPI-RTVS applications.

[0005] Local decisions lead to suboptimal global QoE: Existing methods usually focus on bitrate adaptive adjustment based on user network bandwidth and other local constraints, ignoring the optimization of global QoE. This local optimization strategy may work well in small-scale applications, but often leads to suboptimal global QoE distribution in large-scale networks, which cannot meet the efficiency and consistency of multi-party interaction requirements.

[0006] At the same time, the rapid development of augmented reality (AR) and virtual reality (VR) technologies has significantly driven the demand for high-bandwidth, low-latency real-time video transmission. These technical application scenarios usually involve real-time interaction between multiple users, such as multiplayer online games, virtual meetings, immersive social media, etc. Such applications usually require the transmission of large amounts of data in different network environments, including multi-view video streams, multi-channel audio streams, and possible sensor data, which poses a huge challenge to traditional video streaming solutions, especially in scenarios with high concurrent users. Traditional video streaming technologies can no longer meet this demand. To this end, it is necessary to develop more efficient network resource allocation and adaptive bitrate control mechanisms for large-scale multi-party interactive real-time video streaming transmission to ensure a consistent and high-quality user experience in various complex network environments.

[0007] Therefore, a new transmission scheme that can optimize bandwidth and bit rate allocation in large-scale MPI-RTVS applications is urgently needed to address the above challenges. Summary of the invention

[0008] Purpose of the invention: In view of the problems existing in the prior art in large-scale multi-party interactive real-time video streaming scenarios, such as bandwidth allocation complexity, QoE coupling impact, and suboptimal global QoE caused by local decision-making, the present invention provides a large-scale multi-party interactive real-time video streaming transmission method.

[0009] Technical solution: A large-scale multi-party interactive real-time video streaming transmission method, which includes building a two-layer Stackelberg game model, optimizing strategies for the sender and receiver respectively, and combining the optimization embedding strategy to dynamically adjust the upload and download bandwidth and video bit rate based on the global QoE to ensure the balance and maximization of the interests of all parties;

[0010] The two-layer Stackelberg game model divides the diverse QoE problems in large-scale MPI-RTVS applications into the following two sub-problems from the perspective of the server:

[0011] The first-layer Stackelberg game is used for the upstream bandwidth allocation of the sender, which is regarded as a bandwidth trading problem, where the server is the upstream bandwidth provider and each sender decides the amount of bandwidth to purchase to ensure high-quality video streaming and smooth interaction;

[0012] The second layer Stackelberg game is used for the receiver's downstream bandwidth allocation and bit rate selection. In this layer, the server sells downstream bandwidth, and the sender selects its own downstream bandwidth and the bit rate of each received video stream according to different viewing preferences;

[0013] The two-layer Stackelberg game model combines the joint upload-download bandwidth allocation of MPI-RTVS with adaptive bitrate control to manage the complexity of large-scale MPI-RTVS scenarios and optimize the utility function of the receiver's QoE by maximizing the utility of the three parties.

[0014] The two-layer Stackelberg game model is constructed as follows:

[0015] The utility function of the sender in the first-level game is defined as the sender's income minus the cost. Its income function is the revenue brought by the high-quality video content uploaded by the sender, and the upload bit rate has a diminishing marginal effect on the revenue. The cost function includes the bandwidth purchase cost and the video encoding cost, and its calculation formula is as follows:

[0016]

[0017]

[0018] U i represents the utility function of sender i, a i represents the revenue coefficient related to the video quality of sender i, b i represents the income growth rate coefficient of sender i, r i↑ represents the upload bandwidth / bit rate of sender i, p u Indicates the unit price of the upstream bandwidth. represents the fixed encoding cost, represents the variable coding cost coefficient, I is the set of all senders, and the constraint ensures that the total bandwidth purchased by all senders does not exceed the server's uplink bandwidth capacity W ↑ , and the sending bit rate is r i↑ Must be a non-negative value;

[0019] The second-layer Stackelberg game construction includes:

[0020] The receiver's utility function in the second-layer game is defined as QoE minus the cost of purchasing bandwidth. The receiver's QoE is determined by factors including video quality, video quality stability, and delay tolerance. Its expression is:

[0021]

[0022]

[0023] The constraints indicate that the sum of the downlink bandwidths of all receivers cannot exceed the total downlink bandwidth; the sum of the bit rates of the video streams received by each receiver cannot exceed its downlink bandwidth, and the downlink bit rate of each video stream cannot exceed the uplink bit rate;

[0024] In the above formula, QoE ij represents the video stream received by receiver j from sender i, and its calculation formula is:

[0025]

[0026] Where K is a constant, is the bit rate of the first k time slots, r min is the required minimum bitrate, and is the marginal quality improvement reduced by the logarithmic function of the ratio of the actual bitrate to the minimum bitrate, x ij is the relevant weight coefficient; y ij is the weight coefficient that affects the receiver sensitivity, z ij is the weight coefficient, L ij It represents the delay of receiver j receiving the video stream from sender i, including encoding time, uploading time, transcoding time, downloading time and decoding time. Its calculation formula is as follows:

[0027]

[0028] in, is the encoding speed of sender i, 1 is the upload time, STV is the standard transcoding speed factor, PPT is the processing time per time slot, is the decoding speed of receiver j;

[0029] For two-layer Stackelberg games and nested optimization problems, this method solves them based on a dynamically adjusted optimization solution method.

[0030] Furthermore, the method considers the global QOE including:

[0031] Sender QoE, real-time high-quality communication between all senders;

[0032] QoE at the receiver side, high and stable video quality and low latency for all source streams;

[0033] In this regard, the two-layer Stackelberg game model also includes the following implementation process:

[0034] (1) The server has an upstream bandwidth price p u On the basis of dynamic adjustment, the first-layer Stackelberg game is established with the sender;

[0035] (2) The sender selects the optimal sending bit rate r under the uplink bandwidth unit price set by the server. i↑ To maximize its utility U i ;

[0036] (3) Based on the first-layer Stackelberg game, the server dynamically adjusts the downlink bandwidth unit price pd and establishes the second-layer Stackelberg game with the receiver;

[0037] (4) Each receiver solves its own nested optimization problem to obtain the optimal bit rate allocation ratio η ij ;

[0038] (5) The receiver decides the bit rate r of each video stream received by each receiver based on the optimal bit rate allocation ratio obtained by solving the nested optimization problem for each receiver under the downlink bandwidth unit price set by the server and the first-level game result. ij .

[0039] Furthermore, the method solves the two-layer Stackelberg game and nested optimization problem based on dynamically adjusted optimization. The first-layer Stackelberg game model is defined as:

[0040] S1={I∪{Server},r ↑ ,p u ,U s ,F}, where,U s =[U1,…,U m ];

[0041] I represents the sender set, Server represents the server, r i↑ represents the upload bandwidth of each sender, p u Indicates the unit price of upstream bandwidth, U s represents the set of utility functions of the sender, and F represents the upstream utility function of the server;

[0042] The second-level Stackelberg game model is defined as:

[0043]

[0044] in and W ↓ =[W 1↓ ,…,W m+n↓ ];

[0045] In the formula, represents the receiver set, Server represents the server, W i↓ represents the download bandwidth of each receiver, p d Indicates the unit price of downstream bandwidth, V * represents the set of utility functions of the receiver, and G represents the downstream utility function of the server;

[0046] For the nested problem of the second layer in the two-layer Stackelberg game, the mathematical expression is as follows:

[0047]

[0048] In the formula, is the allocation ratio of the kth time slice, r i↑ is the upload bitrate of sender i calculated in the first-level game S1, is the total download bandwidth purchased by the receiver in the last k time slices;

[0049] In order to solve this optimization problem, the Lagrange multiplier method is introduced to construct the corresponding Lagrange function:

[0050]

[0051] The following Lagrangian function and KKT conditions are assumed:

[0052] 1) Stability conditions:

[0053] 2) Original feasibility conditions:

[0054]

[0055] 3) Complementary relaxation conditions:

[0056]

[0057] β ij (-η ij )=0,for i∈I

[0058] 4) Dual feasibility condition: α ij ≥0,β ij ≥0,for i∈I;

[0059] Based on the above conditions, the optimal solution η of the optimization problem is derived ij , η ij The value of is divided into the following three cases:

[0060] if And α ij =0, then

[0061] if And α ij =0, then

[0062] otherwise,

[0063] Furthermore, the solution of the two-layer Stackelberg game model is as follows:

[0064] The steps to solve the first-level Stackelberg game include:

[0065] (1.1) Input parameters: including the sender's parameter set And the maximum upstream bandwidth unit price Total upstream bandwidth W ↑ , the number of senders m and the convergence threshold ∈;

[0066] (1.2) Initialization: The server initially sets the upstream bandwidth unit price to The convergence flag is set to flase, the iteration counter t is set to 1, and the initialization bit rate vector is initialized to a zero vector;

[0067] (1.3) Iteration process:

[0068] For each sender i, calculate its optimal bit rate, which is calculated based on the sender's parameters and the current bandwidth unit price. The formula is:

[0069] Check whether the sum of the bit rates of all senders exceeds the available uplink bandwidth W↑. If so, scale the bit rates of each sender proportionally to meet the bandwidth constraint, i.e.

[0070] Check the convergence of the bit rate, that is, whether the difference between the bit rate vectors of the current iteration and the previous iteration is less than the threshold ∈; if the condition is met, mark the convergence; otherwise, update the iteration counter and continue the calculation;

[0071] After each iteration, the server updates the uplink bandwidth unit price p based on the current bit rate allocation. u for To maximize its own utility F;

[0072] (1.4) Output results: The optimal uplink bit rate and optimal unit price after final convergence are used as the output of the algorithm;

[0073] The steps to solve the second-level Stackelberg game include:

[0074] (2.1) Input parameters: including the receiver's parameter set {x ij},{y ij},{z ij}, the number of receivers n, the number of senders m, the total system bandwidth STV, the maximum price limit PPT, and the convergence threshold ∈;

[0075] (2.2) Initialization: The server initially sets the downlink bandwidth unit price to The convergence flag is set to false, the iteration counter t is set to 1, and the uplink bit rate vector is initialized to the value obtained from algorithm S1. (2.3) Iteration process:

[0076] For each receiver j, calculate its optimal bandwidth allocation based on the current uplink bit rate and bandwidth allocation;

[0077] Check whether the sum of bandwidth allocations of all receivers is within the allowed downlink bandwidth range. If it exceeds, adjust the bandwidth allocation proportionally to meet the constraint, i.e.,

[0078] Check the convergence of bandwidth allocation, that is, whether the difference between the bandwidth allocation vector of the current iteration and the previous iteration is less than the threshold ∈. If the condition is met, mark the convergence; otherwise, update the iteration counter and continue the calculation;

[0079] After each iteration, the server updates the downlink bandwidth unit price p based on the current bandwidth allocation. d , in order to maximize its own utility;

[0080] (2.4) Output result: The optimal downstream bandwidth allocation W↓ and the optimal unit price p after final convergence d as the output of the algorithm.

[0081] Furthermore, the server utility functions in the two-layer Stackelberg game are the revenue obtained from selling upstream and downstream bandwidth, respectively, and are defined as follows:

[0082] For the upstream bandwidth utility function:

[0083] For the upstream bandwidth utility function: Among them, p u Indicates the unit price of upstream bandwidth, p d Indicates the unit price of downstream bandwidth, r i↑ represents the upload bandwidth / bit rate of sender i, W j↓ represents the downlink bandwidth of receiver j, r↑ is the set of upload bit rates of all senders, and W ↓ is the set of downlink bandwidths of all receivers, and J is the set of all receivers, including all senders and viewers included in the receivers.

[0084] Beneficial effects: The large-scale multi-party interactive real-time video streaming transmission method provided by the present invention solves the problem that the existing multi-party interactive real-time video streaming media transmission scheme is difficult to effectively optimize the upload and download bandwidth allocation and adaptive bit rate control in large-scale scenarios, especially the inability to simultaneously take into account the coupled diversified quality of experience (QoE) of the sender and the receiver. The transmission method proposed by the present invention effectively manages the complexity in large-scale multi-party interactive real-time video streaming scenarios through a two-layer Stackelberg game embedded with an optimization mechanism. The transmission scheme includes in the first layer, the server acts as an uplink bandwidth provider, and the sender purchases bandwidth through game decisions to ensure the high quality of the video stream and the smoothness of the interaction; in the second layer, the server allocates the downlink bandwidth and the bit rate of the video stream according to the viewing preference of the receiver. Through this two-layer architecture, the present invention realizes the joint allocation of upload and download bandwidth and adaptive bit rate control in multi-party interactive real-time video streaming media transmission, significantly optimizes the overall system utility, and improves the user experience. Experimental results show that compared with the existing methods, the scheme has higher scalability and lower computational overhead, significantly improves the quality of service, and performs well in large-scale application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is an application scenario diagram of the method described in the present invention;

[0086] Figure 2 It is the overall framework diagram of the method of the present invention;

[0087] Figure 3 is a comparison chart of the utility in different scenarios;

[0088] Wherein: FIG3(a) is a comparative analysis diagram of the three-party utility functions of the present invention and the existing method in scenario 1 (the network scale is set to (15,150)); FIG3(b) is a comparative analysis diagram of the three-party utility functions of the present invention and the existing method in scenario 2 (the network scale is set to (20,300); FIG3(c) is a comparative analysis diagram of the three-party utility functions of the present invention and the existing method in scenario 3 (the network scale is set to (30,500));

[0089] Figure 4 shows the utility values ​​of all receivers in scenario 1;

[0090] Among them, Figure 4(a) is a distribution diagram of the utility function value of the receiver using the TLSG method in scenario 1; Figure 4(b) is a distribution diagram of the utility function value of the receiver using the fixed strategy method in scenario 1; Figure 4(c) is a distribution diagram of the utility function value of the receiver using the NSGA-A method in scenario 1; Figure 4(d) is a distribution diagram of the utility function value of the receiver using the NSGA-R method in scenario 1;

[0091] Figure 5 It is a QoE comparison of different scenarios in the embodiment. DETAILED DESCRIPTION

[0092] In order to explain the technical solution provided by the present invention in detail, the following is a further introduction in conjunction with the accompanying drawings. Figure 1-2 As shown, the present invention provides a large-scale multi-party interactive real-time video streaming transmission method, which is based on bandwidth allocation and adaptive bit rate control optimization of a double-layer Stackelberg game. The method aims to improve the transmission efficiency and user experience quality in a multi-party real-time interactive video streaming media system.

[0093] Combination Figure 1 As shown, the scenario considered consists of three parties: m senders, a server, and n spectators, where n>>m. For example, in a live performance, there are usually a large number of spectators and a small number of senders. In MPI-RTVS, the sender also needs to receive video streams from other senders, so in the present invention, the consideration of the second-layer game also includes the consideration of the sender as a receiver. The transmission scheme is determined within each time slice. Within each time slice, each sender encodes the source video through a wireless channel with a selected bandwidth and uploads it to the server. The server is responsible for transcoding and forwarding the video streams of all senders, and forwarding the video streams to all receivers at the allocated bit rate to meet network limitations and the diverse QoE of the receivers. In order to ensure that all receivers can get a satisfactory experience, the transmission scheme needs to schedule the uplink bandwidth, downlink bandwidth, and bit rate to meet:

[0094] (1) Sender QoE, i.e., real-time high-quality communication between all senders;

[0095] (2) Viewer QoE, i.e., high stable video quality and low latency for all source streams. Obviously, the sender’s QoE has a great impact on the viewer’s QoE, and the participation of a large number of viewers makes it difficult to design a transmission scheme using traditional optimization solutions.

[0096] In order to optimize the diverse QoE in large-scale MPI-RTVS applications, the present invention designs a two-layer Stackelberg game framework to decompose the original problem into two sub-problems from the perspective of the server.

[0097] First layer (uplink bandwidth allocation of senders): The present invention regards uplink bandwidth allocation as a bandwidth trading problem, where the server is the uplink bandwidth provider and each sender decides the amount of bandwidth to purchase to ensure high-quality video streaming and smooth interaction.

[0098] The second layer (downstream bandwidth allocation and bitrate selection at the receiver): In this layer, the server sells downstream bandwidth, and the sender selects its own downstream bandwidth and the bitrate of each video stream received according to different viewing preferences.

[0099] This two-layer framework combines the joint upload-download bandwidth allocation of MPI-RTVS with adaptive bitrate control, which can effectively manage the complexity of large-scale MPI-RTVS scenarios. By maximizing the utility of the three parties, the utility function of the receiver's QoE can be optimized.

[0100] The specific implementation steps include:

[0101] S1, server upstream bandwidth unit price p u On the basis of dynamic adjustment, the first-layer Stackelberg game is established with the sender;

[0102] S2. The sender selects the optimal sending bit rate ri↑ to maximize its utility U under the uplink bandwidth unit price set by the server. i ;

[0103] S3. Based on the first-layer game, the server dynamically adjusts the downlink bandwidth unit price p d , further establish the second-layer Stackelberg game with the receiver;

[0104] S4. Each receiver solves its own nested optimization problem to obtain the optimal bit rate allocation ratio η ij ;

[0105] S5. The receiver decides the bit rate r of each video stream received by each receiver based on the optimal bit rate allocation ratio obtained by solving the nested optimization problem for each receiver under the downlink bandwidth unit price set by the server and the first-level game result. ij .

[0106] Furthermore, the server utility functions in the two-layer game are the revenue obtained from selling upstream and downstream bandwidths, and the formulas are defined as follows:

[0107] For the upstream bandwidth utility function:

[0108] For the upstream bandwidth utility function: Among them, p u Indicates the unit price of upstream bandwidth, p d Indicates the unit price of downstream bandwidth, r i↑ represents the upload bandwidth / bit rate of sender i, W j↓ represents the downlink bandwidth of receiver j, r↑ is the set of upload bit rates of all senders, and W ↓ is the set of downlink bandwidths of all receivers, and J is the set of all receivers, including all senders and viewers.

[0109] The receiver's utility function in the second-layer game is defined as QoE minus the cost of purchasing bandwidth. The receiver's QoE is determined by multiple factors, including video quality, video quality stability, and delay tolerance. Its calculation formula is as follows:

[0110]

[0111] Among them, QoE ij represents the video stream received by receiver j from sender i. The specific calculation formula is:

[0112]

[0113] Among them: The first row represents the requirement for high video quality in the last K time slots, where K is a constant. is the bit rate of the first k time slots, r min is the minimum bitrate required. The marginal quality improvement is reduced using a logarithmic function of the ratio of the actual bitrate to the minimum bitrate. ij is the relevant weight coefficient.

[0114] The second line uses y ij Quantify the stability of video quality, y ij is the weight coefficient that affects the sensitivity of the receiver. The last line represents the receiver's tolerance to delay, where z ij is the weight coefficient, L ij L represents the delay of receiver j receiving the video stream from sender i. ij It includes encoding time, uploading time, transcoding time, downloading time and decoding time. The calculation formula is as follows:

[0115]

[0116] in, is the encoding speed of sender i, 1 is the upload time (because the upload bit rate and bandwidth are the same), STV is the standard transcoding speed factor, PPT is the processing time per time slot, is the decoding speed of receiver j. The constraints respectively indicate that the sum of the downlink bandwidth of all receivers cannot exceed the total downlink bandwidth, the sum of the bit rates of the video streams received by each receiver cannot exceed its downlink bandwidth, and the downlink bit rate of each video stream cannot exceed the uplink bit rate.

[0117] When solving the double-layer Stackelberg game and nested optimization problems, the present invention designs an optimization solution method based on dynamic adjustment. The specific steps are as follows:

[0118] Solution of the first-level Stackelberg game:

[0119] The game model is defined as: Among them, U s =[U1,…,U m ].

[0120] As the leader of the game, the server adjusts the upstream bandwidth unit price p u , affecting the strategy selection of each sender. As a follower, the sender selects its upload bandwidth r according to the bandwidth price set by the server. i↑ , in order to maximize its utility U i Specifically, each sender determines its optimal upload bandwidth by solving the maximization problem of its utility function. In this process, the server dynamically adjusts p by observing the sender's response. u , in order to maximize its own utility.

[0121] The key to solving the first-level game is to find an equilibrium point, at which the strategy choices of the server and all senders are interdependent and mutually optimal. This equilibrium point can be solved through an iterative algorithm, that is, the server continuously adjusts p u , the sender responds and adjusts r i↑ , until the Nash equilibrium is reached.

[0122] The specific steps include:

[0123] S1-1. Input parameters: including the sender's parameter set And the maximum upstream bandwidth unit price Total upstream bandwidth W ↑ , the number of senders m and the convergence threshold ∈;

[0124] S1-2, the initial server upstream bandwidth unit price is The convergence flag is set to flase, and the iteration counter t is set to 1. At the same time, the initialization bit rate vector is initialized to a zero vector.

[0125] S1-3, iterative process:

[0126] For each sender i, calculate its optimal bit rate, which is calculated based on the sender's parameters and the current bandwidth unit price. The formula is:

[0127]

[0128] Check whether the sum of the bit rates of all senders exceeds the available upstream bandwidth W↑. If so, scale the bit rates of each sender proportionally to meet the bandwidth constraint. That is,

[0129]

[0130] Check the convergence of the bit rate, that is, whether the difference between the bit rate vectors of the current iteration and the previous iteration is less than the threshold ∈. If the condition is met, mark convergence; otherwise, update the iteration counter and continue calculation.

[0131] After each iteration, the server updates the uplink bandwidth unit price p based on the current bit rate allocation. u for To maximize its own utility F.

[0132] S1-4. Output results: The optimal uplink bit rate and optimal unit price after final convergence are used as the output of the algorithm.

[0133] Solution of the second-level Stackelberg game and nested optimization problem:

[0134] Problem definition and decomposition: The optimization goal of the entire system is to improve the overall performance of the system by maximizing the receiver's utility function. In order to solve this problem, the present invention breaks it down into two levels of sub-problems. On the basis of the first level of game, the server further acts as a leader and sets the downlink bandwidth unit price p d , and influence the receiver's strategy choice.

[0135] Nested optimization problem: This part focuses on the optimization of the bit rate selection strategy for each receiver. By introducing the bit rate allocation ratio coefficient η ij , the present invention defines the receiver utility function V j The solution to this optimization problem will serve as the basis for the receiver's utility function formula in the final Stackelberg game.

[0136] Stackelberg game problem: After solving the nested optimization problem, the system further considers the bandwidth purchase strategy game between the receiver and the server. The server sets the bandwidth price p d To maximize its benefits, the receiver chooses bandwidth W j↓ To maximize its utility V j . Finally, the optimal strategies of both parties are determined through the equilibrium solution of the game.

[0137] For the nested problem in the two-layer Stackelberg game, the mathematical expression is defined as follows:

[0138]

[0139] In the formula, is the allocation ratio of the kth time slice, r i↑ is the upload bitrate of sender i calculated in the first-level game S1, is the total download bandwidth purchased by receiver j in the past k-th time slice.

[0140] In order to solve this optimization problem, the Lagrange multiplier method is introduced to construct the corresponding Lagrange function:

[0141]

[0142] The following Lagrangian function and KKT conditions are assumed:

[0143] 1) Stability conditions:

[0144] 2) Original feasibility conditions:

[0145]

[0146] 3) Complementary relaxation conditions:

[0147]

[0148] 4) Dual feasibility condition: α ij ≥0,β ij ≥0,for i∈I.

[0149] Based on the above conditions, the optimal solution η of the optimization problem can be derived ij . Specifically, η ij The value of is divided into the following three cases:

[0150] 1) If And α ij =0, then

[0151] 2) If And α ij =0, then

[0152] 3) Otherwise,

[0153] Furthermore, the modeling and solution of the game problem:

[0154] The second-level Stackelberg game model is defined as: in and W ↓ =[W 1↓ ,…,W m+n↓ ].

[0155] According to the solution results of the nested optimization problem, consider:

[0156] 1) When

[0157] 2) When

[0158] By solving the above game model, we can get the optimal strategy of the server and each receiver by setting the derivative equal to 0, thereby maximizing the utility of the entire system.

[0159] a) When The optimal solution is

[0160] b) When The optimal solution is

[0161] The specific solution steps of the second layer also include:

[0162] S2-1. Input parameters: including the receiver's parameter set {x ij},{y ij},{z ij}, the number of receivers n, the number of senders m, the total system bandwidth STV, the maximum price limit PPT, and the convergence threshold ∈.

[0163] S2-2, Initialization: The server initially sets the downlink bandwidth unit price to The convergence flag is set to false and the iteration counter t is set to 1. At the same time, the uplink bit rate vector is initialized to the value obtained from algorithm S1.

[0164] S2-3, iterative process:

[0165] For each receiver j, the optimal bandwidth allocation is calculated based on the current uplink bit rate and bandwidth allocation.

[0166] Check whether the sum of bandwidth allocations of all receivers is within the allowed downlink bandwidth range. If it exceeds, adjust the bandwidth allocation proportionally to meet the constraint. That is,

[0167] Check the convergence of bandwidth allocation, that is, whether the difference between the bandwidth allocation vector of the current iteration and the previous iteration is less than the threshold ∈. If the condition is met, mark convergence; otherwise, update the iteration counter and continue calculation.

[0168] After each iteration, the server updates the downlink bandwidth unit price p based on the current bandwidth allocation. d , in order to maximize its own utility.

[0169] Output result: optimal downstream bandwidth allocation W↓ and optimal unit price p after final convergence d as the output of the algorithm.

[0170] The present invention combines two-layer game with nested optimization, and effectively coordinates the strategy selection of the sender and the receiver through a double-layer Stackelberg game model. u and p d , which not only ensures the maximization of its own benefits, but also ensures that the utility of each sender and receiver is optimized. This integrated approach enables the system to efficiently handle complex bandwidth and bit rate allocation issues in large-scale MPI-RTVS scenarios, significantly improving the overall system performance and the QoE of all parties, especially in the face of diverse demands and changes in the network environment, and still maintaining stable and efficient transmission quality.

[0171] As shown in Figures 3-5, this embodiment performs a performance evaluation on the two-layer Stackelberg game transmission mechanism (TLSG) proposed in the present invention, and conducts comparative experiments with several baseline methods to verify its effectiveness and efficiency in the large-scale multi-party real-time interactive video streaming (MPI-RTVS) scenario.

[0172] 1. Baseline Method

[0173] Since most existing studies focus on small-scale networks, and their bit rates are usually determined based on scalable video coding (SVC), they are not applicable to the large-scale transcoding scenarios considered in this article. Therefore, this embodiment compares the solution proposed by the present invention with the following existing methods:

[0174] Fixed strategy: The upstream bandwidth, downstream bandwidth, and bit rate are all set to a fixed average value for all senders and receivers.

[0175] NSGA-II genetic algorithm: Due to its strong ability in multi-objective optimization problems, especially in dealing with conflicting objectives inherent in video streaming systems (such as maximizing video quality and minimizing delay), it is used as the baseline algorithm. The present invention implements two variants of NSGA-II, namely NSGA-A (sender and receiver bit rates are evenly distributed) and NSGA-R (according to the formula to allocate bit rate).

[0176] 2. Experimental setup

[0177] In the experiment, multiple senders and viewers upload and download high-quality video streams simultaneously. By adjusting the number of senders and viewers, the scalability of the network under changing scale is tested. The network scale is set to (m,n)∈{(15,150),(20,300),(30,500)}, corresponding to three different experimental scenarios (Scenario 1 to Scenario 3).

[0178] Further, Figures 3(a)-(c) show the comparison of the three-party utility results obtained by using four different algorithms in three different experimental scenarios. Figures 4(a)-(d) show the distribution of the receiver utility values ​​obtained by using four different algorithms in scenario 1. Figure 5 It shows the average QoE value of the receiver in three different experimental scenarios.

[0179] The specific experimental parameters are as follows:

[0180] Bandwidth setting: Upstream bandwidth W↑ is 100Mbps, downstream bandwidth W↓ is 500Mbps, and the maximum unit price of upstream bandwidth The maximum unit price of downlink bandwidth is 2$ / Mbps, the maximum unit price of downlink bandwidth is 10$ / Mbps, and the upload bit rate range is r i↑ ∈(0,20]Mbps.

[0181] Sender parameter settings: including scaling factor a i , Growth factor b i , initial coding cost and coding rate They are randomly generated in the ranges of [10,15], [0.5,1.5], [0.01,0.05], and [0.005,0.02] respectively.

[0182] Receiver parameter settings: r min Set to 0.01, STV to 1, PPT to 0.9, and Randomly generated in the range [10,20]. QoE weight parameter x ij ,y ij 、z ij For the receiver Randomly generated in the range [0.5,1], and the delay coefficient z of the sender ij ∈[1,2]. The historical time window is set to K=10.

[0183] Each method was executed 200 times, and the average results were taken for analysis.

[0184] (III) Performance evaluation

[0185] Utility and QoE: The superiority of the present invention has been fully verified through comparative experiments of utility and QoE values. Experimental results show that TLSG significantly outperforms the baseline method in all test scenarios, especially in terms of the utility of the server and the receiver. Compared with the fixed strategy and NSGA-A, the bitrate allocation strategy of TLSG brings significant utility gains, further improving the overall performance of the system. Detailed analysis shows that TLSG can maintain a high and balanced utility distribution under various network conditions, avoiding the common utility fluctuations and negative utility situations in other methods, and ensuring that the viewing experience of the receiver is always at a high level. In addition, the optimization goal of TLSG is centered on the QoE of the sender and the receiver. It adopts a hierarchical decision-making process based on Stackelberg game theory, adjusts the bitrate according to the real-time QoE requirements, and can provide excellent performance under different network conditions and user preferences. This result once again proves the effectiveness of the bitrate allocation strategy in TLSG in meeting the QoE requirements of the receiver.

[0186] Execution time: To evaluate the computational overhead of TLSG, we measured the execution time of each transmission scheme. The experimental results show that the execution time of TLSG in all scenarios is at the millisecond level. Even with 500 viewers, it can converge within 571 milliseconds, demonstrating its efficiency and practicality in large-scale scenarios. In contrast, although NSGA-II can achieve relatively good performance in terms of utility and QoE, its execution time is too long, especially when the network scale increases, the execution time can be as long as 32 minutes, which is difficult to meet the needs of real-time applications. NSGA-II also has problems such as sensitivity to parameters and the need for a lot of experiments and tuning, which further limits its operability in practical applications. In contrast, TLSG significantly reduces the computational overhead while retaining the advantages of efficient bandwidth management and adaptive bitrate allocation, making it the best choice for large-scale MPI-RTVS applications.

[0187] In summary, the TLSG solution proposed in the present invention not only significantly improves the user experience in large-scale MPI-RTVS applications through intelligent bandwidth management and adaptive bit rate allocation, but also has low computing overhead, fully meets the needs of real-time interactive video streaming applications, and has broad application prospects.

Claims

1. A large-scale multi-party interactive real-time video streaming transmission method, characterized in that: The method includes constructing a two-layer Stackelberg game model, optimizing strategies for the sender and the receiver respectively, and combining the optimization embedding strategy to dynamically adjust the upload and download bandwidth and video bit rate based on the global QoE to ensure the balance and maximization of the interests of all parties; The two-layer Stackelberg game model divides the diverse QoE problems in large-scale MPI-RTVS applications into the following two sub-problems from the perspective of the server: The first-layer Stackelberg game is used for the upstream bandwidth allocation of the sender, which is regarded as a bandwidth trading problem, where the server is the upstream bandwidth provider and each sender decides the amount of bandwidth to purchase to ensure high-quality video streaming and smooth interaction; The second layer Stackelberg game is used for the receiver's downstream bandwidth allocation and bit rate selection. In this layer, the server sells downstream bandwidth, and the sender selects its own downstream bandwidth and the bit rate of each received video stream according to different viewing preferences; The two-layer Stackelberg game model combines the joint upload-download bandwidth allocation of MPI-RTVS with adaptive bitrate control to manage the complexity of large-scale MPI-RTVS scenarios and optimize the utility function of the receiver's QoE by maximizing the utility of the three parties. The two-layer Stackelberg game model is constructed as follows: The utility function of the sender in the first-level game is defined as the sender's income minus the cost. Its income function is the revenue brought by the high-quality video content uploaded by the sender, and the upload bit rate has a diminishing marginal effect on the revenue. The cost function includes the bandwidth purchase cost and the video encoding cost, and its calculation formula is as follows: U i represents the utility function of sender i, a i represents the revenue coefficient related to the video quality of sender i, b i represents the income growth rate coefficient of sender i, r i↑ represents the upload bandwidth / bit rate of sender i, p u Indicates the unit price of the upstream bandwidth. represents the fixed encoding cost, represents the variable coding cost coefficient, i is the set of all senders, and the constraint ensures that the total bandwidth purchased by all senders does not exceed the uplink bandwidth capacity W of the server. ↑ , and the sending bit rate is r i↑ Must be a non-negative value; The second-layer Stackelberg game construction includes: The receiver's utility function in the second-layer game is defined as QoE minus the cost of purchasing bandwidth. The receiver's QoE is determined by factors including video quality, video quality stability, and delay tolerance. Its expression is: The constraints indicate that the sum of the downlink bandwidth of all receivers cannot exceed the total downlink bandwidth; the sum of the bit rates of the video streams received by each receiver cannot exceed its downlink bandwidth, and the downlink bit rate of each video stream cannot exceed the uplink bit rate; In the above formula, QoE ij represents the video stream received by receiver j from sender i, and its calculation formula is: Where K is a constant, is the bit rate of the first k time slots, r min is the required minimum bitrate, and is the marginal quality improvement reduced by the logarithmic function of the ratio of the actual bitrate to the minimum bitrate, x ij is the relevant weight coefficient; y ij is the weight coefficient that affects the receiver sensitivity, z ij is the weight coefficient, L ij It represents the delay of receiver j receiving the video stream from sender i, including encoding time, uploading time, transcoding time, downloading time and decoding time. Its calculation formula is as follows: in, is the encoding speed of sender i, 1 is the upload time, STV is the standard transcoding speed factor, PPT is the processing time per time slot, is the decoding speed of receiver j; For two-layer Stackelberg games and nested optimization problems, this method solves them based on a dynamically adjusted optimization solution method.

2. The large-scale multi-party interactive real-time video stream transmission method according to claim 1 is characterized in that: The method considers the global QOE as follows: Sender QoE, real-time high-quality communication between all senders; QoE at the receiver side, high and stable video quality and low latency for all source streams; In this regard, the two-layer Stackelberg game model also includes the following implementation process: (1) The server has an upstream bandwidth price p u On the basis of dynamic adjustment, the first-layer Stackelberg game is established with the sender; (2) The sender selects the optimal sending bit rate r under the uplink bandwidth unit price set by the server. i↑ To maximize its utility U i ; (3) Based on the first-layer Stackelberg game, the server dynamically adjusts the downlink bandwidth unit price pd and establishes the second-layer Stackelberg game with the receiver; (4) Each receiver solves its own nested optimization problem to obtain the optimal bit rate allocation ratio η ij ; (5) The receiver decides the bit rate r of each video stream received by each receiver based on the optimal bit rate allocation ratio obtained by solving the nested optimization problem for each receiver under the downlink bandwidth unit price set by the server and the first-level game result. ij .

3. The large-scale multi-party interactive real-time video stream transmission method according to claim 1 or 2, characterized in that: This method solves the two-layer Stackelberg game and nested optimization problem based on dynamically adjusted optimization. The first-layer Stackelberg game model is defined as: S1 = {I ∪ {Server}, r ↑ , p u , U s , F}, where, U s = [U1, …, U m ; I represents the sender set, Server represents the server, r i↑ represents the upload bandwidth of each sender, p u Indicates the unit price of upstream bandwidth, U s represents the set of utility functions of the sender, and F represents the upstream utility function of the server; The second-level Stackelberg game model is defined as: in and W ↓ =[W 1↓ ,…,W m+n↓ ]; In the formula, represents the receiver set, Server represents the server, W i↓ represents the download bandwidth of each receiver, p d Indicates the unit price of downstream bandwidth, V * represents the set of utility functions of the receiver, and G represents the downlink utility function of the server; For the nested problem of the second layer in the two-layer Stackelberg game, the mathematical expression is as follows: In the formula, is the allocation ratio of the kth time slice, r i↑ is the upload bit rate of sender i calculated in the first-level game S1, is the total download bandwidth purchased by the receiver in the last k time slices; In order to solve this optimization problem, the Lagrange multiplier method is introduced to construct the corresponding Lagrange function: The following Lagrangian function and KKT conditions are assumed: 1) Stability conditions: 2) Original feasibility conditions: 3) Complementary relaxation conditions: 4) Dual feasibility condition: α ij ≥0,β ij ≥0,for i∈I; Based on the above conditions, the optimal solution η of the optimization problem is derived ij , η ij The value of is divided into the following three cases: if And α ij =0, then if And α ij =0, then otherwise, 4. The large-scale multi-party interactive real-time video stream transmission method according to claim 1, characterized in that: The solution of the two-layer Stackelberg game model is as follows: The steps to solve the first-level Stackelberg game include: (1.1) Input parameters: including the sender's parameter set {a i },{b i }, And the maximum upstream bandwidth unit price Total upstream bandwidth W ↑ , the number of senders m and the convergence threshold ∈; (1.2) Initialization: The server initially sets the upstream bandwidth unit price to The convergence flag is set to flase, the iteration counter t is set to 1, and the initialization bit rate vector is initialized to a zero vector; (1.3) Iteration process: For each sender i, calculate its optimal bit rate, which is calculated based on the sender's parameters and the current bandwidth unit price. The formula is: Check whether the sum of the bit rates of all senders exceeds the available uplink bandwidth W↑. If so, scale the bit rates of each sender proportionally to meet the bandwidth constraint, i.e. Check the convergence of the bit rate, that is, whether the difference between the bit rate vectors of the current iteration and the previous iteration is less than the threshold ∈; if the condition is met, mark the convergence; otherwise, update the iteration counter and continue the calculation; After each iteration, the server updates the uplink bandwidth unit price p based on the current bit rate allocation. u for To maximize its own utility F; (1.4) Output results: The optimal uplink bit rate and optimal unit price after final convergence are used as the output of the algorithm; The steps to solve the second-level Stackelberg game include: (2.1) Input parameters: including the receiver's parameter set {x ij },{y ij },{z ij }, the number of receivers n, the number of senders m, the total system bandwidth STV, the maximum price limit PPT, and the convergence threshold ∈; (2.2) Initialization: The server initially sets the downlink bandwidth unit price to The convergence flag is set to false, the iteration counter t is set to 1, and the uplink bit rate vector is initialized to the value obtained from algorithm S1. (2.3) Iteration process: For each receiver j, calculate its optimal bandwidth allocation based on the current uplink bit rate and bandwidth allocation; Check whether the sum of bandwidth allocations of all receivers is within the allowed downlink bandwidth range. If it exceeds, adjust the bandwidth allocation proportionally to meet the constraint, i.e., Check the convergence of bandwidth allocation, that is, whether the difference between the bandwidth allocation vector of the current iteration and the previous iteration is less than the threshold ∈. If the condition is met, mark the convergence; otherwise, update the iteration counter and continue the calculation; After each iteration, the server updates the downlink bandwidth unit price p based on the current bandwidth allocation. d , in order to maximize its own utility; (2.4) Output result: The optimal downstream bandwidth allocation W↓ and the optimal unit price p after final convergence d as the output of the algorithm.

5. The large-scale multi-party interactive real-time video stream transmission method according to claim 1, characterized in that: The server utility functions in the two-layer Stackelberg game are the revenue obtained from selling upstream and downstream bandwidth, which are defined as follows: For the upstream bandwidth utility function: For the upstream bandwidth utility function: Among them, p u Indicates the unit price of upstream bandwidth, p d Indicates the unit price of downstream bandwidth, r i↑ represents the upload bandwidth / bit rate of sender i, W j↓ represents the downlink bandwidth of receiver j, r↑ is the set of upload bit rates of all senders, and W ↓ is the set of downlink bandwidths of all receivers, and J is the set of all receivers, including all senders and viewers included in the receivers.

Images