Multi-channel video code rate allocation optimization method based on multi-agent reinforcement learning
By optimizing the bitrate allocation of multiple video streams using a multi-agent reinforcement learning model, the problem of insufficient foresight and accuracy in bitrate allocation of traditional methods in multi-video stream scenarios is solved, achieving more efficient resource allocation and overall quality improvement.
Patent Information
- Application Number
- CN202511502105.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-17
- Publication Date
- 2026-01-23
AI Technical Summary
Traditional video bitrate control methods cannot effectively handle the priorities, scene complexity, user viewing experience, and bandwidth limitations of different video streams in multi-stream scenarios. This results in a lack of foresight and accuracy in bitrate allocation strategies, insufficient dynamic adaptive capabilities, and low overall quality and bandwidth utilization.
We employ a multi-agent reinforcement learning approach, combining historical feedback information and future frame prediction. Through a multi-agent deep deterministic policy gradient algorithm, we optimize the bitrate allocation of multiple video streams. We utilize an Actor-Critic network structure for centralized training and distributed execution, dynamically adjusting the bitrate allocation of each video stream to achieve more refined and forward-looking resource allocation.
It improves the foresight and accuracy of video content complexity measurement, realizes intelligent collaboration and dynamic optimization between multiple video streams, and optimizes overall quality and bandwidth utilization.
Smart Images
Figure CN121397221A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of video coding rate control, and particularly relates to a multi-agent reinforcement learning-based multi-path video rate allocation optimization method. BACKGROUND
[0002] With the rapid development of Internet technology, whether it is streaming live, video conference, or security monitoring and virtual reality application, the demand for concurrent transmission of multi-path video streams is growing. Traditional video rate control methods usually optimize single-path video streams, and the goal is to adjust the video coding parameters in real time according to the changes in network bandwidth to achieve a dynamic balance between transmission quality and bandwidth utilization. However, the multi-path video stream scenario is more complex, and it needs to handle the resource allocation problem between multiple data streams. The priority, scene complexity, user viewing experience, and bandwidth limitation of different video streams need to be considered comprehensively. This complexity makes the traditional single-path rate control method unable to be directly applied to the multi-path video transmission scenario.
[0003] The prior art, multi-path rate control technology based on priority allocation, adopts priority scheduling and statistical multiplexing strategies to effectively balance the utilization of bandwidth resources and the quality of service of each path stream, thereby meeting the needs of different video streams and maximizing the utilization efficiency of network resources. Intelligent rate control technology based on machine learning and deep learning takes data-driven as the core, and through real-time prediction and learning, further improves the dynamic adaptability of multi-path video rate control.
[0004] These methods, although deliberately control the rate to improve the transmission efficiency of video streams, still have the following shortcomings:
[0005] 1. The content complexity metric is not accurate enough, and the feedback information cannot effectively depict the content of future frames;
[0006] 2. The rate allocation strategy lacks foresight and accuracy, and may cause excessive compression or insufficient allocation;
[0007] 3. The dynamic adaptive capability is insufficient, and the overall quality of multi-path video streams and bandwidth utilization are low.
[0008] Therefore, a new method is needed to combine video content features and time series prediction technology to accurately depict the complexity of the current frame while predicting the encoding requirements of future frames, so as to achieve more precise and forward-looking rate allocation and maximize the overall bandwidth utilization efficiency. SUMMARY
[0009] In order to solve the above technical problems existing in the prior art, the purpose of the present application is to combine historical feedback information and future frame prediction information, improve the foresight and accuracy of code rate allocation, enhance the code rate allocation capability in the multi-channel video coding environment, and optimize the overall quality and bandwidth utilization of multi-channel video stream, and the technical scheme is as follows:
[0010] A multi-agent reinforcement learning-based multi-channel video code rate allocation optimization method, comprising the following steps:
[0011] Step 1: Collecting video streams, a plurality of cameras collect video streams in real time, and the collected video streams are transmitted to corresponding X265 encoders, which are coordinated by a global resource allocator, and each X265 encoder independently processes a video stream;
[0012] Step 2: Preliminary encoding of video streams and extraction of state information, the X265 encoder encodes the received video streams in groups, and for each group of video encoding sequences in the same time interval, after all frame encodings in the complete set TGOP of sequence pictures encoded at a fixed time interval, the current encoding state S k Related state information is collected from the environment composed of X265 encoders, and a high-dimensional state vector V k is formed by combining to serve as the input of the reinforcement learning model;
[0013] Step 3: Reinforcement learning model decision-making, i.e., code rate allocation action generation, according to the input high-dimensional state vector V k , the Actor network allocates bit rates for each video stream in the next group of TGOP, and dynamically outputs the code rate allocation action of the next group of TGOP;
[0014] Step 4: Assigning bit rates and re-encoding video streams, the joint bit allocator receives the code rate allocation action and converts it into a specific bit rate, which is passed to the corresponding X265 encoder, and the X265 encoder formally encodes the video streams in the next group of TGOP according to the assigned specific bit rate;
[0015] Step 5: Encoding completion and feedback reward, after the encoding is completed, the data stream is transmitted to the joint buffer, at the same time, the structural similarity index SSIM of the quality evaluation index of the output video is extracted, and the reward signal of the video stream is fed back by the environment composed of X265 encoders to the Critic network of the reinforcement learning model for centralized learning and policy evaluation to optimize the bit rate allocation strategy of the Actor network.
[0016] Further, in step 2, the current encoding state S kThe related state information includes: the average quantization parameter QP value of the i-th video sequence in the k-th group of TGOP, the SATD value of the average encoded frame, the average structural similarity value SSIM, and the average SATD value of the future frame predicted from the forward prediction module of the X265 encoder, which are combined to form a high-dimensional vector V k is represented as:
[0017]
[0018] Further, in step 3, the algorithm used by the reinforcement learning model is a multi-agent deep deterministic policy gradient algorithm.
[0019] Further, the framework used by the multi-agent deep deterministic policy gradient algorithm is centralized training and distributed execution, specifically including the following phases:
[0020] (1) Training phase
[0021] All agents share a centralized Critic network, which obtains the state and action information of all video streams, calculates the global reward function, and optimizes the overall resource allocation strategy. The independent Actor network corresponding to each video stream is responsible for dynamically outputting the bit rate allocation action according to the environment state;
[0022] (2) Execution phase
[0023] The Actor network allocates bit rates for each video stream in the next group of TGOP based on the input state vector. The action is represented in the form of a proportion. The bit rate allocation action is passed to each X265 encoder through a joint bit allocator to determine the specific encoding bit number of each video stream.
[0024] Further, the multi-agent deep deterministic policy gradient algorithm uses an Actor-Critic network structure for optimization, and the specific optimization method is as follows:
[0025] (1) Actor network
[0026] The input of the Actor network is the state information of the video stream. The time sequence features are extracted through the GRU layer, and the spatial features are extracted through the two-dimensional convolution network. After the two parts of the features are fused, they are processed through the full connection layer. The full connection layer has two layers of Leaky ReLU function activation network, and the last layer uses the Sigmoid activation function to limit the output to the range of (0, 1), generating a probability distribution of bit rate allocation as the action output;
[0027] (2) Critic network
[0028] The Critic input is state information and action information generated by the Actor network, and after fusing the two features of state information and action information, the state value function is generated by sequentially passing through two layers of Leaky ReLU function activated full connection layer and a linear output layer, and the Critic network evaluates the global effect of the current policy through the reward signal, and updates the policy evaluation model, at the same time, the Actor network updates its parameters to generate a better bit rate allocation action based on the feedback of the optimized Critic network;
[0029] The Actor network and the Critic network are trained cooperatively.
[0030] Further, the composition of the reinforcement learning model in step 3 includes: state space, action selection, reward function.
[0031] Further, the state space includes: feedback information, prediction information, and video content complexity metric parameter information.
[0032] Further, the feedback information includes: the average quantization parameter QP of the encoded GOP, the average SSIM, and the usage rate of the code rate.
[0033] Further, the reward function is represented by the minimum distortion variance composed of the SSIM index, and the reward feedback is represented by:
[0034]
[0035] The above formula r(S k ) represents the total reward value of all paths under the current encoding state, M represents M video sequences, represents the distortion degree of the i-th video stream in the k+1-th group of TGOP, represents the average distortion of all encoded frames in the k+1-th group of TGOP.
[0036] Beneficial effects: (1) The forward-looking and accuracy of the video content complexity metric are improved. (2) The intelligent collaboration and dynamic optimization between multiple video streams are realized. (3) The overall quality and bandwidth utilization of multiple video streams are optimized. BRIEF DESCRIPTION OF DRAWINGS
[0037] Figure 1 It is a model framework diagram of the application;
[0038] Figure 2 It is a feature extraction flowchart. DETAILED DESCRIPTION
[0039] The specific embodiments of the application will be described in detail below with reference to the accompanying drawings.
[0040] The multi-agent reinforcement learning-based multi-path video code rate allocation optimization method of the application utilizes a reinforcement learning model to realize intelligent resource allocation of multi-path video streams.
[0041] The state space in the reinforcement learning model is composed of three key information, i.e., feedback information (such as average QP of the coded GOP, average SSIM, and usage rate of the code rate), prediction information (average SATD value of a fixed number of future frames extracted through a forward prediction module of X265), and parameters related to the complexity of the video content, to comprehensively reflect the dynamic characteristics of the video content and the state of the network resources, and provide multi-dimensional support for the state representation of the reinforcement learning model. The action selection is the output of the reinforcement learning model, which is used to dynamically adjust the code rate proportion of each video stream to realize the optimized allocation of bandwidth resources among the multi-path streams. The reward function is designed by minimizing the distortion variance of the multi-path video, and is specifically based on the SSIM index to measure the consistency and fluency of the video quality, so as to ensure the optimal and balanced overall video quality. Through the framework of the state space, action selection, and reward feedback, the reinforcement learning model can learn the optimal code rate allocation strategy in the dynamic network environment, preferentially guarantee the quality of the key stream, and improve the bandwidth resource utilization efficiency and overall user experience.
[0042] State set: for the state space, after encoding all the frames in a group of TGOP, the average QP, average SATD of the coded frames, average SSIM, and average SATD value of the future frames obtained through the forward prediction module of the i-th video sequence in the k-th group of TGOP are collected from the environment composed of the X265 encoder, these statistical data are continuously fed into the joint rate controller as part of the state space parameters to become the state set, and the parameter information of the state set is shown in Table 1:
[0043]
[0044] Table 1 State set parameter information
[0045] Action set: action is the action of the agent according to the current encoding state S k to allocate the bit rate for the (k+1)-th group of TGOP. The action output is defined as , which represents the bit rate proportion allocated to the i-th sequence in the (k+1)-th group of TGOP.
[0046] Reward function: the feedback function of reward is designed as a minimum distortion variance problem composed of the SSIM index. The structural similarity index SSIM is an index for evaluating the quality of an image, which is used to measure the perceived quality of an image in the compression or encoding process. The feedback reward of the environment composed of the X265 encoder, input with the given current encoding state S k is designed as,
[0047]
[0048] r(S k ) represents the total reward value of all paths under the current encoding state, M represents M paths of video sequences, represents the distortion of the i-th video stream in the k+1-th group of TGOP, represents the average distortion of all encoded frames in the k+1-th group of TGOP.
[0049] As Figure 1 shown, the workflow of multi-path video stream encoding and resource optimization allocation, X265 is selected as the X265 encoder, which is used as a customized Gym environment for training and testing, the specific steps are as follows:
[0050] Step 1: Video stream acquisition and initialization. Multiple cameras (video sequence 1, video sequence 2,..., video sequence N) collect video streams in real time. These video sequences are transmitted to the corresponding X265 encoder. Coordinated by the global resource allocator, each X265 encoder independently processes a video stream. Use GOP to represent the pictures of the sequence encoded at a fixed time interval, and use TGOP to represent all the GOPs of the sequences encoded in the same time interval. In the first TGOP, i.e. a group of video frames, the bit rate is allocated to all sequences as the initial allocation.
[0051] Step 2: Video stream preliminary encoding and state information extraction. Each X265 encoder encodes the received video stream in groups. After all the frames in the k-th group of TGOP are encoded, the system collects the information related to the current encoding state S k from the environment. These features include: the average QP value (quantization parameter) of the i-th video sequence in the k-th group of TGOP; the average SATD value; the average SSIM value (structural similarity); the average SATD value of the future frame predicted by the forward prediction module in X265 (used to predict motion complexity and texture changes). These information are combined to form a high-dimensional state vector V k , which is used as the input of the actor network of the reinforcement learning model. These multi-dimensional information are combined to form a high-dimensional state vector, which fully describes the dynamic characteristics of each group of TGOP, which is used as the input of the reinforcement learning model. The state is represented as follows:
[0052]
[0053] Step 3: Reinforcement learning model decision making, i.e. bit rate allocation action generation. The reinforcement learning model (multi-agent deep deterministic policy gradient algorithm) adopts a centralized training and distributed execution framework, and generates bit rate allocation actions according to the input high-dimensional state vector V k, the Actor network allocates bitrates for each video stream in the next set of TGOP (i.e. the k+1th set) and dynamically outputs the bitrate allocation action for the next set of TGOP Action The proportion of resources allocated to the ith video sequence is represented in the form of a proportion. This step realizes the decision of resource optimization allocation, i.e. the bitrate allocation action.
[0054] Step 4: Allocate bitrates and re-encode video streams, the joint bitrate allocator receives the bitrate allocation action and converts it into a specific bitrate, which is passed to the corresponding X265 encoder. The X265 encoder formally encodes the video streams in the next set (k+1th set) of TGOP according to the allocated specific bitrate.
[0055] Step 5: Encoding completion and reward feedback. After encoding is completed, the data stream is transmitted to the joint buffer. At the same time, the system extracts the quality evaluation index of the output video, i.e. the structural similarity index SSIM. The reward function (minimum distortion variance based on SSIM) is used to measure the balance and overall visual quality of the encoding quality. The reward signal of each video stream is fed back to the reinforcement learning model (Critic network) composed of X265 encoders for centralized learning and policy evaluation to optimize the bitrate allocation strategy of the Actor network.
[0056] The multi-agent reinforcement learning-based multi-path video bitrate allocation optimization method uses a reinforcement learning model. The reinforcement learning model uses a multi-agent deep deterministic policy gradient algorithm to train the neural network. The Critic part of the network is optimized to learn the state value function Q(S k ) given the current encoding state S k ). The Actor part is considered as a policy that selects the best action according to the current state. The multi-agent deep deterministic policy gradient algorithm is optimized through the Actor-Critic network structure, so the Actor-Critic fully connected network is used to solve the multi-path bitrate allocation problem.
[0057] As shown in Figure 2 , the input of the Actor network is the state information of the video stream. The time sequence features are extracted through the GRU layer, and the spatial features are extracted through the two-dimensional convolution network. After the two parts of features are fused, they are processed through the fully connected layer. The fully connected layer includes two layers of Leaky ReLU activated network for mapping the feature space. The last layer uses the Sigmoid activation function to limit the output to the range of (0, 1), generating a probability distribution of bitrate allocation as the action output.
[0058] The Critic network is used to evaluate the pros and cons of the Actor network strategy, and its input includes state information and action information generated by the Actor network. After the state and action features are fused, they are sequentially passed through two fully connected layers with Leaky ReLU activation and a linear output layer to generate a state value function for evaluating the performance of the current strategy in the global system. The output of the Critic network is only used to guide the optimization of the Actor network strategy during training, while the output of the Actor network is used for actual bit rate allocation decisions. Through the collaborative training of the two, balanced optimization of video quality and efficient allocation of bit resources are achieved.
[0059] Through the reward signal of the previous step, the Critic network evaluates the global effect of the current strategy, updates the policy evaluation model, and ensures the efficiency and fairness of global resource allocation. At the same time, the Actor network updates its parameters based on the feedback from the optimized Critic network to generate better bit rate allocation actions. In this process, the reinforcement learning model continuously learns the changing rules in the dynamic network environment and constantly optimizes the allocation strategy. Through multiple rounds of training and real-time feedback, the model gradually converges to the optimal strategy, achieving intelligent resource allocation in the multi-path video rate control scenario.
[0060] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the implementation process of the present application has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or replace some of the technical features with equivalent ones. Any modification, equivalent replacement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A multi-path video code rate allocation optimization method based on multi-agent reinforcement learning, characterized in that, The method comprises the following steps: Step 1: Collecting video streams, multiple cameras collect video streams in real time, and the collected video streams are transmitted to corresponding X265 encoders for coordination through a global resource allocator, and each X265 encoder independently processes a video stream; Step 2: Preliminary encoding of video stream and extraction of state information, the X265 encoder encodes the received video stream in groups, encodes each group of video encoding sequence in the same time interval, and after the encoding of all frames in the entire set TGOP of sequence pictures encoded in fixed time intervals in the video encoding sequence in the same time interval is completed, the current encoding state S k related state information is collected from the environment composed of the X265 encoder k , as the input of the reinforcement learning model; Step 3: Reinforcement learning model decision-making, i.e., code rate allocation action generation, according to the input high-dimensional state vector V k The Actor network allocates bit rates for each video stream in the next set of TGOPs and dynamically outputs the code rate allocation action of the next set of TGOPs. Step 4: Distributing bit rates and secondary coding the video streams, the joint bit allocator receives the code rate distribution action and converts it into a specific bit rate, which is delivered to the corresponding X265 encoder, and the X265 encoder formally encodes the video streams in the next group of TGOP according to the allocated specific bit rate; Step 5: Encoding completion and reward feedback, after the encoding is completed, the data stream is transmitted to the joint buffer, at the same time, the structural similarity index SSIM of the quality evaluation index of the output video is extracted, the reward signal of the video stream is fed back to the Critic network of the reinforcement learning model by the X265 encoder group, centralized learning and policy evaluation are performed to optimize the bit rate allocation strategy of the Actor network.
2. The multi-agent reinforcement learning based multi-path video bit rate allocation optimization method according to claim 1, characterized in that, In step 2, the current coding state S k The related state information includes: the average quantization parameter QP value of the i-th video sequence in the k-th group TGOP, the average SATD value of the encoded frame, the average structural similarity value SSIM, and the average SATD value of the future frame predicted from the forward prediction module of the X265 encoder, which are combined to form a high-dimensional vector V k is represented as:
3. The multi-agent reinforcement learning based multi-path video bit rate allocation optimization method according to claim 1, characterized in that, In step 3, the algorithm used by the reinforcement learning model is a multi-agent deep deterministic policy gradient algorithm.
4. The multi-agent reinforcement learning based multi-path video code rate allocation optimization method according to claim 3, characterized in that, The framework used by the multi-agent deep deterministic policy gradient algorithm is centralized training and distributed execution, and specifically includes the following phases: (1) Training phase All agents share a centralized Critic network, the centralized Critic network obtains the state and action information of all video streams, calculates the global reward function, optimizes the overall resource allocation strategy, and the independent Actor network corresponding to each video stream is responsible for dynamically outputting the code rate allocation action according to the environment state; (2) Execution phase The Actor network allocates bit rates for each video stream in the next group of TGOP according to the input state vector, the action represents resource allocation in the form of a proportion, and the bit rate allocation action is delivered to each X265 encoder through the joint bit allocator to determine the specific encoding bit number of each video stream.
5. The multi-agent reinforcement learning based multi-path video code rate allocation optimization method according to claim 3 or 4, characterized in that, The multi-agent deep deterministic policy gradient algorithm uses an Actor-Critic network structure for optimization, and the specific optimization method is as follows: (1) Actor network The input of the Actor network is the state information of the video stream, the time sequence features are extracted through the GRU layer, and the spatial features are extracted through the two-dimensional convolution network, and after the two features are fused, they are processed through the full connection layer, the full connection layer has two Leaky ReLU function activated networks, and the last layer uses the Sigmoid activation function to limit the output to the range of (0, 1), and generates a probability distribution of bit rate allocation as the action output; (2) Critic network The Critic input is state information and action information generated by the Actor network, and after the two features are fused, they are sequentially processed through two Leaky ReLU function activated full connection layers and a linear output layer to generate a state value function, through the reward signal, the Critic network evaluates the global effect of the current strategy, updates the policy evaluation model, and at the same time, the Actor network updates its parameters based on the feedback of the optimized Critic network to generate a better bit rate allocation action; Both Actor network and Critic network are trained collaboratively.
6. The multi-agent reinforcement learning based multi-path video bit rate allocation optimization method according to claim 1, characterized in that, The composition of the reinforcement learning model in step 3 includes: state space, action selection, reward function.
7. The multi-agent reinforcement learning based multi-path video code rate allocation optimization method according to claim 6, characterized in that, The state space includes: feedback information, prediction information, and video content complexity metric parameter information.
8. The multi-agent reinforcement learning based multi-path video code rate allocation optimization method according to claim 7, characterized in that, The feedback information includes: the average quantization parameter QP of the encoded GOP, the average SSIM, and the proportion of the code rate.
9. The multi-agent reinforcement learning based multi-path video code rate allocation optimization method according to claim 6, characterized in that, The reward function is represented by the minimum distortion variance composed of the SSIM index, and the reward feedback is represented by: r(S k represents the total reward value of all paths under the current encoding state, and M represents the M video sequences, represents the distortion degree of the i-th video stream in the k+1-th group of TGOPs, represents the average distortion degree of all encoded frames in the k+1-th group of TGOPs.