A joint optimization method and apparatus for real-time video stream transmission

By improving the reference relation structure and deep reinforcement learning algorithm, the problem of frame error propagation in real-time video streaming transmission is solved, achieving low-latency and high-efficiency video transmission and improving user experience.

CN123123650APending Publication Date: 2026-11-17CAPITAL NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610923764.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing real-time video streaming methods, the transport layer and the encoding/decoding layer are independent of each other, making it impossible to distinguish between critical reference frames and discardable frames. This leads to a vicious cycle of error propagation and transmission rigidity, affecting the user experience.

Method used

An improved reference relationship structure is used to encode video frames, dividing them into reference frames and non-reference frames. The encoded data is classified into necessary data and discardable data. Necessary data is delivered through a fully reliable transport stream, and discardable data is actively discarded by cyclically resetting the stream. The encoding and transmission parameters are adjusted by combining deep reinforcement learning algorithms.

Benefits of technology

It significantly reduces end-to-end latency, enables rapid error recovery, and improves user experience quality, especially performing exceptionally well under network fluctuation conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123650A_ABST
    Figure CN123123650A_ABST
Patent Text Reader

Abstract

This invention discloses a joint optimization method and apparatus for real-time video stream transmission, belonging to the field of multimedia video transmission technology. The transmitting end employs an improved reference structure to encode video, distinguishing between reference and non-reference frames and classifying data into necessary and discardable categories. The transmission link uses dual-channel distribution; necessary data is transmitted via a reliable stream, while discardable data is automatically discarded upon expiration of the cyclic reset stream. The receiving end completes data decoding and sends back image quality, encoding, and network metrics to the control module. The controller, relying on deep reinforcement learning, comprehensively analyzes feedback information to output commands for coordinated adjustment of bitrate, playback cutoff threshold, and time slice interval, and issues dynamic adjustments to the encoding and transmission strategy to the transmitting end. This invention hierarchically distinguishes data importance for differentiated transmission and combines intelligent algorithms to achieve cross-stage joint optimization, balancing video transmission reliability and network bandwidth utilization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimedia video transmission technology, and in particular to a joint optimization method and apparatus for real-time video stream transmission. Background Technology

[0002] Real-time video streaming serves as a critical infrastructure for remote collaboration, telemedicine, and immersive interactive applications. Its core challenge lies in controlling end-to-end latency to within 150 milliseconds, despite the severe jitter, random packet loss, and unpredictable bandwidth fluctuations prevalent in wireless and mobile networks. Related technologies utilize the collaborative operation of video coding techniques and network transmission protocols to construct a complete system from encoding compression to adaptive transmission. Specifically, this system encompasses key aspects such as video frame encoding, adaptive bitrate, retransmission, and forward error correction (FEC). The coding layer achieves compression efficiency through a rigid group of pictures (GOP) dependent structure, while the transport layer independently adjusts retransmission and redundancy strategies based on network conditions.

[0003] However, existing real-time video streaming methods directly employ an optimization paradigm where the transport layer and codec layer operate independently, without considering the semantic gap between them. This can lead to the transport protocol being unable to see the video semantics, failing to distinguish between critical reference frames and discardable frames, and resorting to indiscriminate retransmission or indiscriminate discarding, thus triggering a vicious cycle of error propagation and transmission rigidity. Specifically, when a reference frame is corrupted, decoding errors cascade along the entire dependency chain, causing persistent visual artifacts; while aggressive retransmissions or FEC at the transport layer inject additional traffic into an already saturated network, exacerbating congestion and further causing packet loss, ultimately severely impacting the user experience of real-time interaction. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a joint optimization method for real-time video streaming transmission.

[0006] Another objective of this invention is to provide a joint optimization device for real-time video streaming.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a joint optimization method for real-time video stream transmission, comprising:

[0010] S10, the transmitting end uses an improved reference relationship structure to encode the video, divides the video frames into reference frames and non-reference frames, and classifies the encoded data into necessary data and discardable data; S20, the transmission component classifies the data, delivers the necessary data through a fully reliable transmission stream, and actively discards the discardable data when the playback deadline is exceeded by cyclically resetting the stream; S30: The receiver decodes the received usable data and feeds back the quality indicators, encoding parameters and network performance data to the controller. S40, the controller is based on a deep reinforcement learning algorithm, which generates a joint control signal of encoding bitrate, playback deadline threshold and time slice interval according to feedback data, and transmits it to the sending end to adjust subsequent encoding and transmission parameters.

[0011] In one embodiment of the present invention, S10 includes: During the encoding process, a selective reference frame selection mechanism is introduced, allowing only a portion of P-frames to serve as references for subsequent frames, thereby dividing P-frames into reference P-frames and non-reference P-frames. By replacing some I-frames with reference P-frames, the frequency of I-frame occurrence is strategically reduced; by explicitly verifying the integrity of all reference frames, there is no need to periodically insert I-frames to constrain error propagation.

[0012] In one embodiment of the present invention, it further includes: The reference frame selection mechanism is determined by an optimization model, which minimizes the total size of the reference frame under the constraint of not exceeding the bandwidth consumption of traditional coding. The total volume of the reference frames is the sum of the volume of the I-frames and the volume of the reference P-frame set. The constraint is that the total volume of the video stream encoded using the improved reference relationship structure does not exceed the total frame volume of the traditional encoding using the default reference structure.

[0013] In one embodiment of the present invention, S20 includes: Necessary data is routed to the necessary stream pool, which carries persistent connections throughout the entire session to ensure full reliability. Data is distributed between streams using a round-robin scheduling method to maximize parallelism and maintain decoding order.

[0014] In one embodiment of the present invention, it further includes: Discardable data is routed to a discardable stream pool, and packets are distributed to each stream at configurable block intervals, with each distribution updating the stream's deadline. The stream reset thread continuously monitors the deadline of each stream. When the deadline of a stream is reached, the stream is terminated and a replacement stream is generated to maintain the capacity of the stream pool.

[0015] In one embodiment of the present invention, S30 includes: The quality metrics fed back by the receiver include the quality mean and quality variance; the encoding parameters include the encoding bitrate, the playback deadline threshold, and the time slice interval; the network performance data includes bandwidth, end-to-end latency, and packet loss rate; and the feedback data also includes buffer level and cumulative packet drop rate.

[0016] In one embodiment of the present invention, S40 includes: The controller aggregates multi-dimensional state observations from several recent time intervals, including network performance metrics, video coding metrics, and historical performance metrics. Network performance metrics include bandwidth, end-to-end latency, and packet loss statistics; video coding metrics include frame size and GOP position; and historical performance metrics include mean quality, quality variance, buffer level, and cumulative drop rate.

[0017] To achieve the above objectives, a second aspect of the present invention provides a joint optimization apparatus for real-time video stream transmission, comprising: The frame layer coding classification module is used by the sending end to encode video using an improved reference relationship structure, dividing video frames into reference frames and non-reference frames, and classifying encoded data into necessary data and discardable data. The differentiated dual-stream transmission scheduling module is used by the transmission component to classify data, deliver necessary data through a fully reliable transmission stream, and actively discard discardable data when the playback deadline is exceeded by cyclically resetting the stream. The receiver decoding status feedback module is used to decode the received usable data and feed back the quality indicators, encoding parameters and network performance data to the controller. The reinforcement learning joint parameter control module is used by the controller to generate joint control signals for the encoding bitrate, playback deadline threshold, and time slice interval based on feedback data using a deep reinforcement learning algorithm, and transmit them to the transmitting end to adjust subsequent encoding and transmission parameters.

[0018] The present invention provides a joint optimization method and apparatus for real-time video stream transmission, which can significantly reduce end-to-end latency of real-time video transmission, achieve rapid error recovery, and improve user experience quality under network fluctuation conditions.

[0019] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the method described in the first aspect embodiment.

[0020] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] Figure 1 This is a flowchart of a joint optimization method for real-time video stream transmission according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a joint optimization method for real-time video stream transmission according to an embodiment of the present invention; Figure 3 This is an improved video frame reference relationship structure diagram according to an embodiment of the present invention; Figure 4 This is a flowchart of a joint optimization system for real-time video stream transmission according to an embodiment of the present invention; Figure 5 This is a structural diagram of a joint optimization device for real-time video stream transmission according to an embodiment of the present invention; Figure 6 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] The following description, with reference to the accompanying drawings, describes a joint optimization method and apparatus for real-time video stream transmission according to an embodiment of the present invention.

[0026] Figure 1 This is a flowchart of a joint optimization method for real-time video stream transmission according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S10, the transmitting end uses an improved reference relationship structure to encode the video, divides the video frames into reference frames and non-reference frames, and classifies the encoded data into necessary data and discardable data; S20, the transmission component classifies the data, delivers the necessary data through a fully reliable transmission stream, and actively discards the discardable data when the playback deadline is exceeded by cyclically resetting the stream; S30: The receiver decodes the received usable data and feeds back the quality indicators, encoding parameters and network performance data to the controller. S40, the controller is based on a deep reinforcement learning algorithm, which generates a joint control signal of encoding bitrate, playback deadline threshold and time slice interval according to feedback data, and transmits it to the sending end to adjust subsequent encoding and transmission parameters.

[0027] Furthermore, embodiments of the present invention propose a joint optimization system for real-time video stream transmission, such as... Figure 2 , Figure 3 and Figure 4 As shown: S1. The transmitting end uses an improved reference relationship structure to encode video, dividing P-frames into reference P-frames and non-reference P-frames, thereby classifying the encoded data into necessary data and discardable data.

[0028] S2. The sending end submits the classified data to the centralized sending queue of the transmission component, and each data packet is marked as "necessary" or "discardable" according to the frame type.

[0029] S3. The distribution thread routes packets to the necessary flow pool (fully reliable delivery) or the discardable flow pool (actively discarded by the flow reset thread when expired) according to the category.

[0030] S4. The receiver decodes the usable data and sends feedback information such as quality score, encoding parameters, and network data back to the controller.

[0031] The S5.SAC reinforcement learning controller summarizes the network, encoding, and historical performance metrics to form a multi-dimensional state observation.

[0032] S6. The controller derives the optimal encoding bitrate, playback deadline threshold, and time slice interval through a joint adaptive algorithm based on SAC, and passes them to the sender for encoding and transmission of the next interval.

[0033] Furthermore, in step S1, the sending end establishes a hierarchical frame dependency structure through an improved reference frame selection mechanism, specifically including the following sub-steps: S100. I-frames are independently encoded; a reference frame selection mechanism is selectively introduced during the encoding process, allowing only a portion of P-frames to serve as references for subsequent frames, thereby dividing P-frames into reference P-frames and non-reference P-frames.

[0034] S101. During encoding, dynamically determine which P-frames can be used as references to ensure that only I-frames and a small number of strategically selected reference P-frames are needed to decode subsequent frames; transmission errors affecting non-reference P-frames are isolated within the frame itself and will not cascade to the rest of the entire GOP.

[0035] S102. To alleviate the bandwidth expansion caused by the structural modification, some I-frames are replaced with reference P-frames, strategically reducing the frequency of I-frame occurrence; since the integrity of all reference frames is explicitly verified, there is no need to periodically insert I-frames to constrain error propagation.

[0036] Furthermore, the reference frame selection optimization model in step S1 is as follows: Let the set of non-reference P frames be... The reference P-frame set is The total size of all frames in a video stream starting with an I-frame is:

[0037] Furthermore, due to the reference frame (I-frame and set) The frames in the reference frame constitute the critical path for error propagation. The optimization objective is to minimize the total size of the reference frame while not exceeding the bandwidth consumption of traditional coding.

[0038] in The total frame size under traditional H.264 encoding using the default reference structure is used as a bandwidth budget reference.

[0039] Furthermore, in step S3, the transmission component implements partially reliable transmission based on DTP (built on top of QUIC), specifically divided into two stream pools: Necessary stream pool: It carries persistent connections throughout the entire session and is dedicated to transmitting I-frames and reference P-frames to ensure full reliability. Data is distributed between streams using round-robin scheduling to maximize parallelism and maintain decoding order, ensuring that packet loss in a single stream does not block the entire transmission pipeline.

[0040] Discardable Stream Pool: A cyclic reset mode is used for non-reference P-frames; packets are allocated to each stream at configurable block intervals, and the deadline of the stream is updated each time it is allocated; the stream reset thread continuously monitors the deadline of each stream, and terminates the stream and derives an alternative stream when the deadline of a stream is reached to maintain the capacity of the stream pool, thereby gracefully discarding old data without the need for application layer intervention.

[0041] Further, in step S4, the receiving end reads the current configuration parameters, specifically including the encoding bitrate (default 2Mbps), playback deadline threshold, and time slice interval; network data includes bandwidth, end-to-end latency, and packet loss rate. After video playback, a user experience score is calculated, and historical performance indicators such as quality mean, quality variance, buffer level, and cumulative drop rate are fed back along with network indicators.

[0042] Furthermore, in step S5, the SAC controller processes 9-dimensional observations from several recent time intervals, including network performance metrics (bandwidth, end-to-end latency, packet loss statistics), video coding metrics (frame size, GOP position), and historical performance metrics (quality mean, quality variance, buffer level, cumulative drop rate).

[0043] Furthermore, in step S6, the controller derives the optimal streaming media parameters using a joint adaptive algorithm based on SAC. The specific calculation steps are as follows: S601. Establish a maximum entropy optimization objective, which maximizes both expected return and policy entropy:

[0044] in The temperature parameter controls the trade-off between maximizing reward and maximizing entropy. For state The entropy of the next strategy.

[0045] S602. Learning Two Soft Q Functions , The overestimation bias is mitigated by pruning the double-Q technique, with the goal of minimizing the soft Bellman residuals.

[0046] in As an experience replay buffer, the value function Implicitly defined as:

[0047] S603. The policy is represented by a Gaussian distribution over the continuous action space, and the policy parameters are updated. To maximize expected return and entropy:

[0048] S604. Temperature Parameters The following optimization autotuning is used, where the target entropy is... For dimension The continuous action space is usually taken as :

[0049] S605. The sampled continuous motion is quantized into the most recent effective bitrate level, playback deadline threshold and time slice interval, and output by the controller to the sending end to complete the joint adjustment of the encoding bitrate, drop control and frame-dependent grouping granularity.

[0050] The action space of this invention includes three main variables: encoding bitrate (determining compression quality and bandwidth consumption), playback deadline threshold (controlling when data packets are deemed expired and can be discarded), and time slice interval (determining the granularity of frame-dependent grouping). Furthermore, the model's generalization ability is primarily determined by network state features rather than video content; as long as the training distribution covers representative network conditions, the learned policy can be transferred to unseen content without retraining.

[0051] In simulation experiments of various real network trajectories, compared with baselines such as SODA, Salsify, and WebRTC, the embodiments of the present invention achieve a 28% to 50% QoS improvement under bandwidth conditions of 1 to 4 Mbps; the error recovery speed is improved by 4 times during traffic bursts and by 5 times during network blackouts.

[0052] To achieve the above embodiments, such as Figure 5 As shown, this embodiment also provides a joint optimization device 10 for real-time video stream transmission, including: The frame layer coding classification module 100 is used by the transmitting end to encode video using an improved reference relationship structure, divide video frames into reference frames and non-reference frames, and classify encoded data into necessary data and discardable data. The differentiated dual-stream transmission scheduling module 200 is used by the transmission component to classify data, deliver necessary data through a fully reliable transmission stream, and actively discard discardable data when the playback deadline is exceeded by cyclically resetting the stream. The receiver decoding status feedback module 300 is used to decode the received usable data and feed back the quality indicators, encoding parameters and network performance data to the controller. The reinforcement learning joint parameter control module 400 is used by the controller to generate joint control signals for the encoding bitrate, playback deadline threshold and time slice interval based on feedback data using a deep reinforcement learning algorithm, and transmit them to the transmitting end to adjust subsequent encoding and transmission parameters.

[0053] The joint optimization device for real-time video stream transmission according to an embodiment of the present invention can significantly reduce end-to-end latency of real-time video transmission and achieve rapid error recovery, thereby improving the quality of user experience under network fluctuation conditions.

[0054] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 6As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0055] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0056] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0057] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A joint optimization method for real-time video stream transmission, characterized in that, include: S10, the transmitting end uses an improved reference relationship structure to encode the video, divides the video frames into reference frames and non-reference frames, and classifies the encoded data into necessary data and discardable data; S20, the transmission component classifies the data, delivers the necessary data through a fully reliable transmission stream, and actively discards the discardable data when the playback deadline is exceeded by cyclically resetting the stream; S30: The receiver decodes the received usable data and feeds back the quality indicators, encoding parameters and network performance data to the controller. S40, the controller is based on a deep reinforcement learning algorithm, which generates a joint control signal of encoding bitrate, playback deadline threshold and time slice interval according to feedback data, and transmits it to the sending end to adjust subsequent encoding and transmission parameters.

2. The method as described in claim 1, characterized in that, S10 includes: During the encoding process, a selective reference frame selection mechanism is introduced, allowing only a portion of P-frames to serve as references for subsequent frames, thereby dividing P-frames into reference P-frames and non-reference P-frames. By replacing some I-frames with reference P-frames, the frequency of I-frame occurrence is strategically reduced; by explicitly verifying the integrity of all reference frames, there is no need to periodically insert I-frames to constrain error propagation.

3. The method as described in claim 2, characterized in that, The method further includes: The reference frame selection mechanism is determined by an optimization model, which minimizes the total size of the reference frame under the constraint of not exceeding the bandwidth consumption of traditional coding. The total volume of the reference frames is the sum of the volume of the I-frames and the volume of the reference P-frame set. The constraint is that the total volume of the video stream encoded using the improved reference relationship structure does not exceed the total frame volume of the traditional encoding using the default reference structure.

4. The method as described in claim 1, characterized in that, S20 includes: Necessary data is routed to the necessary stream pool, which carries persistent connections throughout the entire session to ensure full reliability. Data is distributed between streams using a round-robin scheduling method to maximize parallelism and maintain decoding order.

5. The method as described in claim 1, characterized in that, The method further includes: Discardable data is routed to a discardable stream pool, and packets are distributed to each stream at configurable block intervals, with each distribution updating the stream's deadline. The stream reset thread continuously monitors the deadline of each stream. When the deadline of a stream is reached, the stream is terminated and a replacement stream is generated to maintain the capacity of the stream pool.

6. The method as described in claim 1, characterized in that, S30 includes: The quality metrics fed back by the receiver include the quality mean and quality variance; the encoding parameters include the encoding bitrate, the playback deadline threshold, and the time slice interval; the network performance data includes bandwidth, end-to-end latency, and packet loss rate; and the feedback data also includes buffer level and cumulative packet drop rate.

7. The method as described in claim 1, characterized in that, S40 includes: The controller aggregates multi-dimensional state observations from several recent time intervals, including network performance metrics, video coding metrics, and historical performance metrics. Network performance metrics include bandwidth, end-to-end latency, and packet loss statistics; video coding metrics include frame size and GOP position; and historical performance metrics include mean quality, quality variance, buffer level, and cumulative drop rate.

8. A joint optimization device for real-time video stream transmission, characterized in that, include: The frame layer coding classification module is used by the sending end to encode video using an improved reference relationship structure, dividing video frames into reference frames and non-reference frames, and classifying encoded data into necessary data and discardable data. The differentiated dual-stream transmission scheduling module is used by the transmission component to classify data, deliver necessary data through a fully reliable transmission stream, and actively discard discardable data when the playback deadline is exceeded by cyclically resetting the stream. The receiver decoding status feedback module is used to decode the received usable data and feed back the quality indicators, encoding parameters and network performance data to the controller. The reinforcement learning joint parameter control module is used by the controller to generate joint control signals for the encoding bitrate, playback deadline threshold, and time slice interval based on feedback data using a deep reinforcement learning algorithm, and transmit them to the transmitting end to adjust subsequent encoding and transmission parameters.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.