Real-time video-oriented pseudo-dual-stream video encoding and transmission method and system

Abstract

A pseudo double stream video coding and transmission method for real-time video, comprising: analyzing the real-time video, obtaining the target code rate of the real-time video when a key frame is generated in the real-time video, distributing the target code rate of the real-time video based on a code rate-quantization model to obtain a first code rate stream and a second code rate stream; encoding the first code rate stream and the second code rate stream according to a preset rule to generate two data messages; and storing the two data messages in a Pacer sending queue and sending the data messages to the network according to a preset priority. The application has high compatibility and wide adaptability, does not need to modify the encoder itself, has the high speed and high reliability advantages of the general encoder itself. The application can minimize the long tail delay and the average delay, meet the low delay requirement of various real-time video communication applications, and ensure other index performances through more reliable, low delay and low packet loss transmission.

Description

Technical Field

[0001] This invention relates to the field of video coding and communication technology, specifically to a pseudo-dual-stream video coding and transmission method and system for real-time video. Background Technology

[0002] In recent years, real-time video has rapidly become a popular and promising media format, inspiring unprecedented expectations for entirely new viewing experiences. From phone calls and conferencing to live streaming and cloud gaming, the global real-time video market reached a value of $4.91 billion in 2020 and is projected to continue growing at a CAGR of 10.0% between 2022 and 2028. To deliver unique interactive experiences, optimizing end-to-end (E2E) latency from video capture to playback is crucial, as it directly impacts user engagement and immersion.

[0003] Despite the promising market prospects, the end-to-end latency of real-time video has not yet achieved satisfactory results, exhibiting a significant long-tail distribution in high-latency regions. This is primarily attributed to the substantial differences in video frame sizes. Experimental data shows that in general-purpose real-time video transmission systems based on WebRTC, keyframes are significantly larger than non-keyframes, resulting in significantly higher transmission latency for keyframes compared to non-keyframes. This latency can ultimately far exceed the latency requirements of real-time video applications (e.g., the 200-millisecond latency requirement for video calls). Unlike video-on-demand (VoD), which is less affected, this issue poses a serious challenge to real-time video streaming with stringent latency requirements.

[0004] To optimize latency and mitigate the long-tail effect, existing algorithms cover a variety of methods, from jitter buffering and bitrate adaptation to encoding optimization, but each has its inherent limitations: (1) Jitter buffering algorithms smooth the transmission latency jitter of video frames by introducing waiting time at the receiving end, making video playback smoother and mitigating the long-tail effect, but this increases the overall end-to-end latency of video frames; (2) Video bitrate adaptation algorithms either use manually predefined rules that are more sensitive to latency and packet loss, or use reinforcement learning (RL) to automatically generate strategies that adapt to burst traffic of keyframes. However, these algorithms ultimately sacrifice video bitrate and clarity for the improvement of end-to-end latency; (3) Encoding optimization algorithms either force the keyframe size to be adjusted to the level of non-keyframes, such as a strict constant bitrate (CBR) mode, or minimize the keyframe frequency, or customize the encoder. The first two methods cause the video encoding configuration to deviate from the optimal choice, or affect the clarity of keyframes and subsequent non-keyframes that use them as a reference, or accelerate the accumulation of inter-frame errors, thus affecting video clarity. The last method lacks universality due to compatibility issues with common codecs. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a pseudo-dual-stream video encoding and transmission method and system for real-time video that overcomes or at least partially solves the above problems.

[0006] To address the aforementioned technical problems, the embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, embodiments of the present invention disclose a pseudo-dual-stream video encoding and transmission method for real-time video, comprising:

[0008] S100. Analyze the real-time video. When a keyframe is generated in the real-time video, obtain the target bitrate of the real-time video. Allocate the target bitrate of the real-time video based on the bitrate-quantization model to obtain the first bitrate path stream and the second bitrate path stream.

[0009] S200. Encode the first bit rate stream and the second bit rate stream according to preset rules to generate two data packets;

[0010] S300. The two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to the preset priority.

[0011] Furthermore, in S100, the method for constructing the bitrate-quantization model includes:

[0012] S101. Obtain the frame-level bitrate-quantization model and convert the frame-level bitrate-quantization model into a stream-level bitrate-quantization model;

[0013] S102. Based on the stream-level bitrate-quantization model, model the first bitrate path stream and the second bitrate path stream respectively;

[0014] S103. Based on the parameter variable constraints of the constructed first bitrate path flow and second bitrate path flow models and the model based on the target bitrate, optimize the model to obtain the final first bitrate path flow and second bitrate path flow.

[0015] Furthermore, in S101, the frame-level bitrate-quantization model is as follows:

[0016]

[0017] Where q represents the quantization step size of a frame, c represents the complexity of the frame, which is characterized by the sum of the absolute values ​​of the residual transform coefficients, R represents the amount of data required to encode the frame when the quantization step size is q, and a represents the fitting parameters of the RQ model. During the video encoding process, the model continuously updates the parameter a according to the selected parameters and the actual amount of encoded data in the latest encoded frame, so that the model can adapt to the latest dynamically changing video content and serve as the basis for future dual-stream configuration decisions based on time consistency.

[0018] Further, in S101, the frame-level bitrate-quantization model is converted into a stream-level bitrate-quantization model. Specifically, this involves: using the video frame corresponding to the keyframe as the first frame, and modeling the dual streams from start to end as the basic unit, with a duration of T; pre-modeling the original single stream within this T-time period according to the frame-level bitrate-quantization model, as shown below:

[0019]

[0020] Where b represents the original video stream bitrate, and f represents the original video stream frame rate. For the original video stream resolution, This represents the complexity of the original video. In this basic unit, the first frame is the encoded keyframe with a size of R1, which is a known quantity. All subsequent frames are uncoded non-keyframes with a quantization step size of q. i And complexity c i All are unknown, i∈{2,3,…,Tf}, where Tf represents the number of frames in the unit.

[0021] Further, in S102, the first bitrate stream is modeled according to the stream-level bitrate-quantization model. Specifically, the bitrate of the first bitrate stream is as follows:

[0022]

[0023] Where b′ represents the bit rate of the first bit rate stream and f′ represents the frame rate of the first bit rate stream. For the first bitrate stream clarity, This represents the complexity of the first bitrate stream, which is directly converted from the original single stream, retaining the keyframe R1. However, the difference from the original single stream lies in the significant reduction of the first bitrate stream's frame rate f′ and the expected sharpness of non-keyframes. and complexity The changes.

[0024] Furthermore, in S103, the second bitrate stream is modeled according to the stream-level bitrate-quantization model, and the bitrate of the second bitrate stream is as follows:

[0025]

[0026] Where "b" represents the bit rate of the second bit rate stream, and "f" represents the frame rate of the first bit rate stream. For the first bitrate stream clarity, Let represent the complexity of the first bitrate stream. The first non-critical frame to be replaced needs to maintain the same sharpness q1 as the critical frames in the first stream. Since the frame rate f remains unchanged, its complexity c”1 can also be predicted based on past frames using a linear model. Furthermore, subsequent frames from 2 to Tf need to maintain consistent sharpness and average sharpness.

[0027] Furthermore, in S200, the first bitrate stream and the second bitrate stream are encoded according to preset rules to generate two data packets. The specific method includes: dual-stream encoding is only activated when the original video stream encodes the latest frame as a keyframe according to its default keyframe decision. When generating a keyframe, the video sender immediately creates a secondary encoder to encode the parallel stream. The encoder is essentially a clone of the original encoder. It uses the original stream as the basis to continue the encoding process. Correspondingly, the receiver also activates a secondary decoder and implements a first-decode-first-play strategy on the same frame.

[0028] Furthermore, the first bitrate stream retains the original keyframes and is used to transmit burst traffic generated by intra-frame encoded keyframes. The transmission of these keyframes ensures that decoding or transmission errors do not accumulate rapidly during inter-frame propagation. During the transmission of keyframes in the first bitrate stream, newly captured video frames are not encoded but are skipped extensively until the keyframe transmission is confirmed via RTCP feedback messages. After the keyframe transmission is complete, the latest frame is encoded as a P-frame to continue providing reference functionality. If the P-frame's size is outside a preset range due to its distance from the reference frame, the above process is repeated to obtain the latest P-frame until its size and target bitrate are within the preset range. The second bitrate stream is used for real-time playback and does not consider providing optimal encoding / decoding references for subsequent frames or limiting error propagation. The second bitrate stream is activated when a keyframe appears in the original stream, synchronously encoding the keyframe into a non-keyframe with a smaller data size. Once the second bitrate stream is activated, it will encode subsequently captured video frames, and the video frames will consist entirely of non-keyframes to ensure playback continuity.

[0029] Further, in S300, the two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to a preset priority. The specific method includes: sorting the packets by priority according to their type, with the priorities from high to low being audio packets, retransmission packets, video packets, FEC redundancy packets, and padding packets; and setting the data packets generated by the second bitrate stream encoding to have a higher priority than the data packets generated by the first bitrate stream encoding, preferably transmitting the second bitrate stream. The first bitrate stream is sent when both of the following conditions are met: there are no second bitrate stream data packets to be sent in the Pacer queue and there is still available bitrate budget.

[0030] Secondly, embodiments of the present invention disclose a pseudo-dual-stream video encoding and transmission system for real-time video, comprising: a pseudo-dual-stream encoding bitrate allocation module, a pseudo-dual-stream encoding module, and a transmission priority control module; wherein:

[0031] The pseudo-dual-stream coding bitrate allocation module is used to analyze real-time video. When a keyframe is generated in the real-time video, the target bitrate of the real-time video is obtained, and the target bitrate of the real-time video is allocated based on the bitrate-quantization model to obtain the first bitrate path stream and the second bitrate path stream.

[0032] The pseudo-dual-stream encoding module is used to encode the first bit rate stream and the second bit rate stream according to preset rules to generate two data packets;

[0033] The transmission priority control module is used to temporarily store the two data packets in the Pacer sending queue and send the data packets to the network according to a preset priority.

[0034] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0035] This invention discloses a pseudo-dual-stream video encoding and transmission method for real-time video, which boasts extremely high compatibility and wide adaptability. It can be directly deployed on general-purpose encoders such as H.26x and VPx without requiring modification to the encoder itself, thus possessing the high speed and high reliability advantages inherent in general-purpose encoders. Simultaneously, this invention is perfectly compatible with current mainstream WebRTC-based real-time communication architectures. It minimizes long-tail latency and average latency, meeting the low-latency requirements of various real-time video communication applications (such as video calls, video conferencing, cloud gaming, live video streaming, and vehicle-to-everything (V2X) communication). Furthermore, it ensures other QoE performance metrics, such as video clarity, smoothness, stuttering rate, and frame drop rate, through more reliable, low-latency, and low-packet-loss transmission. This invention can be used in conjunction with a bitrate adaptive algorithm to further enhance video bitrate and clarity. Because this invention provides a stable video bitstream, avoiding increased latency and congestion caused by traffic bursts from keyframes, it can be combined with a bitrate adaptive algorithm to further enhance adaptability and tolerance to wireless communication environments, thereby increasing the average target bitstream of the video.

[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0037] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0038] Figure 1 This is a flowchart of a pseudo-dual-stream video encoding and transmission method for real-time video in Embodiment 1 of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the transmission of two data packets to the network in Embodiment 1 of the present invention.

[0040] Figure 3 This is a structural diagram of a pseudo-dual-stream video encoding and transmission system for real-time video in Embodiment 2 of the present invention. Detailed Implementation

[0041] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0042] To address the problems existing in the prior art, embodiments of the present invention provide a pseudo dual-stream video encoding and transmission method and system for real-time video.

[0043] Example 1

[0044] This invention discloses a pseudo-dual-stream video encoding and transmission method for real-time video, such as... Figure 1 ,include:

[0045] S100. Analyze the real-time video. When a keyframe is generated in the real-time video, obtain the target bitrate of the real-time video. Allocate the target bitrate of the real-time video based on the bitrate-quantization model to obtain the first bitrate path stream and the second bitrate path stream.

[0046] Specifically, when the dual-stream phase is enabled, the pseudo-dual-stream coding module quantitatively allocates the overall target bitrate (determined by other bitrate adaptive algorithms) to the two streams. The goal of this allocation is to ensure that the sum of the bitrates of the two streams does not exceed the overall target bitrate while minimizing the loss of sharpness (and maximizing sharpness).

[0047] In S100 of this embodiment, the method for constructing the bitrate-quantization model includes:

[0048] S101. Obtain the frame-level bitrate-quantization model and convert the frame-level bitrate-quantization model into a stream-level bitrate-quantization model;

[0049] S102. Based on the stream-level bitrate-quantization model, model the first bitrate path stream and the second bitrate path stream respectively;

[0050] S103. Based on the parameter variable constraints of the constructed first bitrate path flow and second bitrate path flow models and the model based on the target bitrate, optimize the model to obtain the final first bitrate path flow and second bitrate path flow.

[0051] Specifically, in S101 of this embodiment, the pseudo-dual-stream bitrate allocation module constructs a bitrate allocation algorithm based on a rate-quantization (RQ) model. The RQ model is dynamically constructed and updated by a general encoder based on the real-time bitrate, quantization step size, and video content. This invention uses the general encoder x264 as an example, but it can also be applied to other types of encoders. The RQ model of the x264 encoder is shown below:

[0052]

[0053] Where q represents the quantization step size of a frame, c represents the complexity of the frame, which is characterized by the sum of absolute transformed differences (SATD), R represents the amount of data required to encode the frame when the quantization step size is q, and a represents the fitting parameters of the RQ model. During video encoding, the model continuously updates the parameter a based on the selected parameters and the actual amount of encoded data in the latest encoded frame, enabling the model to adapt to the latest dynamically changing video content and serving as the basis for future dual-stream configuration decisions based on time consistency.

[0054] This embodiment allocates target bitrates for dual streams at the stream level, avoiding bitrate allocation at the frame or macroblock level and thus preventing significant additional computational overhead. To this end, this embodiment further converts the aforementioned frame-level RQ model to the stream level. In S101 of this embodiment, the frame-level bitrate-quantization model is converted to a stream-level bitrate-quantization model. Specifically, this conversion includes: using the video frame corresponding to the keyframe as the first frame, and modeling the dual streams from start to end as the basic unit, with a duration of T; pre-modeling the original single stream within this T-time period based on the frame-level bitrate-quantization model, as shown below:

[0055]

[0056] Where b represents the original video stream bitrate, and f represents the original video stream frame rate. For the original video stream resolution, This represents the complexity of the original video. In this basic unit, the first frame is the encoded keyframe with a size of R1, which is a known quantity. All subsequent frames are uncoded non-keyframes with a quantization step size of q. i And complexity c i All are unknown, i∈{2,3,…,Tf}, where Tf represents the number of frames in the unit.

[0057] In all video stream-related configurations in the modeling, b is the known target bitrate determined by the bitrate adaptive algorithm, which is further assigned in subsequent dual-stream modeling. f, due to its slow change, is considered to remain consistent with the previous bitrate. With f unchanged, the complexity c of predicting subsequent frames using a linear model based on previous frames can be achieved. i Therefore, given the target bitrate b, the average resolution achievable by a single stream can be estimated as follows:

[0058]

[0059] Next, we continue modeling the dual streams to further determine the bitrate allocation of the two streams while maximizing video playback clarity. In the dual-stream stage, the two encoders each generate two independent and different bitrate streams, where the first bitrate stream reduces the frame rate, and the second bitrate stream adjusts the encoded frame type.

[0060] In S102 of this embodiment, the first bitrate stream is modeled according to the stream-level bitrate-quantization model. The specific method includes: the bitrate of the first bitrate stream is as follows:

[0061]

[0062] Where b′ represents the bit rate of the first bit rate stream, f′ represents the frame rate of the first bit rate stream, and f represents the frame rate of the original video stream. For the first bitrate stream clarity, This represents the complexity of the first bitrate stream, which is directly converted from the original single stream, retaining the keyframe R1. However, the difference from the original single stream lies in the significant reduction of the first bitrate stream's frame rate f′ and the expected sharpness of non-keyframes. and complexity The changes.

[0063] Specifically, reducing the frame rate f′, i.e., skipping the encoding of a large number of subsequent non-critical frames, can significantly reduce the bit rate b′, providing feasibility for dual-stream bit rate allocation that guarantees clarity. Furthermore, since subsequent non-critical frames have not yet been captured and encoded during modeling, their complexity is unknown, and as the inter-frame encoding time interval 1 / f′ increases, the inter-frame differences increase significantly, leading to a decrease in the average complexity of encoding non-critical frames. As a result, the complexity of previously encoded frames cannot be used as a reference value for prediction.

[0064] To estimate the complexity of subsequent non-keyframes under the changing f′ in the above formula. Taking the SATD metric used in x264 as an example, the correlation between SATD and other readily available metrics is modeled. These readily available metrics include the inter-frame time interval 1 / f′ and the sum of absolute differences (SAD) between pixels in each frame, denoted as s′(f′). s′(f′) can be easily calculated from two frames with a time interval of 1 / f′ without requiring actual inter-frame coding as in calculating SATD. Even within the same video, SATD fluctuates significantly within the same inter-frame time interval; conversely, the SATD-SAD scatter plot shows good aggregation. Therefore, the module further utilizes the coupling relationship between SAD and SATD to predict SATD. Specifically, the module uses a cubic function to fit the SATD-SAD scatter plot, and different cubic functions are used for different video content to further improve fitting accuracy. The fitted cubic function is shown below:

[0065] c′(f′)=m1*s′(f′) 3 +m2*s′(f′) 2 +m3*s'(f′)+m4

[0066] Where [m1,m2,m3,m4] are the coefficients of the fitting function, which are updated in real time during video encoding to fit new SATD-SAD scatter points and adapt to dynamic content changes. Based on this, the module predicts the SATD of future frames as a complexity estimate of future frames based on the fitting function. Specifically, in this embodiment, the most recent historical SAD, i.e., s′(f′), is first calculated and used as the future estimate based on short-time consistency. Then, the future SATD is calculated using the future SAD according to the above formula.

[0067] Compared to the original single stream, the second rate stream in the dual-stream stage replaces the encoding of key frames with non-key frames, and is entirely composed of non-key frames in subsequent encoding. The second rate stream minimizes the key frame rate without changing the FPS; in S103 of this embodiment, the second rate stream is modeled according to the stream-level rate-quantization model, and the bitrate of the second rate stream is as follows:

[0068]

[0069] Where "b" represents the bit rate of the second bit rate stream, and "f" represents the frame rate of the first bit rate stream. For the first bitrate stream clarity, Denote the complexity of the first bitrate stream. The first non-key frame to be replaced needs to maintain the same clarity q1 as the key frame in the first stream. Since the frame rate f remains unchanged, its complexity c”1 can also be predicted based on past frames using a linear model. In addition, the subsequent frames from 2 to Tf need to maintain consistent clarity and average clarity.

[0070] After modeling the dual stream and the original single stream, this embodiment depicts the bitrate allocation problem as:

[0071]

[0072] Further transform this problem into finding suitable f′, T for the dual stream and to minimize the video coding clarity loss Δq while ensuring that the sum of bitrates does not exceed the original single stream bitrate b. Where is the optional quantization step size, and T is an integer multiple of, that is η is the constraint factor for the upper limit of the dual stream duration T to prevent the dual stream duration from being too long. The remaining configurations and parameters are known, and they are either the same as the original single stream or directly extracted or estimated from the already encoded frames.

[0073] To solve this optimization problem, considering that the variable range is limited, this embodiment uses an exhaustive search algorithm to find the optimal solution and also uses some pruning operations to reduce the computational complexity. Finally, the algorithm complexity can be estimated as O(f 2 ). After solving the optimal f′, T and , the bitrate allocations b′ and b” assigned by the dual stream encoder can be further calculated. To improve the robustness and adaptability of the algorithm, after allocating the encoder target bitrate, this embodiment does not change the built-in bitrate control mechanism of the encoder. The bitrate control mechanism of stream 2 can autonomously adjust the video configuration based on the allocated bitrate to approach the best selection instead of forcing a specification; for stream 1, the frame rate f′ is also not forced to be specified, but the next frame is encoded after the previous frame is encoded and transmitted, as described in (ii). In addition, when b′ + b” < b after solving, this embodiment will increase their values proportionally to make full use of the overall target bitrate. Finally, the above bitrate allocation strategy changes continuously with the change of the overall target bitrate, and the untransmitted real-time video bits in the Pacer queue are used as the new R1 for subsequent updates to update the optimal bitrate allocation strategy.

[0074] S200. Encode the first bitrate stream and the second bitrate stream according to preset rules to generate two data packets; wherein, in this embodiment, the pseudo dual-stream encoding process can be divided into two stages: a single-stream stage and a dual-stream stage. The single-stream stage is the same as the normal real-time video stream, only encoding, transmitting, decoding and playing the original video stream, without modifying the default algorithms such as bitrate allocation, keyframe decision, and configuration adjustment.

[0075] Dual-stream encoding is only activated when the original video stream encodes the latest frame as a keyframe according to its default keyframe decision. In S200 of this embodiment, the first bitrate stream and the second bitrate stream are encoded according to preset rules to generate two data packets. The specific method includes: dual-stream encoding is only activated when the original video stream encodes the latest frame as a keyframe according to its default keyframe decision; when generating a keyframe, the video sender immediately creates a secondary encoder to encode the parallel stream. The encoder is essentially a clone of the original encoder, which uses the original stream as a basis to continue the encoding process. Correspondingly, the receiving end also activates a secondary decoder and implements a first-decode-first-play strategy on the same frame.

[0076] The first bitrate stream retains the original keyframes and is used to transmit burst traffic generated by intra-frame encoded keyframes. The transmission of these keyframes ensures that decoding or transmission errors do not accumulate rapidly during inter-frame propagation. During the transmission of keyframes in the first bitrate stream, newly captured video frames are not encoded but are largely skipped until a keyframe transmission is confirmed via RTCP feedback. After keyframe transmission, the latest frame is encoded as a P-frame to continue providing reference functionality. If the P-frame's size is outside a preset range due to its distance from the reference frame, the above process is repeated to acquire the latest P-frame until its size and target bitrate are within the preset range. Only then is the dual-stream phase terminated and the second bitrate stream closed. The second bitrate stream is used for real-time playback and does not consider providing optimal encoding / decoding references for subsequent frames or limiting error propagation. The second bitrate stream is activated when a keyframe appears in the original stream, synchronously encoding the keyframe into a smaller non-keyframe. Once activated, the second bitrate stream encodes subsequently captured video frames, which are entirely composed of non-keyframes to ensure playback continuity. As mentioned above, the closure of the second rate stream depends on the transmission status of the first rate stream. Furthermore, although the second rate stream does not contain keyframes and may cause a small amount of error propagation, this effect is negligible because the dual-stream phase only lasts for a very short time.

[0077] S300. The two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to the preset priority.

[0078] In S300 of this embodiment, the two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to a preset priority. The specific method includes: sorting the packets by priority according to their type, with the priorities from high to low being audio packets, retransmission packets, video packets, FEC redundancy packets, and padding packets; and setting the data packets generated by the second bitrate stream encoding to have a higher priority than the data packets generated by the first bitrate stream encoding, preferably transmitting the second bitrate stream. The first bitrate stream is sent when both of the following conditions are met: there are no second bitrate stream data packets to be sent in the Pacer queue and there is still available bitrate budget.

[0079] This priority-controlled transmission process achieves both low-latency real-time video transmission and playback, while ensuring the proper decoding of keyframes to guarantee real-time video quality. For example... Figure 2 As shown, in this scheme, the completion time of sending video frames i to i+4 is earlier than in the original scheme, which greatly reduces the Pacer queuing delay at the video frame sending end, allowing the receiving end to receive and play the video earlier. At the same time, the non-key frame version of the i-th frame is only sent when there are two conditions in the parallel stream of inter-frame coding, and stream 1 serves as the decoding reference frame for subsequent frames, ensuring video clarity.

[0080] This embodiment discloses a pseudo-dual-stream video encoding and transmission method for real-time video, which has extremely high compatibility and wide adaptability. It can be directly deployed on general encoders such as H.26x and VPx without modifying the encoder itself, possessing the high speed and high reliability advantages inherent in general encoders. Simultaneously, this invention is perfectly compatible with current mainstream WebRTC-based real-time communication architectures. This invention minimizes long-tail latency and average latency, meeting the low latency requirements of various real-time video communication applications (such as video calls, video conferencing, cloud gaming, live video streaming, and vehicle networking). At the same time, it ensures other QoE performance indicators, such as video clarity, smoothness, stuttering rate, and frame drop rate, through more reliable, low-latency, and low-packet-loss transmission. This invention can be used in conjunction with a bitrate adaptive algorithm to further enhance video bitrate and clarity. Because this invention can provide a stable video bitstream, avoiding increased latency and congestion caused by traffic bursts from keyframes, it can be combined with a bitrate adaptive algorithm to further enhance adaptability and tolerance to wireless communication environments, increasing the average target bitstream of the video.

[0081] Example 2

[0082] Based on the same inventive concept, this disclosure also provides a pseudo-dual-stream video encoding and transmission system for real-time video, such as... Figure 3It includes: a pseudo-dual-stream coding rate allocation module, a pseudo-dual-stream coding module, and a transmission priority control module; wherein:

[0083] The pseudo-dual-stream coding bitrate allocation module is used to analyze real-time video. When a keyframe is generated in the real-time video, the target bitrate of the real-time video is obtained, and the target bitrate of the real-time video is allocated based on the bitrate-quantization model to obtain the first bitrate path stream and the second bitrate path stream.

[0084] The pseudo-dual-stream encoding module is used to encode the first bit rate stream and the second bit rate stream according to preset rules to generate two data packets;

[0085] The transmission priority control module is used to temporarily store the two data packets in the Pacer sending queue and send the data packets to the network according to a preset priority.

[0086] The specific working methods of the pseudo-dual-stream coding rate allocation module, the pseudo-dual-stream coding module, and the transmission priority control module have been described in detail in Embodiment 1, and will not be repeated here in this embodiment.

[0087] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0088] In the detailed description above, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features in a single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, with each claim representing a separate preferred embodiment of the invention.

[0089] Those skilled in the art will also understand that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments herein can be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in alternative ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0090] The steps of the methods or algorithms described in conjunction with the embodiments herein can be directly embodied in hardware, software modules executed by a processor, or a combination thereof. The software modules can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor and storage medium can exist as discrete components in the user terminal.

[0091] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. This software code can be stored in memory units and executed by a processor. The memory units can be implemented within the processor or outside the processor; in the latter case, they are communicatively coupled to the processor via various means, as is well known in the art.

[0092] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

Claims

1. A pseudo-dual-stream video encoding and transmission method for real-time video, characterized in that, include: S100. Analyze the real-time video. When a keyframe is generated in the real-time video, obtain the target bitrate of the real-time video. Allocate the target bitrate of the real-time video based on the bitrate-quantization model to obtain the first bitrate path stream and the second bitrate path stream. In S100, the method for constructing the bitrate-quantization model includes: S101. Obtain the frame-level bitrate-quantization model and convert it into a stream-level bitrate-quantization model; in S101, the frame-level bitrate-quantization model is as follows: ； in, This represents the quantization step size of an image frame. The complexity of this frame is represented by the sum of the absolute values ​​of the residual transform coefficients. This indicates that when the quantization step size is The amount of data required to encode this frame's data volume. This represents the fitting parameters of the model; during video encoding, the model continuously updates the parameters based on the selected parameters and the actual amount of encoded data in the latest encoded frame. This enables the model to adapt to the latest dynamic changes in video content and serves as the basis for future dual-stream configuration decisions based on time consistency. In S101, the frame-level bitrate-quantization model is converted into a stream-level bitrate-quantization model. Specifically, this involves: using the video frame corresponding to the keyframe as the first frame, modeling the process from the start to the end of the dual streams as the basic unit, with a duration of T; and pre-modeling the process according to the frame-level bitrate-quantization model. The original single flow is modeled as follows: ； in, Indicates the original video stream bitrate. Indicates the frame rate of the original video stream. For the original video stream resolution, This represents the complexity of the original video. In this basic unit, the first frame is the encoded keyframe, and its size is... , is a known quantity, and all subsequent frames are uncoded non-key frames, whose quantization step size is... and complexity Everything is unknown. , Indicates the number of frames in a unit; S102. Model the first bitrate stream and the second bitrate stream according to the stream-level bitrate-quantization model; in S102, the first bitrate stream is modeled according to the stream-level bitrate-quantization model, and the specific method includes: the bitrate of the first bitrate stream is as follows: ； ； in, Indicates the bit rate of the first bit rate stream. Indicates the first bitrate stream frame rate. Indicates the frame rate of the original video stream. For the first bitrate stream clarity, This represents the complexity of the first bitrate stream, which is directly converted from the original single stream, retaining keyframes. However, the difference from the original single stream lies in the first bitrate path stream frame rate. Significant reduction in expected sharpness of non-key frames and complexity Changes; In S102, the second bitrate stream is modeled according to the stream-level bitrate-quantization model, and the bitrate of the second bitrate stream is as follows: ； ； in, Indicates the second bitrate stream bitrate. Indicates the frame rate of the original video stream. For the first bitrate stream clarity, This indicates the complexity of the original video. The first non-keyframe to be replaced needs to maintain the same clarity as the keyframes in the first stream. Due to frame rate Unchanged, its complexity Also based on past frames, predictions are made using a linear model; furthermore, from 2 to... Subsequent frames need to maintain the same resolution and average resolution as the first bitrate stream. ; S103. Based on the parameter variable constraints of the constructed first bitrate path flow and second bitrate path flow models and the model based on the target bitrate, optimize the model to obtain the final first bitrate path flow and second bitrate path flow; S200. Encode the first bitrate stream and the second bitrate stream according to preset rules to generate two data packets; In S200, the first bitrate stream and the second bitrate stream are encoded according to preset rules to generate two data packets; The specific method includes: Dual-stream encoding is only activated when the original video stream encodes the latest frame as a keyframe according to its default keyframe decision; When generating a keyframe, the video sender immediately creates a secondary encoder to encode the parallel stream. The encoder is essentially a clone of the original encoder. It uses the original stream as the basis to continue the encoding process. Correspondingly, the receiving end also activates a secondary decoder and implements a first-decode-first-play strategy on the same frame; The first bitrate stream retains the original keyframes and is used to transmit burst traffic generated by intra-frame encoded keyframes. The transmission of these keyframes ensures that decoding or transmission errors do not accumulate rapidly during inter-frame propagation. During the transmission of keyframes in the first bitrate stream, newly captured video frames are not encoded but are largely skipped until a keyframe transmission is confirmed via an RTCP feedback message. After keyframe transmission is complete, the latest frame is encoded as a P-frame to continue providing reference functionality. If the P-frame's size is outside a preset range due to its distance from the reference frame, the above process is repeated to obtain the latest P-frame until its size and target bitrate are within the preset range. The second bitrate stream is used for real-time playback and does not consider providing optimal encoding / decoding references for subsequent frames or limiting error propagation. The second bitrate stream is activated when a keyframe appears in the original stream, synchronously encoding the keyframe into a smaller non-keyframe. Once activated, the second bitrate stream encodes subsequently captured video frames, and these frames consist entirely of non-keyframes to ensure playback continuity. S300. The two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to the preset priority.

2. The pseudo-dual-stream video encoding and transmission method for real-time video as described in claim 1, characterized in that, In S300, the two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to a preset priority. The specific method includes: sorting the packets by priority according to their type, with the priorities from high to low being audio packets, retransmission packets, video packets, FEC redundancy packets, and padding packets; and setting the data packets generated by the second bitrate stream encoding to have a higher priority than the data packets generated by the first bitrate stream encoding, and prioritizing the transmission of the second bitrate stream. The first bitrate stream is sent when both of the following conditions are met: there are no second bitrate stream data packets to be sent in the Pacer queue and there is still available bitrate budget.

3. A pseudo-dual-stream video encoding and transmission system for real-time video, employing the transmission method described in any one of claims 1-2, characterized in that, include: The module includes a pseudo-dual-stream coding rate allocation module, a pseudo-dual-stream coding module, and a transmission priority control module; among which: The pseudo-dual-stream coding bitrate allocation module is used to analyze real-time video. When a keyframe is generated in the real-time video, the target bitrate of the real-time video is obtained, and the target bitrate of the real-time video is allocated based on the bitrate-quantization model to obtain the first bitrate path stream and the second bitrate path stream. The pseudo-dual-stream encoding module is used to encode the first bit rate stream and the second bit rate stream according to preset rules to generate two data packets; The transmission priority control module is used to temporarily store the two data packets in the Pacer sending queue and send the data packets to the network according to a preset priority.

Images