Real-time video-oriented pseudo double-stream video coding and transmission method and system

Abstract

A real-time video-oriented pseudo double-stream video coding and transmission method comprises the following steps: analyzing a real-time video, acquiring a real-time video target code rate when a key frame is generated in the real-time video, and allocating the real-time video target code rate based on a code rate-quantization model to obtain a first code rate path stream and a second code rate path stream; encoding the first code rate path flow and the second code rate path flow according to a preset rule to generate two paths of data messages; and temporarily storing the two paths of data messages in a Pacer sending queue, and sending the data messages to a network according to a preset priority. The method has extremely high compatibility and wide adaptive capacity, does not need to modify the encoder, and has the advantages of high speed and high reliability of a general encoder. According to the method, the long-tail delay and the average delay can be minimized, the low-delay requirements of various real-time and real-time video communication applications are met, and meanwhile, other index performances are guaranteed through more reliable, low-delay and low-packet-loss transmission.

Description

Technical Field

[0001] The present invention relates to the field of video coding and communication technology, and in particular to a pseudo dual-stream video coding and transmission method and system for real-time video. Background Art

[0002] In recent years, real-time video has rapidly become a popular and promising media form, inspiring unprecedented expectations for new viewing experiences. From calls, conferences, live broadcasts to cloud games, the global real-time video market reached a value of US$4.91 billion in 2020 and is expected to continue to grow at an average annual rate of 10.0% during 2022-2028. In order to provide a unique interactive experience, optimizing the end-to-end (E2E) latency from video capture to playback is crucial because it directly affects user engagement and immersion.

[0003] Despite the promising market, the end-to-end latency of real-time video has not yet achieved satisfactory results and shows a significant long-tail distribution in high-latency areas. This is mainly attributed to the huge differences in video frame sizes. Experimental data shows that in a general real-time video transmission system based on WebRTC, the size of video key frames is significantly larger than non-key frames, resulting in significantly higher transmission delays for key frames than non-key frames, which may eventually far exceed the latency requirements of real-time video applications (e.g., the latency requirement for video calls is 200 milliseconds). Unlike video on demand (VoD), which is less affected, this problem poses a serious challenge to real-time video streams with stringent latency requirements.

[0004] To optimize latency and mitigate the long-tail effect, existing algorithms cover a variety of methods from jitter buffering, bitrate adaptation to coding optimization, but each has inherent limitations: (1) Jitter buffering algorithms smooth the transmission delay jitter of video frames by introducing waiting time at the receiving end, making video playback smoother and mitigating the long-tail effect, but this will increase the overall end-to-end latency of video frames; (2) Video bitrate adaptation algorithms either use artificial predefined rules that are more sensitive to delay and packet loss, or use reinforcement learning (RL) to automatically generate strategies to adapt to key frame burst traffic. However, these algorithms ultimately sacrifice video bitrate and clarity in exchange for improved end-to-end latency; (3) Coding optimization algorithms either force the key frame size to be adjusted to the level of non-key frames, such as a strict constant bitrate (CBR) mode, or minimize the key frame frequency, or customize the encoder. The first two methods cause the video encoding configuration to deviate from the optimal choice, or affect the clarity of key frames and subsequent non-key frames that use them as references, or accelerate the accumulation of inter-frame errors, thereby affecting video clarity. The last method lacks universality due to compatibility issues with common codecs. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a pseudo dual-stream video encoding and transmission method and system for real-time video that overcomes the above problems or at least partially solves the above problems.

[0006] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention discloses a pseudo dual-stream video encoding and transmission method for real-time video, comprising:

[0008] S100. Analyze the real-time video, when a key frame is generated in the real-time video, obtain the real-time video target bit rate, allocate the real-time video target bit rate based on the bit rate-quantization model, and obtain the first bit rate path flow and the second bit rate path flow;

[0009] S200. Encode the first code rate path flow and the second code rate path flow according to preset rules to generate two data packets;

[0010] S300. Temporarily store the two data packets in a Pacer sending queue, and send the data packets to the network according to a preset priority.

[0011] Further, in S100, the method for constructing the rate-quantization model includes:

[0012] S101. Obtain a frame-level rate-quantization model, and convert the frame-level rate-quantization model into a stream-level rate-quantization model;

[0013] S102. Modeling the first bit rate path flow and the second bit rate path flow respectively according to the stream level bit rate-quantization model;

[0014] S103. According to the constructed first bit rate flow path and second bit rate flow path models and the parameter variable constraint relationship of the model based on the target bit rate, the model is optimized to obtain the final first bit rate flow path and second bit rate flow path.

[0015] Further, in S101, the frame level bit rate-quantization model is as follows:

[0016]

[0017] Among them, q represents the quantization step size of a frame of image, c represents the complexity of the frame, which is characterized by the sum of the absolute values ​​of the residual transform coefficients, R represents the amount of data required to encode the frame data when the quantization step size is q, and a represents the fitting parameter of the RQ model; during the video encoding process, the model continuously updates the parameter a according to the selected parameters and the actual amount of encoded data of the latest encoded frame, so that the model can adapt to the latest dynamically changing video content and serve as the basis for future dual-stream configuration decisions based on time consistency.

[0018] Further, in S101, the frame-level bit rate-quantization model is converted into a stream-level bit rate-quantization model, and the specific method includes: taking the video frame corresponding to the key frame as the first frame, and taking the start to the end of the dual stream as the basic unit for modeling, and the duration is T; pre-modeling the original single stream in the T period according to the frame-level bit rate-quantization model, and the model is as follows:

[0019]

[0020] Where b represents the bit rate of the original video stream, f represents the frame rate of the original video stream, is the original video stream definition, Represents the complexity of the original video. In this basic unit, the first frame is the encoded key frame, whose size is R1, which is a known quantity, and all subsequent frames are unencoded non-key frames, whose quantization step size is q i And the complexity c i are all unknown, i∈{2,3,…,Tf}, Tf represents the number of frames in the unit.

[0021] Further, in S102, a first bitrate path flow is modeled according to a stream level bitrate-quantization model, and the specific method includes: the bitrate of the first bitrate path flow is as follows:

[0022]

[0023] Wherein, b′ represents the bit rate of the first bit rate stream, f′ represents the frame rate of the first bit rate stream, is the first bitrate stream clarity, Indicates the complexity of the first bitrate stream. The first bitrate stream is directly converted from the original single stream, retaining the key frame R1. However, the difference from the original single stream is that the frame rate f′ of the first bitrate stream is significantly reduced and the expected clarity of non-key frames is and complexity changes.

[0024] Further, in S103, the second bitrate path flow is modeled according to the stream level bitrate-quantization model, and the bitrate of the second bitrate path flow is as follows:

[0025]

[0026] Wherein, b" represents the bit rate of the second bit rate stream, f represents the frame rate of the first bit rate stream, is the first bitrate stream clarity, represents the complexity of the first bitrate stream. The first non-key frame to be replaced needs to maintain the same clarity q1 as the key frame in the first stream. Since the frame rate f does not change, its complexity c"1 can also be predicted based on the past frames using a linear model. In addition, the subsequent frames from 2 to Tf need to maintain the same clarity and average clarity as

[0027] Furthermore, in S200, the first bit rate path stream and the second bit rate path stream are encoded according to preset rules to generate two data messages; the specific method includes: dual-stream encoding needs to be activated only when the original video stream encodes the latest frame as a key frame according to its default key frame decision; when generating a key frame, the video sender immediately creates a secondary encoder to encode a parallel stream, and the encoder is essentially a clone of the original encoder, which uses the original stream as the basis to continue the encoding process. Correspondingly, the receiving end also activates a secondary decoder and implements a decode-first-playback strategy on the same frame.

[0028] Furthermore, the first bitrate stream retains the original key frames and is used to transmit burst traffic generated by the key frames encoded within the frames. The transmission of the key frames ensures that decoding or transmission errors will not be accelerated and accumulated during the inter-frame propagation process. During the transmission of the key frames in the first bitrate stream, the newly captured video frames are no longer encoded, but are skipped in large quantities until the key frame transmission is confirmed to be completed through the RTCP feedback message. After the key frame transmission is completed, the latest frame is encoded as a P frame to continue to provide a reference function. If the volume of the P frame is outside the preset range due to being far away from the reference frame, the above process is repeated to obtain the latest P frame until the volume of the latest P frame is within the preset range with the target bit rate. The second bitrate stream is used for real-time playback without considering functions such as providing the best encoding and decoding reference for subsequent frames and limiting error propagation. The second bitrate stream is started when a key frame appears in the original stream, and the key frame is synchronously encoded as a non-key frame with a smaller data volume. Once the second bitrate stream is started, the subsequent continuously captured video frames will be encoded, and the video frames are completely composed of non-key frames to ensure the continuity of playback.

[0029] Further, in S300, the two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to a preset priority. The specific method includes: sorting the priorities according to the packet types, and the priorities from high to low are audio packets, retransmission packets, video packets, FEC redundant packets and Padding packets; and the data packets generated by the second code rate stream encoding are set to a higher priority than the data packets generated by the first code rate stream encoding, and the second code rate stream is preferably transmitted. When the two conditions that there are no second code rate stream data packets to be sent in the Pacer queue and there is still available code rate budget are met at the same time, the first code rate stream is sent.

[0030] In a second aspect, an embodiment of the present invention discloses a pseudo dual-stream video encoding and transmission system for real-time video, including: a pseudo dual-stream encoding bit rate allocation module, a pseudo dual-stream encoding module and a transmission priority control module; wherein:

[0031] The pseudo dual-stream encoding bit rate allocation module is used to analyze the real-time video, obtain the real-time video target bit rate when a key frame is generated in the real-time video, and allocate the real-time video target bit rate based on the bit rate-quantization model to obtain a first bit rate path flow and a second bit rate path flow;

[0032] A pseudo dual-stream encoding module, used for encoding the first code rate stream and the second code rate stream according to a preset rule to generate two data messages;

[0033] The transmission priority control module is used to temporarily store the two data messages in the Pacer sending queue and send the data messages to the network according to the preset priority.

[0034] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0035] The present invention discloses a pseudo dual-stream video encoding and transmission method for real-time video, which has extremely high compatibility and wide adaptability, can be directly deployed on general encoders such as H.26x and VPx, does not need to modify the encoder itself, and has the advantages of high speed and high reliability of the general encoder itself. At the same time, the present invention can also be perfectly compatible with the current mainstream WebRTC-based real-time communication architecture; the present invention can minimize long-tail delay and average delay, meet the low-delay requirements of various real-time video communication applications (such as video calls, video conferences, cloud games, video live broadcasts, car networking, etc.), and at the same time, through more reliable, low-delay, low-packet loss transmission to ensure other QoE indicators, such as video clarity, smoothness, stuttering rate, frame loss rate, etc. The present invention can be used in conjunction with a bit rate adaptive algorithm to further enhance the bit rate and clarity of the video. Since the present invention can provide a stable video bit stream, avoid the traffic burst caused by key frames and cause delay increase, congestion and other consequences, it can be combined with the bit stream adaptive algorithm to further enhance the adaptability and tolerance to the wireless communication environment and improve the average target bit stream of the video.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 This is a flowchart of a pseudo dual-stream video encoding and transmission method for real-time video in embodiment 1 of the present invention;

[0039] Figure 2 This is a schematic diagram of sending two data packets to a network in Embodiment 1 of the present invention;

[0040] Figure 3 This is a structural diagram of a pseudo dual-stream video encoding and transmission system for real-time video in Example 2 of the present invention. DETAILED DESCRIPTION

[0041] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0042] In order to solve the problems existing in the prior art, the embodiments of the present invention provide a pseudo dual-stream video encoding and transmission method and system for real-time video.

[0043] Example 1

[0044] The present invention discloses a pseudo dual-stream video encoding and transmission method for real-time video. Figure 1 ,include:

[0045] S100. Analyze the real-time video, when a key frame is generated in the real-time video, obtain the real-time video target bit rate, allocate the real-time video target bit rate based on the bit rate-quantization model, and obtain the first bit rate path flow and the second bit rate path flow;

[0046] Specifically, when the dual-stream stage is turned on, the pseudo-dual-stream encoding module quantitatively allocates the overall target bit rate (determined by other bit rate adaptation algorithms) to the dual streams. The goal of the allocation is to minimize the clarity loss (and maximize the clarity) while ensuring that the sum of the bit rates of the dual streams does not exceed the overall target bit rate.

[0047] In S100 of this embodiment, the method for constructing a rate-quantization model includes:

[0048] S101. Obtain a frame-level rate-quantization model, and convert the frame-level rate-quantization model into a stream-level rate-quantization model;

[0049] S102. Modeling the first bit rate path flow and the second bit rate path flow respectively according to the stream level bit rate-quantization model;

[0050] S103. According to the constructed first bit rate flow path and second bit rate flow path models and the parameter variable constraint relationship of the model based on the target bit rate, the model is optimized to obtain the final first bit rate flow path and second bit rate flow path.

[0051] Specifically, in S101 of this embodiment, the pseudo dual-stream rate allocation module constructs a rate allocation algorithm based on a rate-quantization (RQ) model. The RQ model is dynamically constructed and updated by a general encoder according to the real-time bit rate, quantization step size and video content. The present invention is introduced by taking the general encoder x264 as an example, but it can also be applied to other types of encoders. The RQ model of the x264 encoder is as follows:

[0052]

[0053] Among them, q represents the quantization step size of a frame of image, c represents the complexity of the frame, which is characterized by the sum of the absolute values ​​of the residual transform coefficients (Sum of Absolute Transformed Difference, SATD), R represents the amount of data required to encode the frame data when the quantization step size is q, and a represents the fitting parameter of the RQ model; during the video encoding process, the model continuously updates the parameter a according to the selected parameters and the actual amount of encoded data of the latest encoded frame, so that the model can adapt to the latest dynamically changing video content and serve as the basis for future dual-stream configuration decisions based on time consistency.

[0054] This embodiment allocates a target bit rate for the dual streams at the stream level, avoids bit rate allocation at the frame level or macroblock level, and avoids generating a large amount of additional computational overhead. To this end, this embodiment further converts the above-mentioned frame-level RQ model to the stream level. In S101 of this embodiment, the frame-level bit rate-quantization model is converted to a stream-level bit rate-quantization model. The specific method includes: taking the video frame corresponding to the key frame as the first frame, and taking the start to the end of the dual stream as the basic unit for modeling, with a duration of T; pre-modeling the original single stream in the T period according to the frame-level bit rate-quantization model, and the model is as follows:

[0055]

[0056] Where b represents the bit rate of the original video stream, f represents the frame rate of the original video stream, is the original video stream definition, Represents the complexity of the original video. In this basic unit, the first frame is the encoded key frame, whose size is R1, which is a known quantity, and all subsequent frames are unencoded non-key frames, whose quantization step size is q i And the complexity c i are all unknown, i∈{2,3,…,Tf}, Tf represents the number of frames in the unit.

[0057] In all the video stream related configurations modeled, b is the known target bit rate determined by the bit rate adaptation algorithm, which is further assigned in the subsequent dual-stream modeling. Since f changes slowly, it is considered to be consistent with the previous one. When f remains unchanged, the complexity c of the subsequent frame can be predicted based on the previous frame using a linear model. i Therefore, given the target bit rate b, the average clarity that can be achieved by the original single stream can be estimated as:

[0058]

[0059] Next, we continue to model the dual streams to further determine the bitrate allocation of the two streams while maximizing the video playback clarity. In the dual stream stage, the two encoders generate two independent and different streams, where the first bitrate stream reduces the frame rate and the second bitrate stream adjusts the encoding frame type.

[0060] In S102 of this embodiment, a first bitrate path flow is modeled according to a stream level bitrate-quantization model. The specific method includes: the bitrate of the first bitrate path flow is as follows:

[0061]

[0062] Wherein, b′ represents the bit rate of the first bit rate stream, f′ represents the frame rate of the first bit rate stream, and f represents the frame rate of the original video stream. is the first bitrate stream clarity, Indicates the complexity of the first bitrate stream. The first bitrate stream is directly converted from the original single stream, retaining the key frame R1. However, the difference from the original single stream is that the frame rate f′ of the first bitrate stream is significantly reduced and the expected clarity of non-key frames is and complexity changes.

[0063] Specifically, reducing the frame rate f′, that is, skipping the encoding of a large number of subsequent non-key frames, can significantly reduce the bit rate b′, providing feasibility for achieving dual-stream bit rate allocation with guaranteed clarity. In addition, since the subsequent non-key frames have not been captured and encoded during modeling, their complexity is unknown, and as the inter-frame encoding time interval 1 / f′ increases, the inter-frame difference increases significantly, resulting in an average complexity of non-key frame encoding. Increased, the complexity of the previously coded frame cannot be used as a reference value for prediction.

[0064] To estimate the complexity of subsequent non-key frames when f′ changes in the above formula Taking the SATD indicator used in x264 as an example, the correlation between SATD and other easily accessible indicators is modeled. Other easily accessible indicators include the inter-frame time interval 1 / f′ and the sum of the absolute difference of the pixels between frames (SAD), denoted as s′(f′). s′(f′) can be easily calculated based on two frames with a time interval of 1 / f′, without the need for actual inter-frame encoding as in the calculation of SATD. Even in the same video, SATD will fluctuate significantly within the same inter-frame time interval; in contrast, the SATD-SAD scatter points show good aggregation. Therefore, the module further uses the coupling correlation between SAD and SATD to predict SATD. Specifically, the module uses a cubic function to fit the SATD-SAD scatter points. In order to further improve the fitting accuracy, different video content will use different cubic functions. The fitted cubic function is as follows:

[0065] c′(f′)=m1*s′(f′) 3 +m2*s′(f′) 2 +m3*s'(f′)+m4

[0066] Among them, [m1, m2, m3, m4] are the coefficients of the fitting function, and are updated in real time during the video encoding process to fit the new SATD-SAD scatter points to adapt to dynamic content changes. On this basis, the module predicts the SATD of future frames based on the fitting function as the complexity estimate of future frames. Specifically, this embodiment first calculates the most recent historical SAD, i.e., s′(f′), and uses it as the future estimated value based on short-term temporal consistency, and then uses the future SAD to calculate the future SATD according to the above formula.

[0067] Compared with the original single stream, the second bitrate stream in the dual-stream stage replaces the encoding of key frames with non-key frames, and is completely composed of non-key frames in subsequent encoding. The second bitrate stream minimizes the key frame rate without changing the FPS; in S103 of this embodiment, the second bitrate stream is modeled according to the stream-level bitrate-quantization model, and the bitrate of the second bitrate stream is as follows:

[0068]

[0069] Wherein, b" represents the bit rate of the second bit rate stream, f represents the frame rate of the first bit rate stream, is the first bitrate stream clarity, Denote the complexity of the first bitrate stream. The first replaced non-key frame needs to maintain the same clarity q1 as the key frame in the first stream. Since the frame rate f remains unchanged, its complexity c”1 can also be predicted using a linear model based on past frames. In addition, the subsequent frames from 2 to Tf need to maintain consistent clarity and average clarity

[0070] After modeling the dual stream and the original single stream, this embodiment formulates the bitrate allocation problem as:

[0071]

[0072] Further transform this problem into finding suitable f′, T and for the dual stream to minimize the video coding clarity loss Δq while ensuring that the sum of the bitrates does not exceed the original single stream bitrate b. Where is the optional quantization step size, and T is an integer multiple of, that is η is the constraint factor for the upper limit of the dual stream duration T to prevent the dual stream duration from being too long. The remaining configurations and parameters are known, and they are either the same as the original single stream or directly extracted or estimated from the already encoded frames.

[0073] To solve this optimization problem, considering that the variable range is limited, this embodiment uses an exhaustive search algorithm to find the optimal solution, and at the same time uses some pruning operations to reduce the computational complexity. Finally, the algorithm complexity can be estimated as O(f 2 ). After solving the optimal f′, T and , the bitrate allocations b′ and b” allocated by the dual stream encoder can be further calculated. To improve the robustness and adaptability of the algorithm, after allocating the encoder target bitrate, this embodiment does not change the built-in rate control mechanism of the encoder. The rate control mechanism of the stream 2 encoder can autonomously adjust the video configuration based on the allocated bitrate to approach the best selection, rather than forcing a specification; for stream 1, the frame rate f′ is not forced either, but the next frame is encoded after the previous frame is encoded and transmitted, as described in (ii). In addition, when b′ + b” < b after solving, this embodiment will increase their values proportionally to make full use of the overall target bitrate. Finally, the above bitrate allocation strategy is continuously updated with the change of the overall target bitrate. When updating subsequently, the untransmitted real-time video bits in the Pacer queue are used as the new R1 to update the optimal bitrate allocation strategy.

[0074] S200. Encode the first bit rate path stream and the second bit rate path stream according to preset rules to generate two data packets; wherein, in this embodiment, the pseudo dual-stream encoding process can be divided into two stages: a single-stream stage and a dual-stream stage. The single-stream stage is the same as the normal real-time video stream, and only encodes, transmits, decodes and plays the original video stream without modifying the default algorithms such as bit rate allocation, key frame decision, and configuration adjustment.

[0075] The dual-stream encoding needs to be activated only when the original video stream encodes the latest frame as a key frame according to its default key frame decision. In S200 of this embodiment, the first bit rate path stream and the second bit rate path stream are encoded according to a preset rule to generate two data packets; the specific method includes: the dual-stream encoding needs to be activated only when the original video stream encodes the latest frame as a key frame according to its default key frame decision; when generating a key frame, the video sender immediately creates a secondary encoder to encode a parallel stream, and the encoder is essentially a clone of the original encoder, which uses the original stream as a basis to continue the encoding process. Correspondingly, the receiving end also activates a secondary decoder and implements a decode-first-play strategy on the same frame.

[0076] Among them, the first code rate stream retains the original key frame and is used to transmit the burst traffic generated by the key frame encoded in the frame. The transmission of the key frame ensures that the decoding or transmission error will not be accelerated and accumulated during the inter-frame propagation process; during the transmission of the key frame of the first code rate stream, the newly captured video frame is no longer encoded, but skipped in large quantities until the key frame transmission is confirmed to be completed through the RTCP feedback message; when the key frame transmission is completed, the latest frame is encoded as a P frame to continue to provide a reference function. If the volume of the P frame is outside the preset range due to being far away from the reference frame, the above process is repeated to obtain the latest P frame until the volume of the latest P frame and the target bit rate are within the preset range, then the dual stream stage is terminated and the second code rate stream is closed. The second code rate stream is used for real-time playback, without considering the functions of providing the best encoding and decoding reference for subsequent frames and limiting error propagation. The second code rate stream is started when a key frame appears in the original stream, and the key frame is synchronously encoded as a non-key frame with a smaller data volume; once the second code rate stream is started, the subsequent continuously captured video frames will be encoded, and the video frames are completely composed of non-key frames to ensure the continuity of playback. As mentioned above, the closing of the second code rate stream depends on the transmission status of the first code rate stream. In addition, although the second code rate stream does not contain key frames and may cause a small amount of error propagation, this effect is almost negligible because the dual-stream stage only lasts for a short time.

[0077] S300. Temporarily store the two data packets in a Pacer sending queue, and send the data packets to the network according to a preset priority.

[0078] In S300 of the present embodiment, the two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to a preset priority. The specific method includes: sorting the priorities according to the packet types, and the priorities from high to low are audio packets, retransmission packets, video packets, FEC redundant packets and Padding packets; and the data packets generated by the second code rate stream encoding are set to a higher priority than the data packets generated by the first code rate stream encoding, and the second code rate stream is preferably transmitted. When the two conditions that there are no second code rate stream data packets to be sent in the Pacer queue and there is still available code rate budget are met at the same time, the first code rate stream is sent.

[0079] In the transmission process of this priority control, low-latency real-time video transmission and playback are achieved, and the normal decoding of key frames is ensured to ensure the real-time video quality. Figure 2 As shown, in this scheme, the sending completion time of the i-th to i+4-th video frames is earlier than that of the original scheme, which greatly reduces the Pacer queuing delay at the video frame sender, allowing the receiver to receive and play earlier; at the same time, the non-key frame version of the i-th frame is also sent along with the parallel stream of inter-frame coding. Stream 1 is used as the decoding reference frame for subsequent frames to ensure video clarity.

[0080] The pseudo dual-stream video encoding and transmission method for real-time video disclosed in this embodiment has extremely high compatibility and wide adaptability, can be directly deployed on general encoders such as H.26x and VPx, does not need to modify the encoder itself, and has the advantages of high speed and high reliability of the general encoder itself. At the same time, the present invention can also be perfectly compatible with the current mainstream real-time communication architecture based on WebRTC; the present invention can minimize long-tail delay and average delay, meet the low delay requirements of various real-time video communication applications (such as video calls, video conferences, cloud games, video live broadcasts, car networking, etc.), and at the same time, through more reliable, low-delay, low-packet loss transmission to ensure other QoE indicators, such as video clarity, smoothness, stuttering rate, frame loss rate, etc. The present invention can be used in conjunction with the bit rate adaptive algorithm to further enhance the bit rate and clarity of the video. Since the present invention can provide a stable video bit stream, avoid the traffic burst caused by key frames and cause delay increase, congestion and other consequences, it can be combined with the bit stream adaptive algorithm to further enhance the adaptability and tolerance to the wireless communication environment and improve the average target bit stream of the video.

[0081] Example 2

[0082] Based on the same inventive concept, the disclosed embodiment also provides a pseudo dual-stream video encoding and transmission system for real-time video, such as Figure 3, including: a pseudo dual-stream encoding rate allocation module, a pseudo dual-stream encoding module and a transmission priority control module; wherein:

[0083] The pseudo dual-stream encoding bit rate allocation module is used to analyze the real-time video, obtain the real-time video target bit rate when a key frame is generated in the real-time video, and allocate the real-time video target bit rate based on the bit rate-quantization model to obtain a first bit rate path flow and a second bit rate path flow;

[0084] A pseudo dual-stream encoding module, used for encoding the first code rate stream and the second code rate stream according to a preset rule to generate two data messages;

[0085] The transmission priority control module is used to temporarily store the two data messages in the Pacer sending queue and send the data messages to the network according to the preset priority.

[0086] Among them, the specific working methods of the pseudo dual-stream encoding rate allocation module, the pseudo dual-stream encoding module and the transmission priority control module have been described in detail in Example 1, and will not be repeated in this embodiment.

[0087] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0088] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0089] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0090] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0091] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0092] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A pseudo dual-stream video encoding and transmission method for real-time video, characterized in that: include: S100. Analyze the real-time video, when a key frame is generated in the real-time video, obtain the real-time video target bit rate, allocate the real-time video target bit rate based on the bit rate-quantization model, and obtain the first bit rate path flow and the second bit rate path flow; S200. Encode the first code rate path flow and the second code rate path flow according to preset rules to generate two data packets; S300. Temporarily store the two data packets in a Pacer sending queue, and send the data packets to the network according to a preset priority.

2. A pseudo dual-stream video encoding and transmission method for real-time video as claimed in claim 1, characterized in that: In S100, the method for constructing the rate-quantization model includes: S101. Obtain a frame-level rate-quantization model, and convert the frame-level rate-quantization model into a stream-level rate-quantization model; S102. Modeling the first bit rate path flow and the second bit rate path flow respectively according to the stream level bit rate-quantization model; S103. According to the constructed first bit rate flow path and second bit rate flow path models and the parameter variable constraint relationship of the model based on the target bit rate, the model is optimized to obtain the final first bit rate flow path and second bit rate flow path.

3. A pseudo dual-stream video encoding and transmission method for real-time video as claimed in claim 2, characterized in that: In S101, the frame level bit rate-quantization model is as follows: Among them, q represents the quantization step size of a frame of image, c represents the complexity of the frame, which is characterized by the sum of the absolute values ​​of the residual transform coefficients, R represents the amount of data required to encode the frame data when the quantization step size is q, and a represents the fitting parameter of the RQ model; during the video encoding process, the model continuously updates the parameter a according to the selected parameters and the actual amount of encoded data of the latest encoded frame, so that the model can adapt to the latest dynamically changing video content and serve as the basis for future dual-stream configuration decisions based on time consistency.

4. A pseudo dual-stream video encoding and transmission method for real-time video as claimed in claim 2, characterized in that: In S101, the frame-level rate-quantization model is converted into a stream-level rate-quantization model. The specific method includes: taking the video frame corresponding to the key frame as the first frame, taking the start to the end of the dual stream as the basic unit for modeling, and the duration is T; and pre-modeling the original single stream in the T period according to the frame-level rate-quantization model. The model is as follows: Where b represents the bit rate of the original video stream, f represents the frame rate of the original video stream, is the original video stream definition, Represents the complexity of the original video. In this basic unit, the first frame is the encoded key frame, whose size is R1, which is a known quantity, and all subsequent frames are unencoded non-key frames, whose quantization step size is q i And the complexity c i are all unknown, i∈{2,3,…,Tf}, Tf represents the number of frames in the unit.

5. A pseudo dual-stream video encoding and transmission method for real-time video as claimed in claim 2, characterized in that: In S102, a first bitrate path flow is modeled according to a stream level bitrate-quantization model. The specific method includes: the bitrate of the first bitrate path flow is as follows: Wherein, b′ represents the bit rate of the first bit rate stream, f′ represents the frame rate of the first bit rate stream, and f represents the frame rate of the original video stream. is the first bitrate stream clarity, Indicates the complexity of the first bitrate stream. The first bitrate stream is directly converted from the original single stream, retaining the key frame R1. However, the difference from the original single stream is that the frame rate f′ of the first bitrate stream is significantly reduced and the expected clarity of non-key frames is and complexity changes.

6. A pseudo dual-stream video encoding and transmission method for real-time video as claimed in claim 2, characterized in that: In S103, a second bitrate path flow is modeled according to a stream level bitrate-quantization model, and the bitrate of the second bitrate path flow is as follows: Wherein, b" represents the bit rate of the second bit rate stream, f represents the frame rate of the first bit rate stream, is the first bitrate stream clarity, represents the complexity of the first bitrate stream. The first non-key frame to be replaced needs to maintain the same clarity q1 as the key frame in the first stream. Since the frame rate f does not change, its complexity c"1 can also be predicted based on the past frames using a linear model. In addition, the subsequent frames from 2 to Tf need to maintain the same clarity and average clarity as 7. A pseudo dual-stream video encoding and transmission method for real-time video as claimed in claim 1, characterized in that: In S200, the first bit rate path stream and the second bit rate path stream are encoded according to preset rules to generate two data packets; the specific method includes: dual-stream encoding needs to be activated only when the original video stream encodes the latest frame as a key frame according to its default key frame decision; when the key frame is generated, the video sender immediately creates a secondary encoder to encode the parallel stream, and the encoder is essentially a clone of the original encoder, which uses the original stream as the basis to continue the encoding process. Correspondingly, the receiving end also activates a secondary decoder and implements a decode-first-playback strategy on the same frame.

8. The pseudo dual-stream video encoding and transmission method for real-time video according to claim 1, characterized in that: The first code rate path retains the original key frame and is used to transmit the burst traffic generated by the key frame encoded within the frame, and the transmission of the key frame is used to ensure that decoding or transmission errors will not be accelerated and accumulated during the inter-frame propagation process; During the transmission of the key frame of the first bitrate stream, the newly captured video frame is no longer encoded, but skipped in large numbers until the key frame transmission is confirmed to be completed through the RTCP feedback message; when the key frame transmission is completed, the latest frame is encoded as a P frame to continue to provide a reference function. If the volume of the P frame is outside the preset range due to being far away from the reference frame, the above process is repeated to obtain the latest P frame until the volume of the latest P frame and the target bitrate are within the preset range; the second bitrate stream is used for real-time playback, without considering the functions of providing the best encoding and decoding reference for subsequent frames and limiting error propagation. The second bitrate stream is started when a key frame appears in the original stream, and the key frame is synchronously encoded into a non-key frame with a smaller data volume; Once the second bitrate stream is turned on, the subsequent continuously captured video frames will be encoded, and the video frames are completely composed of non-key frames to ensure the continuity of playback.

9. The pseudo dual-stream video encoding and transmission method for real-time video according to claim 1, characterized in that: In S300, the two data packets are temporarily stored in the Pacer sending queue, and the data packets are sent to the network according to a preset priority. The specific method includes: sorting the priorities according to the packet types, and the priorities from high to low are audio packets, retransmission packets, video packets, FEC redundant packets and Padding packets; and the data packets generated by the second code rate stream encoding are set to a higher priority than the data packets generated by the first code rate stream encoding, and the second code rate stream is preferably transmitted. When the two conditions that there are no second code rate stream data packets to be sent in the Pacer queue and there is still available code rate budget are met at the same time, the first code rate stream is sent.

10. A pseudo dual-stream video encoding and transmission system for real-time video, characterized in that: include: Pseudo dual-stream encoding rate allocation module, pseudo dual-stream encoding module and transmission priority control module; wherein: The pseudo dual-stream encoding bit rate allocation module is used to analyze the real-time video, obtain the real-time video target bit rate when a key frame is generated in the real-time video, and allocate the real-time video target bit rate based on the bit rate-quantization model to obtain a first bit rate path flow and a second bit rate path flow; A pseudo dual-stream encoding module, used for encoding the first code rate stream and the second code rate stream according to a preset rule to generate two data messages; The transmission priority control module is used to temporarily store the two data messages in the Pacer sending queue and send the data messages to the network according to the preset priority.

Images