Adaptive cross-frame forward error correction method and device for low-latency video streaming

Abstract

This invention proposes an adaptive cross-frame forward error correction method for low-latency video streams, comprising: acquiring cached video frame data within a protection block; calculating the minimum expected total cost achievable by immediately sending the current protection block as a first minimum expected total cost; calculating the minimum expected total cost achievable by sending the protection block to add the next frame of video as a second minimum expected total cost; if the second minimum expected total cost is less than the first minimum expected total cost, keeping the protection block open to add the next frame of video; if the second minimum expected total cost is greater than or equal to the first minimum expected total cost, closing the protection block and encoding and sending the current protection block. This method significantly improves the video stream's resilience to sudden packet loss and image quality while ensuring a low-latency interactive experience in mobile network environments.

Description

Technical Field

[0001] This invention relates to the field of real-time video streaming, and more particularly to an adaptive cross-frame forward error correction method and apparatus for low-latency video streams. Background Technology

[0002] With the development of mobile communication networks, real-time video streaming (RTVS) applications (such as high-definition video conferencing, cloud gaming, and low-latency live streaming) have become major traffic sources on the internet. Unlike video-on-demand services, the core requirement of RTVS applications is "strong interactivity," typically requiring end-to-end latency to be controlled within 150ms. However, mobile network channels are inherently highly dynamic, and multipath effects and signal attenuation easily lead to data packet loss, becoming the primary factor restricting transmission quality. To combat packet loss and ensure low latency, existing technologies typically employ forward error correction (FEC) mechanisms: the sending end adds error correction information using coding algorithms to generate redundant data packets when transmitting the original media data. The receiving end can recover lost data using these redundant data packets without waiting for retransmission; FEC eliminates the latency penalty of retransmission. Existing mainstream FEC schemes typically employ a "per-frame protection" strategy. The basic principle is to treat each video frame as an independent FEC (Encoding Block), and based on historical packet loss rate predictions, allocate a certain proportion of redundant data packets to that frame. The basic process is as follows: Figure 1 As shown, the system assumes that packet loss is relatively uniform over time, attempting to make each frame bear its own packet loss risk independently. However, this existing forward error correction (FEC) mechanism suffers from inefficient resource allocation, and a large number of damaged frames cannot be recovered through receiver FEC. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes an adaptive cross-frame forward error correction method and apparatus for low-latency video streams, which maximizes packet loss recovery efficiency while ensuring a low-latency interactive experience.

[0004] This invention provides an adaptive cross-frame forward error correction method for low-latency video streams, comprising:

[0005] Get the cached video frame data within the protection block, calculate the minimum expected total cost that can be achieved by immediately sending the current protection block, and use it as the first minimum expected total cost;

[0006] Calculate the minimum expected total cost that can be achieved by sending the guard block to add the next frame of video, and use it as the second minimum expected total cost;

[0007] If the second minimum expected total cost is less than the first minimum expected total cost, then keep the protection block open to add the next frame of video;

[0008] If the second minimum expected total cost is greater than or equal to the first minimum expected total cost, then the protection block is closed and the current protection block encoding is sent.

[0009] In one embodiment of the present invention, the first minimum expected total cost and the second minimum expected total cost are both calculated based on a spatiotemporal cost joint model; the spatiotemporal cost joint model defines the expected total cost as a combination of time cost and space cost.

[0010] The space overhead corresponds to the bandwidth consumption caused by forward error correction redundancy, and the time overhead corresponds to the latency caused by cross-frame waiting, transmission, decoding and retransmission.

[0011] In one embodiment of the present invention, the process of calculating the first minimum expected total cost is specifically as follows:

[0012] The protection boundary is set to the current Nth frame of video. All optional redundant data packets within the preset redundancy range are traversed and substituted into the spatiotemporal overhead joint model to calculate the corresponding expected total overhead. The minimum expected total overhead is taken as the first minimum expected total overhead, and the number of redundant data packets corresponding to the first minimum expected total overhead is the first optimal redundancy of the current protection block.

[0013] In one embodiment of the present invention, the process of calculating the second minimum expected total cost is specifically as follows:

[0014] Predict the size of the next frame of video, add the next frame of video to the protection block, and extend the protection boundary to N+1 frames of video;

[0015] Traverse all the optional redundant data packets within the preset redundancy range, substitute them into the spatiotemporal overhead joint model to calculate the corresponding expected total overhead, take the minimum expected total overhead as the second minimum expected total overhead, and the number of redundant data packets corresponding to the second minimum expected total overhead is the second optimal redundancy of the current protection block.

[0016] In one embodiment of the present invention, the method of predicting the size of the next frame of video includes any one of the following: target bitrate prediction based on adaptive bitrate, prediction based on time series analysis model, or directly reading the size parameters of the next frame of video pre-buffered by the video encoder.

[0017] In one embodiment of the present invention, sending the current protection block code includes:

[0018] Extract all cached video frames within the protection block and perform forward error correction coding. Generate a corresponding number of redundant data packets based on the first optimal redundancy amount corresponding to the first minimum expected total overhead. Combine the redundant data packets with the original data packets to form a transmission sequence for transmission. The original data packets are generated from the original video frames after video encoding.

[0019] In one embodiment of the present invention, it further includes:

[0020] Obtain the size and frame interval information of the current frame video within the protected block;

[0021] Obtain the current network status, which includes packet loss rate, round-trip time, one-way delay, and transmission rate;

[0022] And, to obtain protected boundaries.

[0023] In one embodiment of the present invention, the weighting coefficients of spatial overhead and temporal overhead in the spatiotemporal overhead joint model can be dynamically adjusted: when insufficient device power or traffic billing sensitivity is detected, the weight coefficient corresponding to spatial overhead is increased; when video keyframes are detected in the current protection block, the penalty weight corresponding to retransmission delay in temporal overhead is increased.

[0024] In one embodiment of the present invention, the spatiotemporal cost joint model defines the expected total cost as a combination of time cost and space cost, including: a linear weighted combination of time cost and space cost, or a combination based on logarithmic utility function, a combination based on fractional utility function, an exponentially penalized combination, or a higher-order polynomial combination.

[0025] In one embodiment of the present invention, the metrics for measuring spatial overhead include: transmission bit rate, or, video multi-method evaluation fusion metric VMAF, structural similarity index metric SSIM, and peak signal-to-noise ratio PSNR.

[0026] In one embodiment of the present invention, the forward error correction coding algorithm used to generate redundant data packets includes: Reed-Solomon coding, or default coding, Cauchy coding, fountain coding, and random linear network coding.

[0027] In one embodiment of the present invention, if the forward error correction decoding fails when the receiving end decodes the protection block data, a hybrid automatic retransmission request mechanism is triggered to merge the subsequently retransmitted data packets with the received error packets for decoding.

[0028] Another aspect of the present invention provides an adaptive cross-frame forward error correction device for low-latency video streams, comprising:

[0029] A video encoder is used to encode raw video streams to generate native data packets;

[0030] The decision-maker is configured to, for the video frame data already cached within the protection block, calculate the minimum expected total cost that can be achieved by immediately sending the current protection block, as the first minimum expected total cost; calculate the minimum expected total cost that can be achieved by sending the protection block with the next video frame added, as the second minimum expected total cost; and, if the second minimum expected total cost is less than the first minimum expected total cost, output an instruction to keep the protection block open; if the second minimum expected total cost is greater than or equal to the first minimum expected total cost, output an instruction to close the protection block.

[0031] The forward error correction encoder is used to receive instructions output by the decision-maker and perform corresponding operations: if a protection block keep open instruction is received, the currently encoded video frame is stored in the protection block buffer; if a protection block close instruction is received, forward error correction encoding is performed on all buffered video frames in the protection block to generate redundant data packets.

[0032] A packet transmitter is used to combine the redundant data packets and the original data packets into a transmission sequence for transmission.

[0033] As can be seen from the above solutions, the advantages of the present invention are:

[0034] The adaptive cross-frame forward error correction method for low-latency video streams provided by this invention calculates the minimum expected total cost of immediately sending the current protection block and the minimum expected total cost of sending the protection block for adding the next frame of video when each frame of video encoding is completed. These two calculations are compared to determine whether to perform the operation of "keeping the protection block open to add the next frame of video" or "closing the protection block and encoding and sending the current protection block." This method employs frame-by-frame real-time decision-making logic, automatically balancing the packet loss recovery benefits of cross-frame redundancy with the latency costs of waiting. It strictly controls the additional waiting latency introduced by cross-frame encoding, achieving a dynamic balance between recovery rate and latency. Without increasing average bandwidth overhead, it simultaneously ensures a low-latency interactive experience and a high recovery success rate in sudden packet loss scenarios. It does not rely on long-term network state prediction, significantly improving the video stream's resilience to sudden packet loss and image quality. Attached Figure Description

[0035] Figure 1 A schematic diagram illustrating the "frame-by-frame protection" strategy employed in existing FEC (Frame-by-Frame Protection) schemes;

[0036] Figure 2 The diagram illustrates the overall flow of an adaptive cross-frame forward error correction method for low-latency video streams according to an embodiment of the present invention.

[0037] Figure 3 A schematic diagram of the structure of an adaptive cross-frame forward error correction device for low-latency video streams provided in an embodiment of the present invention is shown.

[0038] The attached figures are labeled as follows:

[0039] 10: Sending end;

[0040] 20: Receiver;

[0041] 300: Adaptive cross-frame forward error correction device for low-latency video streams;

[0042] 310: Video encoder;

[0043] 320: Decision Maker;

[0044] 330: Forward error correction encoder;

[0045] 340: Packet sender;

[0046] 410: Video decoder;

[0047] 420: Forward error correction decoder. Detailed Implementation

[0048] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0049] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] Studies have shown that high packet loss events in mobile networks are characterized by extremely short bursts, with approximately 80% of these events lasting only one frame interval. This rapidly changing channel characteristic makes it difficult for conventional prediction algorithms based on historical statistical data to accurately track packet loss in the next frame. This leads to insufficient redundancy allocation during bursts and excessive redundancy allocation during stable network conditions. Furthermore, current technologies typically protect each video frame as an independent coding block. This strategy forcibly isolates redundant resources between frames, preventing the use of unused redundant resources in adjacent frames for recovery when a frame experiences sudden packet loss. This temporal resource isolation is the root cause of the high recovery failure rate even with high redundancy rates.

[0051] To address this, the present invention provides an adaptive cross-frame forward error correction method for low-latency video streams. After each video frame is encoded, instead of immediately deciding whether to send it, the method first calculates the expected total cost of QoE for two strategies: "immediately send the current protection block" and "wait for the next video frame to merge and send the protection block". By comparing the cost values ​​of the two strategies, the method dynamically decides whether to end the current protection block to generate redundant data packets for transmission, or to keep the protection block open to wait for the next video frame.

[0052] Specifically, refer to Figure 2 As shown, Figure 2 A flowchart illustrating an adaptive cross-frame forward error correction method provided in an embodiment of the present invention is shown.

[0053] An adaptive cross-frame forward error correction method for low-latency video streams includes the following steps:

[0054] Step S1: When each frame of video encoding is completed, obtain the cached video frame data in the protection block, and calculate the minimum expected total cost that can be achieved by immediately sending the current protection block, as the first minimum expected total cost.

[0055] Step S2: Calculate the minimum expected total cost that can be achieved by sending the guard block to add the next frame of video, and use it as the second minimum expected total cost.

[0056] Step S3: If the second minimum expected total cost is less than the first minimum expected total cost, then keep the protection block open to add the next frame of video.

[0057] Step S4: If the second minimum expected total cost is greater than or equal to the first minimum expected total cost, then close the protection block and send the current protection block encoding.

[0058] In this embodiment, a frame-by-frame real-time decision-making logic is adopted. Each frame compares the overhead of the two strategies of "send immediately" and "wait for the next frame". It automatically weighs the packet loss recovery benefits of cross-frame redundancy against the latency cost of waiting. Without increasing the average bandwidth overhead, it ensures both low-latency interactive experience and recovery success rate in sudden packet loss scenarios, without relying on long-term network state prediction.

[0059] In one embodiment, the cached video frame data d(1)...d(N) within the protection block are first obtained, and the size and frame interval information of the current frame video are obtained from the video encoder; at the same time, the current network status, including packet loss rate, round-trip time (RTT), one-way delay (OWD), and sending rate, is obtained from the underlying transport protocol stack (such as QUIC); and the current redundancy protection boundary is obtained from the FEC encoder.

[0060] In one embodiment, to quantify the impact of different decisions on user experience, a spatiotemporal overhead joint model is constructed. Based on this model, a first minimum expected total overhead and a second minimum expected total overhead are calculated. The spatiotemporal overhead joint model defines the expected total overhead as a combination of time overhead and space overhead, such as a linearly weighted combination. However, it is not limited to a linear combination and can also employ other nonlinear combinations (e.g., combinations based on logarithmic utility functions, fractional utility functions, exponentially penalized combinations, or higher-order polynomial combinations). Any model that accurately maps the combined impact of latency and bandwidth consumption on user experience (QoE) falls within the scope of this invention.

[0061] The space overhead corresponds to the bandwidth consumption caused by forward error correction redundancy, and the time overhead corresponds to the latency caused by cross-frame waiting, transmission, decoding and retransmission.

[0062] In a specific implementation scheme, the spatiotemporal overhead joint model defines the expected total overhead E. overhead The weighted sum of time and space costs:

[0063]

[0064] in, Space overhead, also known as redundancy rate, is calculated using the following formula: λ is a weighting coefficient used to adjust the system's sensitivity to bandwidth consumption. N represents the number of redundant data packets, and N represents the number of unsent video frames currently accumulated in the protection block.

[0065] In one embodiment, the metrics for measuring spatial overhead include: transmission bitrate, or video perceptual quality metrics such as Video Multi-Method Evaluation Fusion (VMAF), Structural Similarity Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR). In this case, "spatial overhead" represents the "video quality loss" item, thereby achieving more direct visual experience optimization.

[0066] The time overhead reflects the probabilistic impact of the FEC strategy on frame transmission delay, and is a weighted average of "successful recovery delay" and "failure recovery delay". It is represented as:

[0067]

[0068] Let the total number of data packets within the protected block (i.e., the sum of native data packets and redundant data packets) be... The network packet loss rate is . No. The probability of recovering frame-by-frame video using FEC. for:

[0069]

[0070] First item This indicates the total number of lost packets. The probability of ""; the second term This indicates the total number of lost packets. The probability that "the i-th frame has no packet loss" is calculated by subtracting the two terms. This gives the probability that "the i-th frame of the video has packet loss but has been recovered".

[0071] Similarly, the probability of recovery failure for:

[0072]

[0073] In the formula, the first term This indicates the total number of packets lost in the protection block. The probability of "redundant data packets (i.e., FEC failure)"; the second term This represents the probability that "the protection block fails, but the i-th frame of video does not experience packet loss (remains intact)". Subtracting the two gives the probability that "FEC fails and the i-th frame of video is indeed damaged (requires retransmission)".

[0074] For the expected delay component of the time overhead, if recovery is successful via FEC, the expected delay of the frame depends on the retransmission strategy employed. .definition For retransmission policy indicator variables: This indicates FEC coupled retransmission (serial), meaning that retransmission is only triggered if FEC fails; This indicates independent retransmission (parallel), meaning retransmission is triggered immediately after packet loss, and the final latency depends on the faster of FEC recovery and ARQ retransmission. If FEC recovery is achieved through coupled retransmission, the expected latency of the frame depends on the time it takes for the entire protection block to complete. This is the main cost of cross-frame protection. For the first frame in the protection block... For a single frame of video, it must wait for subsequent frames. arrive All redundant data packets were encoded and transmitted.

[0075] For the first in the protection block Frame video, its successful recovery latency The calculation is as follows:

[0076]

[0077] in, The delay required for the entire protection block to complete (the main cost of cross-frame protection):

[0078]

[0079] in: To account for the encoding delay while waiting for subsequent frames to be generated, For frame interval; For the serialization transmission delay of the remaining data packets, For transmission rate; This represents the one-way propagation delay of the physical link.

[0080] The latency for direct retransmission (only in parallel strategies) (The following is considered a competing path), defined as follows:

[0081]

[0082] On the other hand, if FEC recovery fails, the system must fall back to ARQ retransmission. This means that latency will increase by at least one RTT round-trip time, as well as potential waiting time for multiple retransmissions. The expected video frame latency at this point also depends on the retransmission strategy. :

[0083]

[0084] when (Serial Strategy): The system must first wait for the entire FEC protection block to complete transmission (time consumption) This process confirms decoding failure before initiating a retransmission. This results in a significant increase in latency, adding at least one RTT of retransmission overhead to the FEC block latency.

[0085] when (In parallel strategy) Since the retransmission request is sent immediately upon packet loss and does not depend on the completion of the FEC block, the failure latency is determined solely by the retransmission process itself (i.e., ).

[0086] in This represents the expected number of retransmission rounds. Expected number of retransmissions. This reflects the current network packet loss rate and the difficulty in completely receiving a frame of video data (including...) (Number of data packets), the number of retransmission rounds the system is expected to perform, calculated using the following formula:

[0087]

[0088] in This indicates the maximum number of retransmission rounds allowed by the system. This is a system-preset upper limit to prevent network congestion or excessive latency caused by unlimited retransmissions; Indicates the first The total number of data packets contained in a frame of video; This represents the average packet loss rate within the current protected block. It's worth noting that in actual system deployments, the desired retransmission rounds can be specified. Set it to an empirical constant (e.g., 1) to reduce computational complexity and ensure real-time decision-making efficiency.

[0089] In certain scenarios, especially high-dynamic-range motion scenes, the video frame size may momentarily exceed the ABR target value, causing its transmission time to exceed one frame interval. Therefore, It needs to be expanded to include both "transmission-restricted" and "generation-restricted" cases.

[0090]

[0091] in This is a transmission constraint, meaning that when network bandwidth is low or encoding overshoot occurs, and video frames cannot be transmitted within the frame interval, the bottleneck lies in the transmission capacity of the physical link. In this case, the latency depends on the total time required to send all remaining native and redundant data packets in the protection block. The constraint is that when network bandwidth is sufficient, video frames can be transmitted within their generation intervals; the bottleneck lies in waiting for the encoding of the last frame N in the guard block. In this case, the latency is mainly determined by the encoding wait time. It is determined by the transmission time of the last batch of data packets.

[0092] In one embodiment, the weighting coefficients of spatial and temporal overhead in the spatiotemporal overhead joint model can be dynamically adjusted: when insufficient device power or sensitivity to traffic metering is detected, the weighting coefficient corresponding to spatial overhead is increased. When a video keyframe is detected within the current protection block, the penalty weight ω corresponding to the retransmission delay in the time overhead is increased to reduce the failure probability. By dynamically balancing the waiting time overhead, unnecessary expansion of the protection boundary is avoided, and the increased latency and ineffective repair caused by excessive waiting in fixed multi-frame protection are eliminated, achieving dynamic optimization of transmission reliability and real-time interactive experience.

[0093] By constructing a joint evaluation model of temporal and spatial overhead, latency and bandwidth are quantified in a unified manner. This model creatively maps the "time overhead" (decoding latency caused by waiting across frames) and "spatial overhead" (bandwidth occupied by redundant data packets) brought about by FEC protection into a comprehensive negative impact on user experience (QoE). By introducing a high penalty weight for recovery failure (i.e., retransmission), the model can accurately identify and avoid "invalid recovery" (i.e., the frame is recovered but the playback deadline has been missed), thereby improving picture quality while strictly controlling tail latency.

[0094] In one embodiment, the above-described spatiotemporal overhead joint evaluation model is used to obtain the cached video frame data within the protection block. This video frame data is a set of unsent frames (N frames). The minimum expected total overhead that can be achieved by immediately sending the current protection block is calculated as the first minimum expected total overhead.

[0095] Specifically, the protection boundary is set to the current Nth frame of the video within the protection block. The number of all optional redundant data packets red within a preset redundancy range (e.g., 0%~100%) is traversed, and each is substituted into the spatiotemporal overhead joint model to calculate the corresponding expected total overhead. The minimum expected total overhead is then taken as the first minimum expected total overhead E. overhead_now And the number of redundant data packets corresponding to the first minimum expected total overhead, red now This represents the first optimal redundancy for the current protection block. The preset redundancy range can be determined by the redundancy rate, which is the preset ratio of the number of redundant data packets to the total number of original data packets. The number of redundant data packets cannot exceed the total number of original data packets.

[0096] Furthermore, assuming the next video frame is added to the protection block, the minimum expected total cost achievable by sending the protection block with the added next video frame is calculated, which is taken as the second minimum expected total cost. Specifically, the size of the next video frame is predicted, and the next video frame is added to the protection block, extending the protection boundary to N+1 video frames; similarly, the number of all optional redundant data packets within a preset redundancy range (e.g., 0%~100%) is traversed, and the corresponding expected total cost is calculated by substituting them into the spatiotemporal cost joint model, with the minimum expected total cost taken as the second minimum expected total cost E. overhead_next And the number of redundant data packets corresponding to the second minimum expected total overhead is the second optimal redundancy of the current protection block. next .

[0097] By dynamically adjusting the protection boundary and redundancy at the frame level, the problem of sudden packet loss is solved while effectively controlling the tail latency.

[0098] In one specific implementation, the method for predicting the size of the next frame of video includes any of the following: target bitrate prediction based on adaptive bitrate, prediction based on time series analysis models (such as LSTM, linear regression), or directly reading the size parameters of the next frame of video pre-buffered by the video encoder, but the present invention is not limited thereto.

[0099] In one embodiment, a first minimum expected total cost E is further compared. overhead_now With the second minimum expected total expenditure E overhead_next Size.

[0100] If the second minimum expected total cost is less than the first minimum expected total cost, i.e., E overhead_next <Eoverhead_now This indicates that the "cross-frame smoothing benefits" (using lossless frames to mitigate the risk of sudden packet loss) brought by introducing the next frame of video outweigh the "waiting latency cost," so the protection block remains open to add the next frame of video. At this time, the instruction to keep the protection block open is output, i.e., the instruction pb=0, and the next frame of video is not sent yet. The loop for the next frame of video begins, where pb is the protection boundary flag.

[0101] If the second minimum expected total cost is greater than or equal to the first minimum expected total cost, i.e., E overhead_next ≧ E overhead_now If the delay cost of waiting exceeds the benefit of waiting across frames, the marginal benefit of continuing to wait diminishes, and the delay risk increases, then a protection block shutdown instruction is output, i.e., instruction pb=1, closing the protection block and sending the current protection block encoding. Specifically, all buffered video frames (N frames of data) within the protection block are extracted and forward error correction encoding is performed. Based on the first optimal redundancy amount corresponding to the first minimum expected total cost, a corresponding number of redundant data packets are generated, i.e., red... now A redundant data packet is generated and combined with the original data packet to form a transmission sequence.

[0102] The raw data packets are generated from original video frames after video encoding. Each original video frame, after being compressed and encoded by a standard video encoder such as H.264 / H.265 / AV1, is divided into several raw data packets of fixed / variable length. These raw data packets directly store the original content necessary for playback, such as video pixel values ​​and encoding parameters. If these contents are missing and cannot be recovered, the corresponding video frame will exhibit screen tearing, stuttering, or even be undecodeable. The raw data packets do not contain any redundant information related to error correction. Redundant data packets, on the other hand, are auxiliary data packets generated by a forward error correction coding algorithm and are used solely for packet loss recovery. They do not carry any independently decodeable video content. Their core function is to cope with transmission packet loss. When some raw data packets within the same protection block are lost, the receiving end can use "unlost raw data packets + redundant data packets" to reverse-calculate the lost data packet content through error correction decoding, avoiding triggering retransmissions and reducing latency.

[0103] By real-time sensing of network channel status, after each frame is encoded, it determines in real time whether to "keep the guard block open" to obtain cross-frame gain or "close the guard block" to avoid latency costs, and dynamically adjusts the number of frames encoded for the guard block. This mechanism, which updates decisions in real time based on the latest status, is more adaptable to highly dynamic network environments and drastically fluctuating video bitrates. This allows video frames suffering severe burst packet loss in mobile networks to be recovered using the surplus redundant resources in adjacent frames. By constructing a "cross-frame resource pool," it effectively smooths out transient fluctuations in the packet loss rate, significantly improving the recovery success rate in burst packet loss scenarios without increasing average bandwidth overhead.

[0104] In one embodiment, the forward error correction coding can employ Reed-Solomon (RS) codes, XOR codes (default codes), Cauchy codes, RaptorQ codes, or RLNC (random linear network coding). As long as the coding method supports cross-data packet recovery, the adaptive boundary decision logic of this invention can be applied; this invention does not impose specific limitations.

[0105] Furthermore, in one embodiment, if forward error correction decoding fails when decoding protection block data at the receiving end, a hybrid automatic retransmission request mechanism can be further triggered to merge and decode the subsequently retransmitted data packets with the already received erroneous data packets, thereby further improving the recovery rate.

[0106] Therefore, in summary, the adaptive cross-frame forward error correction method for low-latency video streams provided by this invention calculates the minimum expected total cost achievable by immediately sending the current protection block and the minimum expected total cost achievable by sending the protection block for adding the next frame of video when each frame of video encoding is completed. These two calculations are compared to determine whether to perform the operation of "keeping the protection block open to add the next frame of video" or "closing the protection block and encoding and sending the current protection block." This method employs frame-by-frame real-time decision-making logic, automatically balancing the packet loss recovery benefits of cross-frame redundancy with the latency costs of waiting. It strictly controls the additional waiting latency introduced by cross-frame encoding, achieving a dynamic balance between recovery rate and latency. Without increasing average bandwidth overhead, it simultaneously ensures a low-latency interactive experience and a high recovery success rate in sudden packet loss scenarios. It does not rely on long-term network state prediction, significantly improving the video stream's resilience to sudden packet loss and image quality.

[0107] In one embodiment, an adaptive cross-frame forward error correction device for low-latency video streams is further provided. This embodiment is a device embodiment corresponding to the above embodiment, such as... Figure 3 As shown, Figure 3 This diagram illustrates the overall structure of an adaptive cross-frame forward error correction device for low-latency video streams according to an embodiment of the present invention. This device embodiment can be implemented in conjunction with the embodiments described above. The relevant technical details mentioned in the above embodiments remain valid in this device embodiment, and will not be repeated here to avoid repetition.

[0108] An adaptive cross-frame forward error correction device 300 for low-latency video streams includes:

[0109] The video encoder 310 is used to encode the raw video stream to generate native data packets and transmit the video size d(i) of each frame to the decision maker in real time.

[0110] Decision maker 320 is configured to, for the video frame data already cached within the protection block, calculate the minimum expected total cost that can be achieved by immediately sending the current protection block as a first minimum expected total cost; calculate the minimum expected total cost that can be achieved by sending the protection block with the next video frame added as a second minimum expected total cost; and, if the second minimum expected total cost is less than the first minimum expected total cost, output an instruction to keep the protection block open; if the second minimum expected total cost is greater than or equal to the first minimum expected total cost, output an instruction to close the protection block.

[0111] The forward error correction encoder 330 is used to receive instructions output by the decision-maker and perform corresponding operations: if a protection block keep open instruction is received, the currently encoded video frame is stored in the protection block buffer; if a protection block close instruction is received, forward error correction encoding is performed on all buffered video frames in the protection block to generate redundant data packets.

[0112] Packet transmitter 340 is used to combine the redundant data packets and the original data packets into a transmission sequence for transmission, and to feed back the current network status (RTT, packet loss rate, bandwidth, etc.) to the decision-maker.

[0113] In this embodiment, the video encoder 310, decision maker 320, forward error correction encoder 330, and packet transmitter 340 are deployed at the transmitting end 10, and corresponding forward error correction decoder 420, video decoder 410, etc. are deployed at the receiving end 20.

[0114] Furthermore, this adaptive cross-frame forward error correction device can not only run as a software module on a general-purpose CPU, but also be embedded in an FPGA, ASIC, or SmartNIC, thereby accelerating the FEC calculation and decision-making process through hardware offloading.

[0115] In addition, embodiments of the present invention also provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the aforementioned adaptive cross-frame forward error correction method for low-latency video streams, and achieve the same technical effect.

[0116] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described adaptive cross-frame forward error correction method for low-latency video streams, and achieves the same technical effect.

[0117] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. An adaptive cross-frame forward error correction method for low-latency video streams, characterized in that, include: Get the cached video frame data within the protection block, calculate the minimum expected total cost that can be achieved by immediately sending the current protection block, and use it as the first minimum expected total cost; Calculate the minimum expected total cost that can be achieved by sending the guard block to add the next frame of video, and use it as the second minimum expected total cost; If the second minimum expected total cost is less than the first minimum expected total cost, then keep the protection block open to add the next frame of video; If the second minimum expected total cost is greater than or equal to the first minimum expected total cost, then the protection block is closed and the current protection block encoding is sent.

2. The method according to claim 1, characterized in that, Both the first minimum expected total cost and the second minimum expected total cost are calculated based on the spatiotemporal cost joint model; the spatiotemporal cost joint model defines the expected total cost as the combined value of time cost and space cost. The space overhead corresponds to the bandwidth consumption caused by forward error correction redundancy, and the time overhead corresponds to the latency caused by cross-frame waiting, transmission, decoding and retransmission.

3. The method according to claim 2, characterized in that, The process of calculating the first minimum expected total cost is as follows: The protection boundary is set to the current Nth frame of video. All optional redundant data packets within the preset redundancy range are traversed and substituted into the spatiotemporal overhead joint model to calculate the corresponding expected total overhead. The minimum expected total overhead is taken as the first minimum expected total overhead, and the number of redundant data packets corresponding to the first minimum expected total overhead is the first optimal redundancy of the current protection block.

4. The method according to claim 3, characterized in that, The process of calculating the second minimum expected total cost is as follows: Predict the size of the next frame of video, add the next frame of video to the protection block, and extend the protection boundary to N+1 frames of video; Traverse all the optional redundant data packets within the preset redundancy range, substitute them into the spatiotemporal overhead joint model to calculate the corresponding expected total overhead, take the minimum expected total overhead as the second minimum expected total overhead, and the number of redundant data packets corresponding to the second minimum expected total overhead is the second optimal redundancy of the current protection block.

5. The method according to claim 4, characterized in that, The method for predicting the size of the next frame of video includes any of the following: target bitrate prediction based on adaptive bitrate, prediction based on time series analysis model, or directly reading the size parameters of the next frame of video pre-buffered by the video encoder.

6. The method according to claim 3, characterized in that, Send the current protection block encoding, including: Extract all cached video frames within the protection block and perform forward error correction coding. Generate a corresponding number of redundant data packets based on the first optimal redundancy amount corresponding to the first minimum expected total overhead. Combine the redundant data packets with the original data packets to form a transmission sequence for transmission. The original data packets are generated from the original video frames after video encoding.

7. The method according to claim 1, characterized in that, Also includes: Obtain the size and frame interval information of the current frame video within the protected block; Obtain the current network status, which includes packet loss rate, round-trip time, one-way delay, and transmission rate; And, to obtain protected boundaries.

8. The method according to claim 2, characterized in that, The weighting coefficients of spatial overhead and temporal overhead in the spatiotemporal overhead joint model can be dynamically adjusted: when insufficient device power or sensitivity to traffic billing is detected, the weight coefficient corresponding to spatial overhead is increased; when video keyframes are detected in the current protection block, the penalty weight corresponding to retransmission delay in temporal overhead is increased.

9. The method according to claim 2, characterized in that, The joint time-space cost model defines the expected total cost as a combination of time cost and space cost, including: a linearly weighted combination of time cost and space cost, or a combination based on a logarithmic utility function, a combination based on a fractional utility function, an exponentially penalized combination, or a combination of higher-order polynomials.

10. The method according to claim 2, characterized in that, The metrics for spatial overhead include: transmission bit rate, or, video multi-method evaluation fusion metric VMAF, structural similarity index metric SSIM, and peak signal-to-noise ratio PSNR.

11. The method according to claim 6, characterized in that, The forward error correction coding algorithms used to generate redundant data packets include: Reed-Solomon coding, or default coding, Cauchy coding, fountain coding, and random linear network coding.

12. The method according to claim 1, characterized in that, When the receiving end decodes the protection block data, if the forward error correction decoding fails, the hybrid automatic retransmission request mechanism is triggered to merge the subsequently retransmitted data packets with the already received erroneous data packets for decoding.

13. An adaptive cross-frame forward error correction device for low-latency video streams, characterized in that, Include: A video encoder is used to encode raw video streams to generate native data packets; The decision-maker is configured to, for the video frame data already cached within the protection block, calculate the minimum expected total cost that can be achieved by immediately sending the current protection block, as the first minimum expected total cost; calculate the minimum expected total cost that can be achieved by sending the protection block with the next video frame added, as the second minimum expected total cost; and, if the second minimum expected total cost is less than the first minimum expected total cost, output an instruction to keep the protection block open; if the second minimum expected total cost is greater than or equal to the first minimum expected total cost, output an instruction to close the protection block. A forward error correction encoder is used to receive instructions output by the decision-maker and perform corresponding operations: if a protection block keep open instruction is received, the currently encoded video frame is stored in the protection block buffer; If a protection block closing command is received, forward error correction encoding is performed on all buffered video frames within the protection block to generate redundant data packets; A packet transmitter is used to combine the redundant data packets and the original data packets into a transmission sequence for transmission.

Images