Video packet loss recovery method based on image odd-even row / column reorganization and block encoding

Abstract

This invention belongs to the field of video packet loss repair technology, specifically disclosing a video packet loss repair method based on image odd-even row / column reassembly and block coding. The method includes: at the transmitting end, reassembling the original video frames using odd-even row / column reassembly to generate a reassembled image; encoding the reassembled image using a single encoder; separating two sub-streams from the single encoded stream based on region markers, corresponding to the encoded block data of odd-numbered and even-numbered regions respectively, and encapsulating each separated sub-stream with data packets; at the receiving end, receiving and parsing the data packets, and performing block-level precise repair when packet loss is detected; and interleaving the pixels of the repaired odd-numbered and even-numbered row / column regions to restore the original image format. This invention employs the above-mentioned video packet loss repair method based on image odd-even row / column reassembly and block coding, without increasing latency or bandwidth, significantly improving the effect of video transmission packet loss repair, and is particularly suitable for real-time video communication scenarios.

Description

Technical Field

[0001] This invention relates to the field of video packet loss repair technology, and in particular to a video packet loss repair method based on image parity row / column recombination and block coding. Background Technology

[0002] With the rapid development of online video applications, the transmission quality of video data in unreliable network environments has become a key factor limiting user experience. Especially in real-time video transmission scenarios, network packet loss can lead to a severe degradation in image quality, impacting user experience. Therefore, packet loss recovery methods are crucial for the user experience of online video applications.

[0003] Existing video packet loss repair technologies mainly suffer from the following problems:

[0004] 1. Traditional Forward Error Correction (FEC) technology: It recovers lost packets by adding redundant data, but it increases bandwidth overhead by about 30% and is inefficient when bandwidth is limited.

[0005] 2. Automatic Repeat Request (ARQ) mechanism: Requests retransmission after detecting packet loss, introducing a delay of 200-500ms, which is not suitable for real-time video transmission scenarios.

[0006] 3. Multiple Description Coding (MDC): This method samples the original image into multiple low-resolution images, encodes each image separately, and transmits them through multiple channels. The entire image can be reconstructed as long as one channel receives the complete image. While it offers some resilience against packet loss, it has the following drawbacks:

[0007] 1) When packet loss occurs, the image is reconstructed by interpolating or directly enlarging a low-resolution encoded image, resulting in a significant decrease in image quality.

[0008] 2) If all encoded descriptions cannot be received completely, repair is not possible;

[0009] 3) Interpolation operations on the entire image are costly;

[0010] 4) Multiple images are encoded independently, and the total bandwidth occupied after encoding is higher than that without segmentation.

[0011] 4. Traditional error concealment techniques: such as interpolation, weighted averaging of forward and backward frames, motion prediction, and other intra-frame and inter-frame error concealment techniques. Due to the low similarity between adjacent regions of complex images and the low similarity between consecutive frames of fast-moving images, obvious mosaic or image distortion will occur when the packet loss rate exceeds 10%, and video quality cannot be guaranteed.

[0012] Therefore, a video packet loss repair solution with low latency, low bandwidth overhead, high packet loss resistance, and high image quality is needed. This solution should be able to achieve accurate packet loss repair at the code block level. Summary of the Invention

[0013] The purpose of this invention is to provide a video packet loss repair method based on image odd / even row / column recombination and block coding. It makes full use of the high similarity features between odd and even rows / columns in video images and combines the characteristics of block coding technology. By rearranging and combining odd / even rows / columns and a precise block-level packet loss repair mechanism, it achieves efficient packet loss repair without increasing latency and bandwidth.

[0014] To achieve the above objectives, this invention provides a video packet loss repair method based on image parity row / column recombination and block coding, comprising the following steps:

[0015] S1. At the sending end, the original video frames are reassembled by combining the odd and even rows / columns of the image to generate a reassembled image;

[0016] S2. Encode the reconstructed image using a single encoder;

[0017] S3. Based on the region marker, separate two sub-streams, TS1 and TS2, from the single coded stream, corresponding to the coded block data of odd-numbered regions and even-numbered regions respectively, and encapsulate each separated sub-stream into data packets;

[0018] S4. Receive and parse data packets at the receiving end, and perform block-level precise repair when packet loss is detected;

[0019] S5. The pixels of the repaired odd-numbered row / column regions and even-numbered row / column regions are interleaved and recombined to restore the original image format.

[0020] Preferably, in S1, the image odd-even row / column recombination includes an odd-even row recombination mode and an odd-even column recombination mode;

[0021] The odd-even row recombination pattern is as follows:

[0022] Extract all odd-numbered and even-numbered row pixels from the original image F(n);

[0023] The odd-numbered rows are rearranged to the upper half of the image, maintaining their original order;

[0024] Even-numbered rows are rearranged to the lower half of the image, maintaining their original order;

[0025] Generate a reconstructed image F'(n) with the same width and height as the original image F(n).

[0026] The preferred parity column recombination pattern is:

[0027] Extract all odd-numbered and even-numbered columns of pixels from the original image F(n);

[0028] The odd-numbered columns are rearranged to the left half of the image, maintaining their original order;

[0029] The even-numbered columns are rearranged to the right half of the image, maintaining their original order;

[0030] Generate a reconstructed image F'(n) with the same width and height as the original image F(n).

[0031] Preferably, S2 is as follows:

[0032] The single encoder uses a standard algorithm based on block coding to encode the reconstructed whole image F'(n).

[0033] Preferably, the single encoder employs a standard algorithm based on block coding, including any one of H.264, H.265, and AV1.

[0034] Preferably, S3 is as follows:

[0035] For the odd-even row recombination pattern:

[0036] TS1 corresponds to all coded block data in the odd-numbered row region;

[0037] TS2 corresponds to all coded block data in even-numbered row regions;

[0038] For the odd-even column recombination pattern:

[0039] TS1 corresponds to all coded block data in the odd-numbered column region;

[0040] TS2 corresponds to all coded block data in the even-numbered column region.

[0041] Preferably, in S4, the specific process of receiving and parsing data packets at the receiving end is as follows:

[0042] S41. Receive and identify TS1 and TS2 data streams from different transmission channels, and extract the timestamp, sequence number and coded block location information from each data packet;

[0043] S42. Calculate the packet loss situation and the distribution information of the coding blocks contained in TS1 and TS2 respectively, maintain the packet loss matrix L(x,y,t) to record the spatiotemporal packet loss distribution, and realize packet loss location at the coding block level.

[0044] S43. Reassemble the received TS1 and TS2 data packets in their original encoding order to create a complete data stream for decoding, and mark the lost data block areas.

[0045] Preferably, in S4, the block-level precise repair based on packet loss includes:

[0046] Processing of regions without packet loss: For coded blocks in both odd and even regions where there is no packet loss, the pixel data of the corresponding region is directly extracted from the decoded reconstructed image;

[0047] One-sided packet loss region repair: For cases where only odd-numbered regions or only even-numbered regions of encoded blocks are lost, the lost regions are filled using inter-row / inter-column interpolation algorithms;

[0048] Double-sided packet loss region repair: When the encoded block at the same location in both regions is lost, the traditional packet loss repair scheme is used.

[0049] Preferably, S5 is as follows:

[0050] For the odd-even row recombination pattern:

[0051] Based on the spatial relationships of the original image, the pixels in the odd-numbered and even-numbered row regions are interleaved and recombined:

[0052] Pixels in the odd-numbered row region are returned to their original odd-numbered row positions;

[0053] Pixels in even-numbered row regions are returned to their original even-numbered row positions;

[0054] For the odd-even column recombination pattern:

[0055] Based on the spatial relationships of the original image, the pixels in the odd-numbered and even-numbered column regions are interleaved and rearranged:

[0056] Pixels in the odd-numbered column region are returned to their original odd-numbered column positions;

[0057] The pixels in the even-numbered column region are returned to their original even-numbered column positions.

[0058] Therefore, the video packet loss repair method based on image parity row / column recombination and block coding adopted in this invention has the following beneficial effects:

[0059] (1) Unlike the MDC scheme, this invention does not divide the odd and even rows / columns into frames for encoding and transmission, but instead recombines the odd and even rows / columns. In the odd and even row recombination method, all odd rows are placed in the upper half of the image and all even rows are placed in the lower half of the image, thus forming an image with the same width, height and resolution as the original image. In the odd and even column recombination method, all odd columns are placed in the left half of the image and all even columns are placed in the right half of the image, thus forming an image with the same width, height and resolution as the original image.

[0060] In both reconstruction methods, the reconstructed image is encoded by a single encoder, which can fully utilize the similarity between odd and even rows / columns for compression. Under the same encoding quality, the bandwidth usage (bit rate) of the encoded image is comparable to that of the original image directly encoded, while the MDC scheme is significantly higher than that of the original image directly encoded.

[0061] (2) Unlike the MDC scheme, the present invention only performs interpolation repair on the specific lost image blocks, rather than interpolating and enlarging the entire low-resolution image, which greatly improves the repair efficiency and image quality, and also improves the repair success rate. Even if both odd-numbered row / column data packets and even-numbered row / column data packets are lost during transmission, as long as the lost image blocks are not in the same position, the entire image can be repaired. In this case, the MDC scheme cannot perform the repair.

[0062] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall process of an embodiment of the video packet loss repair method based on image parity row / column recombination and block coding of the present invention;

[0064] Figure 2 This is a schematic diagram of image parity row / column reconstruction, representing an embodiment of the video packet loss repair method based on image parity row / column reconstruction and block coding of the present invention;

[0065] Figure 3 This is a flowchart of the single encoder encoding process of an embodiment of the video packet repair method based on image parity row / column recombination and block coding of the present invention;

[0066] Figure 4 This is a flowchart illustrating the block-level precise repair process of a video packet loss repair method based on image parity row / column recombination and block coding, according to an embodiment of the present invention.

[0067] Figure 5 This is a schematic diagram of the odd row restoration and image reconstruction process in an embodiment of the video packet loss repair method based on image odd row / column recombination and block coding of the present invention. Detailed Implementation

[0068] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0069] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0070] like Figure 1 As shown, the video packet loss repair method based on image parity row / column recombination and block coding includes the following steps:

[0071] S1, such as Figure 2 As shown, at the sending end, the original video frames are reassembled by alternating rows / columns to generate a reassembled image;

[0072] Image odd row / column reorganization includes odd row reorganization mode and odd column reorganization mode. Either of the two reorganization modes can be selected according to the content characteristics.

[0073] a) The odd-even row reorganization pattern is:

[0074] Extract all odd-numbered and even-numbered row pixels from the original image F(n);

[0075] Odd-numbered rows (rows 1, 3, 5...) are rearranged to the upper half of the image, maintaining their original order; even-numbered rows (rows 2, 4, 6...) are rearranged to the lower half of the image, maintaining their original order; a reconstructed image F'(n) is generated, with the same width and height as the original image F(n).

[0076] b) The odd / even column reorganization pattern is:

[0077] Extract all odd-numbered and even-numbered columns of pixels from the original image F(n);

[0078] The odd-numbered columns (rows 1, 3, 5...) are rearranged to the left half of the image, maintaining their original order; the even-numbered columns (rows 2, 4, 6...) are rearranged to the right half of the image, maintaining their original order; a reconstructed image F'(n) is generated, with the same width and height as the original image F(n).

[0079] Reconstructed image characteristics:

[0080] The spacing between adjacent rows / columns is increased (the vertical / horizontal spacing between odd rows / columns and even rows / columns is twice that of the original image). Spatial correlation between pixels within a region (odd or even region) is higher than the correlation between regions. Odd-even row recombination performs better on horizontally textured images, while odd-even column recombination performs better on vertically textured images.

[0081] S2, such as Figure 3 As shown, a single encoder is used to encode the reconstructed entire image, specifically as follows:

[0082] The single encoder uses a block-based coding standard algorithm (including any one of H.264, H.265, and AV1) to encode the reconstructed whole image F'(n). The coding block size can be selected as 16×16 or 32×32 pixels or other suitable sizes.

[0083] Rate control optimization: A single encoder can make full use of the high correlation within the odd and even row / column regions to compress the recombined image data. The overall bit rate is comparable to that of directly encoding the original image, which is significantly lower than that of the MDC scheme.

[0084] S3. Based on the region markers, separate two sub-streams, TS1 and TS2, from the single coded stream, corresponding to the coded block data of odd-numbered and even-numbered regions respectively. Encapsulate each separated sub-stream into data packets, specifically:

[0085] For the odd-even row recombination pattern:

[0086] TS1 corresponds to all coded block data in the odd-numbered row region.

[0087] TS2 corresponds to all coded block data in even-numbered row regions.

[0088] For the odd-even column recombination pattern:

[0089] TS1 corresponds to all coded block data in the odd-numbered column region.

[0090] TS2 corresponds to all coded block data in the even-numbered column region.

[0091] Packet construction:

[0092] Each separate sub-stream is encapsulated with an independent data packet; the data packet size is controlled within the network MTU range, and area identifiers and block location information are added to each data packet.

[0093] Design differentiated routing and transmission strategies:

[0094] 1. Employs Multipath Transmission Technology (MPRTP), allowing two streams to be transmitted through different physical links:

[0095] In a multi-access network environment, data can be transmitted through different network interfaces (such as WiFi / 4G / 5G); at the same time, in a single network environment, different network port numbers can be set for the two streams.

[0096] 2. Implement an interleaved transmission strategy: The two streams are sent at different times to reduce the probability that network congestion will affect both streams simultaneously.

[0097] S4. The receiving end receives and parses data packets. When packet loss is detected, block-level precise repair is performed. The specific process of receiving and parsing data packets at the receiving end is as follows:

[0098] S41, Data Stream Differentiation: Receive and identify TS1 and TS2 data streams from different transmission channels; extract the timestamp, sequence number, and coded block location information from each data packet.

[0099] S42. Packet Loss Statistics: Collect statistics on packet loss in TS1 and TS2 and the distribution information of the coded blocks contained therein; maintain a packet loss matrix L(x,y,t) to record the spatiotemporal packet loss distribution and realize packet loss location at the coded block level.

[0100] S43, Data Stream Reassembly: Reassemble the received TS1 and TS2 data packets in their original encoded order; prepare a complete data stream for decoding, and mark the lost data block areas.

[0101] Decoding and Block-Level Repair:

[0102] Decoding process:

[0103] The reconstructed data stream is decoded using the corresponding video decoder.

[0104] For the odd-even row recombination pattern: decode the recombined image containing the upper half (odd row region) and the lower half (even row region).

[0105] For the odd-even column recombination pattern: decode the recombined image containing the left half (odd column region) and the right half (even column region).

[0106] like Figure 4 As shown, block-level precise repair based on packet loss includes three different repair strategies:

[0107] a) Processing of regions without packet loss: For coded blocks in both odd and even regions where there is no packet loss, the pixel data of the corresponding region is directly extracted from the decoded reconstructed image.

[0108] b) Single-sided packet loss region repair: For cases where only odd-numbered or only even-numbered regions of encoded blocks are lost, the lost regions are filled using inter-row / inter-column interpolation algorithms. The specific process is as follows:

[0109] For the odd-even row recombination pattern:

[0110] When packet loss is detected in a video image, if a coded block (i,j) in an odd-numbered row region is lost, while a coded block (i,j) at the same position in an even-numbered row region remains intact:

[0111] First, extract pixel data from intact even-numbered row regions;

[0112] Then, the missing odd-numbered rows are filled using an inter-row interpolation algorithm. Specifically, the pixel value I(2i+1,j) of the missing row is equal to the average of the pixel values ​​I(2i,j) and I(2i+2,j) of the two adjacent intact even-numbered rows, i.e., I(2i+1,j) = (I(2i,j) + I(2i+2,j)) / 2. Finally, the above operations are used to generate the repaired complete coding block.

[0113] Conversely, if the even-numbered row region (i,j) is lost, while the odd-numbered row region (i,j) is intact:

[0114] Then extract pixel data from intact odd-numbered row regions;

[0115] Next, the missing even-numbered rows are filled using an inter-row interpolation algorithm. At this point, the pixel value I(2i,j) of the missing row is the average of the pixel values ​​I(2i-1,j) and I(2i+1,j) of the two adjacent complete odd-numbered rows, i.e., I(2i,j) = (I(2i-1,j) + I(2i+1,j)) / 2; thus, the repaired complete coding block is generated.

[0116] For the odd-even column recombination pattern:

[0117] If the odd-numbered column region block (i,j) is missing, and the even-numbered column region block (i,j) is intact: First, extract pixel data from the intact even-numbered column region block; then, use the inter-column interpolation algorithm to fill the missing odd-numbered column, where the pixel value I(i,2j+1) of the missing column is the average of the pixel values ​​I(i,2j) and I(i,2j+2) of the two adjacent intact even-numbered columns, i.e., I(i,2j+1)=(I(i,2j)+I(i,2j+2)) / 2; finally, after the filling is completed, the repaired complete coded block is generated.

[0118] If an even-numbered column region (i,j) is lost, while an odd-numbered column region (i,j) remains intact:

[0119] First, extract pixel data from intact odd-numbered column regions;

[0120] Then, the missing even-numbered columns are filled by inter-column interpolation algorithm. The pixel value I(i,2j) of the missing column is determined by the average of the pixel values ​​I(i,2j-1) and I(i,2j+1) of the two adjacent complete odd-numbered columns, that is, I(i,2j)=(I(i,2j-1)+I(i,2j+1)) / 2; finally, the repaired complete coding block is obtained.

[0121] c) Double-sided packet loss region repair: For cases where the coded blocks at the same location in both regions are lost, i.e., if the odd-numbered region block (i,j) is lost and the even-numbered region block (i,j) is lost, then the traditional packet loss repair scheme is used.

[0122] S5. Image Reconstruction and Post-processing: such as... Figure 5 As shown, the pixels of the repaired odd-numbered row / column regions and even-numbered row / column regions are interleaved and rearranged to restore the original image format, specifically:

[0123] 1) Parity row / column restoration:

[0124] For the odd-even row recombination pattern:

[0125] Based on the spatial relationships of the original image, the pixels in the odd-numbered and even-numbered row regions are interleaved and recombined:

[0126] The pixels in the odd-numbered row regions are returned to their original odd-numbered row (row 1, 3, 5...) positions.

[0127] The pixels in the even-numbered row regions are returned to their original even-numbered row (row 2, 4, 6...) positions.

[0128] For the odd-even column recombination pattern:

[0129] Based on the spatial relationships of the original image, the pixels in the odd-numbered and even-numbered column regions are interleaved and rearranged:

[0130] The pixels in the odd-numbered column regions are returned to their original odd-numbered column (column 1, 3, 5...) positions;

[0131] The pixels in the even-numbered column regions are returned to their original even-numbered column (2, 4, 6...) positions.

[0132] 2) Image enhancement processing:

[0133] Adaptive edge enhancement of the repaired area can be performed as needed to improve the sharpness of the reconstructed image; this is an optional step.

[0134] Example

[0135] This embodiment describes a specific implementation of the present invention in a cross-platform network video conferencing software system, which aims to provide enterprise users with high-quality, low-latency, and highly reliable remote video communication services.

[0136] (I) System Software Architecture:

[0137] 1. Client:

[0138] 1) Platform Support: Windows 10 / 11, macOS 11+, Linux (Ubuntu 20.04+)

[0139] 2) Development Language: The core logic is implemented using C++17, and the UI layer uses native frameworks (Windows: WinUI3, macOS: AppKit, Linux: Qt6).

[0140] 3) Core Modules:

[0141] Data Acquisition Module:

[0142] Capture camera video and microphone audio using native system APIs (such as MediaFoundation, AVFoundation, V4L2).

[0143] Preprocessing module: includes noise suppression, echo cancellation, and face detection (used to dynamically adjust the priority of the coding region).

[0144] Odd / even row / column reorganization module: Implemented in C++, it executes odd / even row or odd / even column reorganization logic based on configuration (or dynamic decision).

[0145] Encoding module: Video encoding is performed using the integrated and optimized OpenH264v2.3.1 library, and audio encoding is performed using Opusv1.3.1.

[0146] Block-level repair and decoding module: Integrates FFmpegv5.1 library for decoding, and executes block-level repair logic (implemented in C++) based on the received packet loss information after decoding.

[0147] Network transmission module: Implements UDP transmission based on the asio library, supports dual-channel transmission (such as one primary WiFi and one backup 4G / 5G, or two different ports); implements RTP / RTCP protocol, including custom packet loss reporting and block location information extensions.

[0148] Rendering module: Uses platform graphics APIs (Windows: Direct3D11, macOS: Metal, Linux: Vulkan / OpenGL) to efficiently render video footage.

[0149] Network status monitoring module: Real-time monitoring of packet loss rate, RTT, and available bandwidth (estimated through packet loss and latency) provides input for dynamic adjustments.

[0150] 2. Server:

[0151] 1) Architecture: Based on microservice architecture, deployed on a public cloud platform (such as Alibaba Cloud ECS).

[0152] 2) Functions: Signaling interaction (based on WebSocket), user management, conference room management, and selective forwarding units (SFU) for media stream distribution; the server does not directly participate in encoding / decoding and repair, but is only responsible for efficiently forwarding TS1 and TS2 data packets.

[0153] 3) Development language: Go 1.18 / Node.js 16.

[0154] (II) Parameter Configuration and Dynamic Adjustment Strategy:

[0155] 1. Default video parameters:

[0156] 1) Resolution: Initially set to 1280×720.

[0157] 2) Frame rate: Initially set to 25fps.

[0158] 3) Encoding format: H.264BaselineProfile, QP (QuantizationParameter) initial value 28.

[0159] 2. Odd / Even Recombination Pattern Selection:

[0160] 1) Default mode: Odd-even row reorganization.

[0161] 2) Dynamic selection strategy: Before encoding begins, the client quickly analyzes the horizontal and vertical gradient energy of the first few frames (e.g., 5 frames).

[0162] If (horizontal gradient energy > 1.5 * vertical gradient energy), then choose to recombine odd and even rows.

[0163] If the vertical gradient energy is greater than 1.5 times the horizontal gradient energy, then the odd and even columns are reorganized.

[0164] If neither of the above conditions is met, the default or the previously selected mode will be retained. This analysis will be repeated every 30 seconds or when a significant change in the scene is detected.

[0165] 3. Encoding block size: fixed at 16×16 pixels.

[0166] 4. Bandwidth adaptation logic:

[0167] 1) The network status monitoring module updates the network assessment results every second.

[0168] 2) Target bitrate adjustment: Adjust the target bitrate of OpenH264 based on the estimated available bandwidth.

[0169] 3) Resolution / Frame Rate Adjustment:

[0170] If available bandwidth is less than 500kbps, the resolution will be reduced to 640×480@15fps.

[0171] If available bandwidth is less than 300kbps, the resolution will be reduced to 320×240@15fps.

[0172] Once the bandwidth recovers and stabilizes for more than 5 seconds, gradually increase the resolution and frame rate.

[0173] 4) QP Adjustment: With the resolution / frame rate unchanged, fine-tune the QP value (±2 range) based on buffer usage and packet loss rate.

[0174] (III) Specific process of block-level repair:

[0175] 1. Reception and Buffering: The client network module receives UDP packets from TS1 and TS2, puts them into an independent jitter buffer, and sorts and deduplicates them according to the RTP sequence number.

[0176] 2. Packet loss detection: Detect missing RTP sequence numbers in each flow and record lost packets and their timestamps.

[0177] 3. Block Location Mapping: Based on the custom extended information in the data packets, the lost data packets are mapped to specific coded block locations (i,j) in the reconstructed image F'(n). A packet loss matrix L(i,j) is generated to mark the lost blocks.

[0178] 4. Data sent to decoder: The received TS1 and TS2 data blocks (whether complete or not) are sent to the FFmpeg decoder in sequence.

[0179] 5. Obtain decoded pixels: The decoder outputs the pixel data of the reconstructed image after decoding.

[0180] 6. Perform block-level repair: Traverse all encoded blocks (i,j):

[0181] 1) If the odd-numbered region block corresponding to L(i,j) is lost, while the even-numbered region block corresponding to L(i,j) is intact:

[0182] Obtain even-numbered region block data from the decoded pixels;

[0183] Call the `interpolate_odd_block(even_block_data)` function: This function uses intact even-numbered row / column block pixel data and generates repaired odd-numbered row / column block pixels by interpolating between rows or columns (e.g., for row reorganization, using the adjacent even-numbered row pixels I(2i,j) and I(2i+2,j) to perform linear interpolation to calculate I(2i+1,j) = (I(2i,j) + I(2i+2,j)) / 2).

[0184] The merged code yields the repaired, complete encoded block.

[0185] 2) If the even-numbered region block corresponding to L(i,j) is lost while the odd-numbered region block corresponding to L(i,j) is intact:

[0186] Obtain odd-numbered region block data from the decoded pixels.

[0187] Call the `interpolate_even_block(odd_block_data)` function: This function uses intact odd-numbered row / column block pixel data and generates repaired even-numbered row / column block pixels through similar inter-row or inter-column interpolation (e.g., for row reorganization, using adjacent odd-numbered row pixels I(2i-1,j) and I(2i+1,j) to perform linear interpolation to calculate I(2i,j) = (I(2i-1,j) + I(2i+1,j)) / 2).

[0188] The merged code yields the repaired, complete encoded block.

[0189] 3) If L(i,j) corresponds to double-sided loss: Call the `temporal_spatial_concealment(i,j,previous_frame_data,Neighboring_blocks_data)` function. This function is called when both odd and even region blocks are lost. It attempts to hide the error using temporal (the pixel data `previous_frame_data` at the corresponding position (i,j) in the previous frame) and spatial (the neighboring blocks `neighboring_blocks_data` in the current frame, such as the blocks above, below, left, and right). Specific strategies could include:

[0190] a) Time replacement: Directly copy the pixels of the corresponding block from the previous frame (suitable for static or slow-moving areas).

[0191] b) Spatial interpolation / copying: weighted average or direct copying based on the pixels of adjacent available blocks (suitable for areas with simple textures).

[0192] c) Motion Vector Estimation and Compensation: If available, estimate the motion of the current block using the motion vectors of neighboring blocks and compensate pixels from the corresponding positions in the previous frame (applicable to moving regions). This function aims to generate a visually acceptable alternative block to reduce distortion.

[0193] 4) If there is no packet loss, the decoded pixel data can be used directly.

[0194] 7. Image restoration: Rearrange all processed blocks according to the original odd / even row / column order to obtain the final display frame F(n).

[0195] 8. Rendering and Display: The restored frame is sent to the rendering module for display.

[0196] (iv) Implementation details:

[0197] 1. Dual-channel transmission: Utilize the operating system's multi-network interface binding capability, or simply use two different UDP ports to send TS1 and TS2 to the SFU; the SFU distinguishes the stream based on the receiving port and forwards it to other participants. The receiving end also listens on both ports.

[0198] 2. Packet loss information synchronization: Through the custom extended fields in the RTCP report, the receiver periodically (e.g., once per second) feeds back the packet loss statistics of TS1 and TS2 to the sender, and the sender can adjust the encoding strategy accordingly (e.g., increase the frequency of key frames).

[0199] (V) Performance Data (Simulated Network Test):

[0200] Test environment: Two PCs (Intel Core i7, 16GB RAM, Windows 10), with different packet loss rates and latency introduced through a network simulator (Clumsy).

[0201] 1. CPU utilization (1280x720@25fps encoding + decoding):

[0202] 1) 0% packet loss: approximately 12% (an increase of approximately 1% compared to directly encoding the original image), compared to the MDC scheme (dual encoder): CPU usage is reduced by approximately 10%.

[0203] 2) 20% random packet loss: approximately 13% (block-level repair computation overhead), compared to the MDC scheme (dual encoder, whole frame interpolation): CPU usage is reduced by approximately 24%.

[0204] 2. Bandwidth consumption (1280x720@25fps):

[0205] When the target bitrate is 1.5Mbps, the actual bandwidth usage is approximately 1.55Mbps (including RTP / RTCP overhead), which is about 22% lower than the MDC solution of the same quality (approximately 2.0Mbps) and about 25-30% lower than FEC (which requires additional redundant packets).

[0206] 3. Subjective Quality (MOS) (30% random packet loss):

[0207] 1) This invention: 4.1 ± 0.2;

[0208] 2) Traditional interpolation method: 2.4 ± 0.3;

[0209] 3) MDC (Single Description Loss Repair): 3.0 ± 0.2;

[0210] 4. Repair latency: The additional computational latency introduced by the block-level repair process is <2ms. End-to-end latency is mainly determined by network transmission and encoding / decoding (typically in the range of 100-200ms).

[0211] This embodiment verifies that the present invention can significantly improve the robustness and user experience of video communication in unstable network environments in typical software video conferencing scenarios without significantly increasing system complexity and resource consumption.

[0212] Other application scenarios of this invention

[0213] 1. Remote video dialogue with robots

[0214] In scenarios where AI robots interact remotely with humans, the quality of video communication is crucial to the user experience. This invention can be applied to:

[0215] 1) Service robots interact with users remotely via video;

[0216] 2) Remote video connection between smart home robots and family members;

[0217] 3) Real-time video communication for service robots in public places;

[0218] 4) Video communication between assistive robots and people with mobility impairments.

[0219] Features of this invention in this scenario:

[0220] 1) Ensure clear transmission of facial expressions and body movements to enhance the realism of the interaction;

[0221] 2) Reduce the impact of network fluctuations on robot response speed;

[0222] 3) Maintain communication quality in unstable network environments such as public WiFi;

[0223] 4) Reduce the negative impact of video stuttering on the human-computer interaction experience.

[0224] 2. Security monitoring

[0225] In remote video surveillance systems, cameras are typically deployed in areas with unstable network conditions. This invention's technology ensures that even in poor network environments, the monitoring center can still obtain high-quality surveillance footage, making it particularly suitable for:

[0226] 1) Urban security monitoring system;

[0227] 2) Highway traffic monitoring;

[0228] 3) Monitoring of facilities in remote areas.

[0229] 3. Remote video conferencing

[0230] In scenarios involving remote work and cross-regional meetings, this invention can significantly improve meeting quality and efficiency:

[0231] 1) Routine remote meetings and collaborations of multinational corporations;

[0232] 2) Large-scale video conferencing system (supporting dozens to hundreds of participants);

[0233] 3) Team collaboration in a hybrid office environment;

[0234] 4) Remote emergency decision-making meetings of government agencies.

[0235] Advantages of this invention for remote conferencing:

[0236] 1) Reduce participant disconnections and screen lag caused by network fluctuations;

[0237] 2) Maintain a high-quality video stream to ensure accurate transmission of facial expressions and micro-expressions;

[0238] 3) Reduce bandwidth requirements so that users of mobile devices and those in weak network environments can also have a good experience;

[0239] 4. Industrial remote control

[0240] In the field of industrial automation remote monitoring and control, this invention can be applied to:

[0241] 1) Remote control of mining equipment;

[0242] 2) Remote control of robots in hazardous environments;

[0243] 3) Monitoring of intelligent factory production lines.

[0244] 5. Online Education

[0245] In educational scenarios such as online classrooms and remote training, this technology can:

[0246] 1) Ensure clear and smooth transmission of teaching videos to reduce the impact of network instability on teaching;

[0247] 2) Adapt to the complex and ever-changing network environment on the student's end;

[0248] 3) Reduce bandwidth requirements and improve universality.

[0249] Therefore, the present invention adopts the above-mentioned video packet loss repair method based on image odd-even row / column recombination and block coding. Based on image odd-even row / column recombination and block coding, it utilizes the high similarity of odd-even rows and columns, and achieves efficient packet loss repair without increasing latency and bandwidth through innovative recombination and block-level repair mechanisms. This improves the robustness of video communication and user experience, solves the defects of existing technologies, and is applicable to various real-time video communication scenarios.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video packet loss repair method based on image odd / even row / column recombination and block coding, characterized in that, Includes the following steps: S1. At the sending end, the original video frames are reassembled by combining the odd and even rows / columns of the image to generate a reassembled image; S2. Encode the reconstructed image using a single encoder; S3. Based on the region marker, separate two sub-streams, TS1 and TS2, from the single coded stream, corresponding to the coded block data of odd-numbered regions and even-numbered regions respectively, and encapsulate each separated sub-stream into data packets; S4. Receive and parse data packets at the receiving end, and perform block-level precise repair when packet loss is detected; S5. Interleave and recombine the pixels of the repaired odd-numbered row / column regions and even-numbered row / column regions to restore the original image format; In S1, image odd / even row / column rearrangement includes odd / even row rearrangement mode and odd / even column rearrangement mode. The odd / even row rearrangement mode is as follows: Extract all odd-numbered and even-numbered row pixels from the original image F(n); The odd-numbered rows are rearranged to the upper half of the image, maintaining their original order; Even-numbered rows are rearranged to the lower half of the image, maintaining their original order; Generate a reconstructed image F'(n) with the same width and height as the original image F(n); The odd-even column reorganization pattern is as follows: Extract all odd-numbered and even-numbered columns of pixels from the original image F(n); The odd-numbered columns are rearranged to the left half of the image, maintaining their original order; The even-numbered columns are rearranged to the right half of the image, maintaining their original order; Generate a reconstructed image F'(n) with the same width and height as the original image F(n); The specific selection of the odd / even recombination mode is as follows: The default mode is parity row reorganization; Before encoding begins, the client analyzes the horizontal and vertical gradient energies of the previous few frames: If the horizontal gradient energy > 1.5 If the vertical gradient energy is used, then odd-even row recombination is selected; If the vertical gradient energy > 1.5 For horizontal gradient energy, odd and even columns are recombined. S3 specifically refers to: For the odd-even row recombination pattern: TS1 corresponds to all coded block data in the odd-numbered row region; TS2 corresponds to all coded block data in even-numbered row regions; For the odd-even column recombination pattern: TS1 corresponds to all coded block data in the odd-numbered column region; TS2 corresponds to all coded block data in the even-numbered column region; In S4, the specific process of receiving and parsing data packets at the receiving end is as follows: S41. Receive and identify TS1 and TS2 data streams from different transmission channels, and extract the timestamp, sequence number and coded block location information from each data packet; S42. Calculate the packet loss situation and the distribution information of the coding blocks contained in TS1 and TS2 respectively, maintain the packet loss matrix L(x,y,t) to record the spatiotemporal packet loss distribution, and realize packet loss location at the coding block level. S43. Reassemble the received TS1 and TS2 data packets according to their original encoding order to obtain a complete data stream for decoding, and mark the lost data block areas; In S4, block-level precise repair based on packet loss includes: Processing of regions without packet loss: For coded blocks in both odd and even regions where there is no packet loss, the pixel data of the corresponding region is directly extracted from the decoded reconstructed image; One-sided packet loss region repair: For cases where only odd-numbered regions or only even-numbered regions of encoded blocks are lost, the lost regions are filled using inter-row / inter-column interpolation algorithms; Double-sided packet loss region repair: When the encoded block at the same location in both regions is lost, the traditional packet loss repair scheme is used.

2. The video packet loss repair method based on image parity row / column recombination and block coding according to claim 1, characterized in that, S2 specifically refers to: The single encoder uses a standard algorithm based on block coding to encode the reconstructed whole image F'(n).

3. The video packet loss repair method based on image parity row / column recombination and block coding according to claim 2, characterized in that, The single encoder employs a standard block-based coding algorithm, including any one of H.264, H.265, and AV1.

4. The video packet loss repair method based on image parity row / column recombination and block coding according to claim 3, characterized in that, S5 specifically refers to: For the odd-even row recombination pattern: Based on the spatial relationships of the original image, the pixels in the odd-numbered and even-numbered row regions are interleaved and recombined: Pixels in the odd-numbered row region are returned to their original odd-numbered row positions; Pixels in even-numbered row regions are returned to their original even-numbered row positions; For the odd-even column recombination pattern: Based on the spatial relationships of the original image, the pixels in the odd-numbered and even-numbered column regions are interleaved and rearranged: Pixels in the odd-numbered column region are returned to their original odd-numbered column positions; The pixels in the even-numbered column region are returned to their original even-numbered column positions.

Images