Video packet loss repair method based on image odd-even row / column recombination and block coding

Abstract

The invention belongs to the technical field of video packet loss repair, and particularly discloses a video packet loss repair method based on image odd-even row / column recombination and block coding, which comprises the following steps: at a sending end, carrying out image odd-even row / column recombination on an original video frame to generate a recombined image; a single encoder is used to encode the recombined whole image; separating two substreams from the single coding stream according to the region mark, respectively corresponding to the coding block data of the odd number region and the even number region, and carrying out data packet packaging on each separated substream; receiving and analyzing the data packet at a receiving end, and performing block-level accurate repair when packet loss is detected; and the pixels of the repaired odd-numbered line / column area and even-numbered line / column area are staggered and recombined, and the original image format is restored. According to the video packet loss repair method based on image odd-even row / column recombination and block coding, delay and bandwidth are not increased, the video transmission packet loss repair effect is greatly improved, and the video packet loss repair method is particularly suitable for a real-time video communication scene.

Description

Technical Field

[0001] The present invention relates to the technical field of video packet loss repair, and in particular to a video packet loss repair method based on image parity row / column reorganization and block coding. Background Art

[0002] With the rapid development of online video applications, the transmission quality of video data over unreliable networks has become a key factor limiting user experience. Especially in real-time video transmission scenarios, network packet loss can lead to a significant degradation in image quality, impacting the user experience. Therefore, packet loss recovery methods are crucial to the user experience of online video applications.

[0003] Existing video packet loss repair technologies have the following main problems:

[0004] 1. Traditional forward error correction (FEC) technology: It adds redundant data to recover lost data packets, but increases bandwidth overhead by about 30% and is inefficient when bandwidth is limited.

[0005] 2. Automatic Repeat Request (ARQ) mechanism: Requests retransmission after detecting packet loss, introducing a 200-500ms delay and is not suitable for real-time video transmission scenarios.

[0006] 3. Multiple Description Coding (MDC): This method downsamples the original image into multiple low-resolution images, encodes each image separately, and transmits them through multiple channels. As long as one channel can receive the image completely, the entire image can be reconstructed. Although it has a certain degree of robustness against packet loss, it has the following drawbacks:

[0007] 1) When packet loss occurs, the image quality of the reconstructed image is severely degraded due to interpolation or direct zooming based on a low-resolution coded image.

[0008] 2) If all coded descriptions cannot be received completely, repair cannot be performed;

[0009] 3) The cost of interpolation operation for the entire image is high;

[0010] 4) Multiple channels of images are encoded independently, and the total bandwidth occupied after encoding is higher than that without segmentation.

[0011] 4. Traditional error concealment techniques: These include interpolation, weighted averaging of forward and backward frames, motion prediction, and other intra-frame and inter-frame error concealment techniques. Because adjacent regions of complex images are not very similar, and the similarity between previous and next frames of fast-moving images is also not very high, noticeable mosaics or image distortion will occur when the packet loss rate exceeds 10%, and video quality cannot be guaranteed.

[0012] Therefore, a video packet loss repair solution with low latency, low bandwidth overhead, high anti-packet loss capability and high image quality is needed. This solution should be able to achieve accurate packet loss repair at the coding block level. Summary of the Invention

[0013] The purpose of the present invention is to provide a video packet loss repair method based on image odd-even row / column reorganization and block coding, which fully utilizes the high similarity characteristics of odd rows / columns and even rows / columns in video images, and combines the characteristics of block coding technology. By rearranging and combining odd-even rows / columns and an accurate block-level packet loss repair mechanism, efficient packet loss repair is achieved without increasing delay and bandwidth.

[0014] To achieve the above object, the present invention provides a video packet loss repair method based on image parity row / column reorganization and block coding, comprising the following steps:

[0015] S1. At the sending end, the original video frame is reorganized into odd and even rows / columns to generate a recombined image;

[0016] S2, using a single encoder to encode the entire reconstructed image;

[0017] S3, separating two sub-streams, TS1 and TS2, from the single coded stream according to the region tag, corresponding to the coded block data of the odd region and the even region respectively, and encapsulating each separated sub-stream into data packets;

[0018] S4. Receive and parse data packets at the receiving end, and perform block-level accurate repair when packet loss is detected;

[0019] S5. Interleave and reorganize the pixels in the repaired odd-numbered row / column region and the even-numbered row / column region to restore the original image format.

[0020] Preferably, in S1, the image odd-even row / column reorganization includes an odd-even row reorganization mode and an odd-even column reorganization mode;

[0021] The parity row reorganization mode is:

[0022] Extract all odd-numbered and even-numbered pixels from the original image F(n) respectively;

[0023] The odd-numbered rows are rearranged to the upper half of the image, maintaining their original order;

[0024] Even-numbered rows are rearranged to the lower half of the image, maintaining their original order;

[0025] Generate a reconstructed image F'(n), which has the same width and height as the original image F(n).

[0026] Preferably, the parity column reorganization mode is:

[0027] Extract all odd-numbered and even-numbered pixels from the original image F(n) respectively;

[0028] The odd-numbered columns are rearranged to the left half of the image, maintaining the original order;

[0029] Even-numbered columns are rearranged to the right half of the image, maintaining their original order;

[0030] Generate a reconstructed image F'(n), which has the same width and height as the original image F(n).

[0031] Preferably, S2 is specifically:

[0032] The single encoder uses a standard algorithm based on block coding to encode the reconstructed entire image F'(n).

[0033] Preferably, the single encoder adopts a standard algorithm based on block coding including any one of H.264, H.265, and AV1.

[0034] Preferably, S3 is specifically:

[0035] For even-odd row reorganization mode:

[0036] TS1 corresponds to all coded block data in the odd-numbered row area;

[0037] TS2 corresponds to all coded block data in the even-numbered row area;

[0038] For parity column reorganization mode:

[0039] TS1 corresponds to all coded block data in the odd column area;

[0040] TS2 corresponds to all coded block data in the even column area.

[0041] Preferably, in S4, the specific process of receiving and parsing the data packet at the receiving end is:

[0042] S41, receiving and identifying TS1 and TS2 data streams from different transmission channels, and extracting the timestamp, sequence number, and coding block location information from each data packet;

[0043] S42: Count the packet loss in TS1 and TS2 and the distribution information of the coding blocks contained therein, maintain the packet loss matrix L(x, y, t) to record the spatiotemporal packet loss distribution, and implement packet loss location at the coding block level;

[0044] S43: Reassemble the received TS1 and TS2 data packets according to the original coding order to obtain a complete data stream for decoding, and mark the lost data block area.

[0045] Preferably, in S4, performing block-level precise repair according to packet loss conditions includes:

[0046] Processing of areas without packet loss: For coding blocks with no packet loss in both odd and even areas, the pixel data of the corresponding areas are directly extracted from the decoded reconstructed image.

[0047] Unilateral packet loss area repair: When only odd-numbered or even-numbered areas have lost code blocks, the lost areas are filled using inter-row / inter-column interpolation algorithms.

[0048] Double-sided packet loss area repair: When the coding blocks at the same position in two areas are lost, the traditional packet loss repair solution is used.

[0049] Preferably, S5 is specifically:

[0050] For even-odd row reorganization mode:

[0051] According to the spatial position relationship of the original image, the pixels in the odd-numbered row area and the even-numbered row area are reorganized in an interlaced manner:

[0052] The pixels in the odd-numbered row area are placed back to their original odd-numbered row positions;

[0053] The pixels in the even-numbered row area are placed back to their original even-numbered row positions;

[0054] For parity column reorganization mode:

[0055] According to the spatial position relationship of the original image, the pixels in the odd-numbered column area and the even-numbered column area are reorganized in an interleaved manner:

[0056] The pixels in the odd column area are placed back to the original odd column position;

[0057] The pixels in the even column area are placed back to their original even column positions.

[0058] Therefore, the present invention adopts the above-mentioned video packet loss repair method based on image parity row / column reorganization and block coding, and the beneficial effects are as follows:

[0059] (1) Unlike the MDC scheme, the present invention does not split the odd and even rows / columns and then perform frame encoding transmission, but recombines the odd and even rows / columns; wherein, the odd-even row recombination method repositions all odd rows in the upper half of the image and all even rows in the lower half of the image, thereby forming an image with the same width, height and resolution as the original image; and the odd-even column recombination method repositions all odd columns in the left half of the image and all even columns in the right half of the image, also forming an image with the same width, height and resolution as the original image.

[0060] In both reconstruction methods, the reconstructed image is encoded by one encoder, which can fully utilize the similarity between odd rows / columns and even rows / columns for compression. Under the same encoding quality, the bandwidth occupancy (bit rate) after encoding is equivalent to that of directly encoding the original image, while the MDC scheme is significantly higher than direct encoding of the original image.

[0061] (2) Unlike the MDC scheme, the present invention only performs interpolation repair on specific lost image blocks, rather than interpolating and amplifying the entire low-resolution image. This greatly improves the repair efficiency and image quality, and also increases the repair success rate. Even if both odd-numbered row / column data packets and even-numbered row / column data packets are lost during transmission, the entire image can be repaired as long as the positions of the lost image blocks are different. In this case, the MDC scheme cannot perform repair.

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a schematic diagram of the overall flow of an embodiment of a video packet loss repair method based on image parity row / column reorganization and block coding according to the present invention;

[0064] Figure 2 2. It is a schematic diagram of image parity row / column reorganization according to an embodiment of a video packet loss repair method based on image parity row / column reorganization and block coding of the present invention;

[0065] Figure 3 This is a single encoder encoding flow chart of an embodiment of a video packet loss repair method based on image parity row / column reorganization and block encoding of the present invention;

[0066] Figure 4 This is a flow chart of block-level precise repair of a video packet loss repair method embodiment based on image parity row / column reorganization and block coding of the present invention;

[0067] Figure 5 1 is a schematic diagram of the parity row restoration and image reconstruction process of an embodiment of a video packet loss repair method based on image parity row / column reorganization and block coding of the present invention. DETAILED DESCRIPTION

[0068] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0069] Unless otherwise defined, technical or scientific terms used in the present invention shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0070] like Figure 1 As shown, the video packet loss repair method based on image parity row / column reorganization and block coding includes the following steps:

[0071] S1, such as Figure 2 As shown, at the sending end, the original video frame is reorganized into odd and even rows / columns to generate a recombined image;

[0072] Image odd-even row / column reassembly includes odd-even row reassembly mode and odd-even column reassembly mode. Either of the two reassembly modes can be selected according to content characteristics.

[0073] a) The parity row reorganization mode is:

[0074] Extract all odd-numbered and even-numbered pixels from the original image F(n) respectively;

[0075] The odd rows (rows 1, 3, 5, etc.) are rearranged to the upper half of the image, maintaining the original order; the even rows (rows 2, 4, 6, etc.) are rearranged to the lower half of the image, maintaining the original order; and a reconstructed image F'(n) is generated, which has the same width and height as the original image F(n).

[0076] b) The parity column reorganization mode is:

[0077] Extract all odd-numbered and even-numbered pixels from the original image F(n) respectively;

[0078] The odd columns (rows 1, 3, 5, etc.) are rearranged to the left half of the image, maintaining the original order; the even columns (rows 2, 4, 6, etc.) are rearranged to the right half of the image, maintaining the original order; and a reconstructed image F'(n) is generated, which has the same width and height as the original image F(n).

[0079] Characteristics of reconstructed images:

[0080] The spacing between adjacent rows and columns is increased (the vertical and horizontal spacing between odd and even rows and columns is twice that of the original image). The spatial correlation between pixels within a region (odd or even) is higher than the correlation between regions. Odd-even row reassembly performs better on horizontally textured images, while odd-even column reassembly performs better on vertically textured images.

[0081] S2, such as Figure 3 As shown, a single encoder is used to encode the entire reorganized image, specifically:

[0082] The single encoder uses a standard algorithm based on block coding (including any one of H.264, H.265, and AV1) to encode the reconstructed entire image F'(n). The coding block size can be selected as 16×16 or 32×32 pixels or other appropriate sizes.

[0083] Rate control optimization: A single encoder can fully utilize the high correlation within the odd and even row / column regions to compress the reorganized image data. The overall bit rate is comparable to that of directly encoding the original image and significantly lower than that of the MDC scheme.

[0084] S3. Separate two sub-streams, TS1 and TS2, from the single coded stream according to the region tag, corresponding to the coded block data of the odd and even regions respectively, and encapsulate each separated sub-stream into data packets, specifically:

[0085] For even-odd row reorganization mode:

[0086] TS1 corresponds to all coded block data in the odd-numbered row area.

[0087] TS2 corresponds to all coded block data in the even-numbered row area.

[0088] For parity column reorganization mode:

[0089] TS1 corresponds to all coded block data in the odd column area.

[0090] TS2 corresponds to all coded block data in the even column area.

[0091] Data Pack Construction:

[0092] Each separated sub-stream is encapsulated in an independent data packet; the data packet size is controlled within the network MTU range, and the area identifier and block location information are added to each data packet.

[0093] Design differentiated routing and transmission strategies:

[0094] 1. Use multi-path transmission technology (MPRTP) to allow two streams to be transmitted through different physical links:

[0095] In a multi-access network environment, transmission can be carried out through different network interfaces (such as WiFi / 4G / 5G); at the same time, in a single network, different network port numbers can be set for the two streams.

[0096] 2. Implement an interleaved transmission strategy: The two streams are sent at different times to reduce the probability that network congestion will affect both streams at the same time.

[0097] S4. Receive and parse data packets at the receiving end. When packet loss is detected, perform block-level precise repair. The specific process of receiving and parsing data packets at the receiving end is as follows:

[0098] S41. Data stream differentiation: Receive and identify TS1 and TS2 data streams from different transmission channels; extract the timestamp, sequence number, and coding block location information in each data packet.

[0099] S42. Packet loss statistics: Count the packet loss in TS1 and TS2 and the distribution information of the coding blocks contained therein respectively; maintain the packet loss matrix L(x, y, t) to record the spatiotemporal packet loss distribution and realize packet loss location at the coding block level.

[0100] S43, data stream reassembly: reassemble the received TS1 and TS2 data packets according to the original encoding order; prepare a complete data stream for decoding, and mark the lost data block area.

[0101] Decoding and block-level repair:

[0102] Decoding process:

[0103] Use the corresponding video decoder to decode the reassembled data stream.

[0104] For the odd-even row reassembly mode: a reassembled image including the upper half (odd row area) and the lower half (even row area) is decoded.

[0105] For the odd-even column reassembly mode: decode the reassembled image containing the left half (odd column area) and the right half (even column area).

[0106] like Figure 4 As shown in Figure 2, block-level precise repair based on packet loss conditions includes three different repair strategies:

[0107] a) Processing of areas without packet loss: For coding blocks with no packet loss in both odd and even areas, the pixel data of the corresponding areas are directly extracted from the decoded reconstructed image.

[0108] b) Repairing Unilateral Packet Loss Areas: For cases where only odd-numbered or even-numbered areas have lost coded blocks, inter-row / inter-column interpolation algorithms are used to fill in the lost areas. The specific process is as follows:

[0109] For even-odd row reorganization mode:

[0110] When packet loss of coded blocks in a video image is detected, if the coded block (i, j) in the odd-numbered row area is lost, while the coded block (i, j) at the same position in the even-numbered row area is intact:

[0111] First, pixel data is extracted from intact even-row area blocks;

[0112] Then, the lost odd rows are filled in using an inter-row interpolation algorithm. The specific calculation method is that the pixel value of the lost row I(2i+1,j) is equal to the average of the pixel values of the two adjacent intact even rows I(2i,j) and I(2i+2,j), that is, I(2i+1,j) = (I(2i,j) + I(2i+2,j)) / 2; finally, the above operation generates a repaired complete coding block.

[0113] On the contrary, if the even-numbered row block (i, j) is lost, but the odd-numbered row block (i, j) is intact:

[0114] Then extract pixel data from the intact odd-numbered row area blocks;

[0115] Next, the inter-row interpolation algorithm is used to fill in the lost even rows. At this time, the pixel value of the lost row I(2i,j) is the average of the pixel values of the two adjacent perfect odd rows I(2i-1,j) and I(2i+1,j), that is, I(2i,j) = (I(2i-1,j) + I(2i+1,j)) / 2; thus, a complete coding block is generated after repair.

[0116] For parity column reorganization mode:

[0117] If the odd column region block (i, j) is lost and the even column region block (i, j) is intact: First, extract pixel data from the intact even column region block; then, use the inter-column interpolation algorithm to fill in the lost odd column, where the lost column pixel value I(i, 2j+1) is the average of the two adjacent intact even column pixel values I(i, 2j) and I(i, 2j+2), that is, I(i, 2j+1) = (I(i, 2j) + I(i, 2j+2)) / 2; finally, after completing the filling, generate the repaired complete coding block.

[0118] If the even-numbered column block (i, j) is lost, while the odd-numbered column block (i, j) is intact:

[0119] First, extract pixel data from intact odd-numbered column blocks;

[0120] The missing even columns are then filled using the inter-column interpolation algorithm. The missing column pixel value I(i,2j) is determined by the average of the pixel values of the two adjacent perfect odd columns I(i,2j-1) and I(i,2j+1), that is, I(i,2j) = (I(i,2j-1) + I(i,2j+1)) / 2. Finally, the complete coding block is obtained after repair.

[0121] c) Double-sided packet loss repair: For the case where the coding blocks at the same position in two regions are lost, that is, if the odd-numbered region block (i, j) is lost and the even-numbered region block (i, j) is lost, the traditional packet loss repair scheme is used.

[0122] S5. Image reconstruction and post-processing: Figure 5 As shown, the pixels in the repaired odd-numbered row / column area and the even-numbered row / column area are interleaved and reassembled to restore the original image format, specifically:

[0123] 1) Odd-even row / column restoration:

[0124] For even-odd row reorganization mode:

[0125] According to the spatial position relationship of the original image, the pixels in the odd-numbered row area and the even-numbered row area are reorganized in an interlaced manner:

[0126] The pixels in the odd-numbered row area are put back to the original odd-numbered row (1, 3, 5...) positions.

[0127] The pixels in the even-numbered row area are put back to the original even-numbered row (2, 4, 6...) positions.

[0128] For parity column reorganization mode:

[0129] According to the spatial position relationship of the original image, the pixels in the odd-numbered column area and the even-numbered column area are reorganized in an interleaved manner:

[0130] The pixels in the odd column area are put back to the original odd column (1, 3, 5... columns) position;

[0131] The pixels in the even column area are placed back to the original even column (2, 4, 6... columns) positions.

[0132] 2) Image enhancement processing:

[0133] It is an optional step to perform edge adaptive enhancement on the repaired area according to actual needs to improve the sharpness of the reconstructed image.

[0134] Example

[0135] This embodiment describes a specific implementation method of applying the present invention to a cross-platform network video conferencing software system, which is designed to provide high-quality, low-latency, and high-reliability remote video communication services to enterprise users.

[0136] (1) System software architecture:

[0137] 1. Client:

[0138] 1) Platform support: Windows 10 / 11, macOS 11+, Linux (Ubuntu 20.04+)

[0139] 2) Development Language: The core logic is implemented in C++17, and the UI layer uses native frameworks (Windows: WinUI3, macOS: AppKit, Linux: Qt6)

[0140] 3) Core modules:

[0141] Acquisition module:

[0142] Use system native APIs (such as MediaFoundation, AVFoundation, V4L2) to capture camera video and microphone audio.

[0143] Preprocessing module: including noise suppression, echo cancellation, and face detection (used to dynamically adjust the priority of the coding area).

[0144] Parity row / column reorganization module: implemented in C++, performs parity row or parity column reorganization logic based on configuration (or dynamic decision).

[0145] Encoding module: Integrates the optimized OpenH264v2.3.1 library for video encoding and uses Opusv1.3.1 for audio encoding.

[0146] Block-level repair and decoding module: Integrates the FFmpegv5.1 library for decoding, and executes block-level repair logic based on the received packet loss information after decoding (implemented in C++).

[0147] Network transmission module: implements UDP transmission based on the asio library, supports dual-channel transmission (such as one primary WiFi, one backup 4G / 5G, or two different ports); implements RTP / RTCP protocol, including customized packet loss reporting and block location information extension.

[0148] Rendering module: Use platform graphics API (Windows: Direct3D11, macOS: Metal, Linux: Vulkan / OpenGL) to efficiently render video images.

[0149] Network status monitoring module: real-time monitoring of packet loss rate, RTT, and available bandwidth (through packet loss and delay estimation) to provide input for dynamic adjustment.

[0150] 2. Server:

[0151] 1) Architecture: Based on a microservices architecture, deployed on a public cloud platform (such as Alibaba Cloud ECS).

[0152] 2) Functions: Signaling interaction (based on WebSocket), user management, conference room management, and media stream distribution using the Selective Forwarding Unit (SFU). The server does not directly participate in encoding, decoding, or repair, but is only responsible for efficiently forwarding TS1 and TS2 packets.

[0153] 3) Development language: Go1.18 / Node.js16.

[0154] (2) Parameter configuration and dynamic adjustment strategy:

[0155] 1. Default video parameters:

[0156] 1) Resolution: The initial setting is 1280×720.

[0157] 2) Frame rate: Initially set to 25fps.

[0158] 3) Coding format: H.264 Baseline Profile, QP (Quantization Parameter) initial value 28.

[0159] 2. Parity reassembly mode selection:

[0160] 1) Default mode: odd-even row reorganization.

[0161] 2) Dynamic selection strategy: Before encoding begins, the client quickly analyzes the horizontal and vertical gradient energy of the first few frames (e.g., 5 frames).

[0162] If (horizontal gradient energy>1.5*vertical gradient energy), then select odd-even row recombination.

[0163] If the vertical gradient energy is greater than 1.5*horizontal gradient energy), then odd-even column recombination is selected.

[0164] If neither of the above conditions is met, the default or last selected mode is maintained. This analysis is repeated every 30 seconds or when a significant scene change is detected.

[0165] 3. Coding block size: fixed at 16×16 pixels.

[0166] 4. Bandwidth adaptation logic:

[0167] 1) The network status monitoring module updates the network assessment results every second.

[0168] 2) Target bitrate adjustment: Adjust the target bitrate of OpenH264 based on the estimated available bandwidth.

[0169] 3) Resolution / frame rate adjustment:

[0170] If the available bandwidth is less than 500kbps, the resolution is reduced to 640×480@15fps.

[0171] If the available bandwidth is less than 300kbps, the resolution is reduced to 320×240@15fps.

[0172] When the bandwidth is restored and stabilized for more than 5 seconds, gradually increase the resolution and frame rate.

[0173] 4) QP adjustment: When the resolution / frame rate remains unchanged, fine-tune the QP value (±2 range) based on the buffer occupancy and packet loss rate.

[0174] (3) Specific process of block-level repair:

[0175] 1. Receiving and buffering: The client network module receives the UDP data packets of TS1 and TS2, puts them into an independent jitter buffer, and sorts and removes duplicates according to the RTP sequence number.

[0176] 2. Packet loss detection: Detect missing RTP sequence numbers in each stream and record lost packets and their timestamps.

[0177] 3. Block Position Mapping: Based on the custom extended information in the data packet, the lost data packet is mapped to the specific coding block position (i, j) in the reconstructed image F'(n). The packet loss matrix L(i, j) is generated and the lost blocks are marked.

[0178] 4. Send data to decoder: Send the received TS1 and TS2 data blocks (whether complete or not) to the FFmpeg decoder in sequence.

[0179] 5. Get decoded pixels: The decoder outputs the decoded reconstructed image pixel data.

[0180] 6. Perform block-level repair: traverse all encoding blocks (i, j):

[0181] 1) If the odd-numbered block corresponding to L(i,j) is lost, while the even-numbered block corresponding to L(i,j) is intact:

[0182] Get even area block data from decoded pixels;

[0183] Call the interpolate_odd_block(even_block_data) function: This function uses the intact even row / column block pixel data to generate the repaired odd row / column block pixels through inter-row or inter-column interpolation (for example, for row reorganization, use the upper and lower adjacent even row pixels I(2i,j) and I(2i+2,j) for linear interpolation calculation I(2i+1,j)=(I(2i,j)+I(2i+2,j)) / 2).

[0184] Merge to get the repaired complete coding block.

[0185] 2) If the block corresponding to the even-numbered region of L(i,j) is lost and the block corresponding to the odd-numbered region of L(i,j) is intact:

[0186] Get odd area block data from decoded pixels.

[0187] Call the interpolate_even_block(odd_block_data) function: This function uses the intact odd row / column block pixel data to generate the repaired even row / column block pixels through similar inter-row or inter-column interpolation (for example, for row reorganization, use the upper and lower adjacent odd row pixels I(2i-1,j) and I(2i+1,j) for linear interpolation calculation I(2i,j)=(I(2i-1,j)+I(2i+1,j)) / 2).

[0188] Merge to get the repaired complete coding block.

[0189] 3) If L(i,j) is lost on both sides: call temporal_spatial_concealment(i,j,previous_frame_data,neighboring_blocks_data) function: This function is called when both the odd and even blocks are lost. It attempts to use temporal (previous_frame_data of the pixel data at position (i,j) in the previous frame) and spatial (neighboring_blocks_data of the neighboring blocks in the current frame, such as the upper, lower, left, and right blocks) information for error concealment. Specific strategies can be:

[0190] a) Temporal replacement: directly copy the pixels of the corresponding block in the previous frame (suitable for static or slow-motion areas).

[0191] b) Spatial interpolation / copying: weighted averaging or direct copying of pixels from adjacent available blocks (suitable for areas with simple textures).

[0192] c) Motion Vector Estimation and Compensation: If available, the motion of the current block is estimated using the motion vectors of neighboring blocks and pixels are compensated from the corresponding positions in the previous frame (applicable to moving areas). This function aims to generate a visually acceptable replacement block to reduce distortion.

[0193] 4) If there is no packet loss, the decoded pixel data is used directly.

[0194] 7. Image restoration: Rearrange all processed blocks in the original odd-even row / column order to obtain the final display frame F(n).

[0195] 8. Rendering and display: Send the restored frame to the rendering module for display.

[0196] (IV) Additional implementation details:

[0197] 1. Dual-channel transmission: Utilize the operating system's multi-network interface binding capability, or simply use two different UDP ports to send TS1 and TS2 to the SFU; the SFU distinguishes the streams based on the receiving port and forwards them to other participants. The receiving end also listens on two ports.

[0198] 2. Packet loss information synchronization: Through the custom extension field in the RTCP report, the receiver regularly (e.g., once per second) feeds back the packet loss statistics of TS1 and TS2 to the sender. The sender can adjust the encoding strategy accordingly (e.g., increase the key frame frequency).

[0199] (V) Performance data (simulated network test):

[0200] Test environment: Two PCs (Intel Core i7, 16GB RAM, Windows 10), with different packet loss rates and delays introduced through a network simulator (Clumsy).

[0201] 1.CPU usage (1280x720@25fps encoding + decoding):

[0202] 1) 0% packet loss: about 12% (about 1% increase compared to directly encoding the original image), compared to the MDC solution (dual encoder): CPU usage is reduced by about 10%.

[0203] 2) 20% random packet loss: about 13% (block-level repair computation overhead), compared to the MDC solution (dual encoder, full-frame interpolation): CPU usage is reduced by about 24%.

[0204] 2. Bandwidth consumption (1280x720@25fps):

[0205] When the target bit rate is 1.5Mbps, the actual usage is about 1.55Mbps (including RTP / RTCP overhead), which saves about 22% compared to the MDC solution of the same quality (about 2.0Mbps) and about 25-30% compared to FEC (which requires adding redundant packets).

[0206] 3. Subjective quality MOS (30% random packet loss):

[0207] 1) Present invention: 4.1±0.2;

[0208] 2) Traditional interpolation method: 2.4±0.3;

[0209] 3) MDC (single description loss repair): 3.0±0.2;

[0210] 4. Repair Delay: The additional computational delay introduced by the block-level repair process is less than 2ms. End-to-end latency is primarily determined by network transmission and codecs (typically in the 100-200ms range).

[0211] This embodiment verifies that the present invention can significantly improve the robustness and user experience of video communication in an unstable network environment in a typical software video conferencing scenario without significantly increasing system complexity and resource consumption.

[0212] Other application scenarios of the present invention

[0213] 1. Robot remote video conversation

[0214] In scenarios where artificial intelligence robots interact remotely with humans, the quality of video communication is crucial to the user experience. This invention can be applied to:

[0215] 1) Remote video interaction between service robots and users;

[0216] 2) Remote video connection between smart home robots and family members;

[0217] 3) Real-time video communication of service robots in public places;

[0218] 4) Video communication between assistive robots and people with limited mobility.

[0219] The features of the present invention in this scenario are:

[0220] 1) Ensure clear transmission of facial expressions and body movements to enhance interactive realism;

[0221] 2) Reduce the impact of network fluctuations on the robot's response speed;

[0222] 3) Maintain communication quality in unstable network environments such as public WiFi;

[0223] 4) Reduce the negative impact of video freezes on the human-computer interaction experience.

[0224] 2. Security monitoring

[0225] In remote video surveillance systems, cameras are often deployed in areas with unstable network conditions. The technology of this invention can ensure that the monitoring center can still obtain high-quality monitoring images even in harsh network environments. It is particularly suitable for:

[0226] 1) Urban security monitoring system;

[0227] 2) Highway traffic monitoring;

[0228] 3) Monitoring of facilities in remote areas.

[0229] 3. Remote video conferencing

[0230] In enterprise remote office and cross-regional meeting scenarios, this invention can significantly improve meeting quality and efficiency:

[0231] 1) Daily remote meetings and collaborations for multinational companies;

[0232] 2) Large-scale video conferencing system (supporting dozens to hundreds of participants);

[0233] 3) Team collaboration in a hybrid office environment;

[0234] 4) Remote emergency decision-making meetings of government agencies.

[0235] Advantages of this invention for remote conferencing:

[0236] 1) Reduce participant disconnection and screen freezes caused by network fluctuations;

[0237] 2) Maintain high-quality video streams to ensure accurate transmission of facial expressions and micro-expressions;

[0238] 3) Reduce bandwidth requirements, so that users on mobile devices and in weak network environments can also get a good experience;

[0239] 4. Industrial remote control

[0240] In the field of industrial automation remote monitoring and control, the present invention can be applied to:

[0241] 1) Remote control of mining equipment;

[0242] 2) Remote control of robots in hazardous environments;

[0243] 3) Smart factory production line monitoring.

[0244] 5. Online Education

[0245] In educational scenarios such as online classrooms and remote training, this technology can:

[0246] 1) Ensure clear and smooth transmission of teaching videos and reduce the impact of network instability on teaching;

[0247] 2) Adapt to the complex and changing network environment of students;

[0248] 3) Reduce bandwidth requirements and improve universality.

[0249] Therefore, the present invention adopts the above-mentioned video packet loss repair method based on image parity row / column reorganization and block coding. Based on image parity row / column reorganization and block coding, it utilizes the high similarity of parity rows and columns, and through innovative reorganization arrangement and block-level repair mechanism, it can achieve efficient packet loss repair without increasing delay and bandwidth, improve the robustness of video communication and user experience, solve the defects of existing technologies, and be applicable to a variety of real-time video communication scenarios.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video packet loss repair method based on image parity row / column reorganization and block coding, characterized in that: The following steps are involved: S1. At the sending end, the original video frame is reorganized into odd and even rows / columns to generate a recombined image; S2, using a single encoder to encode the entire reconstructed image; S3, separating two sub-streams, TS1 and TS2, from the single coded stream according to the region tag, corresponding to the coded block data of the odd region and the even region respectively, and encapsulating each separated sub-stream into data packets; S4. Receive and parse data packets at the receiving end, and perform block-level accurate repair when packet loss is detected; S5. Interleave and reorganize the pixels in the repaired odd-numbered row / column region and the even-numbered row / column region to restore the original image format.

2. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 1 is characterized in that: In S1, the image odd-even row / column reorganization includes an odd-even row reorganization mode and an odd-even column reorganization mode; The parity row reorganization mode is: Extract all odd-numbered and even-numbered pixels from the original image F(n) respectively; The odd-numbered rows are rearranged to the upper half of the image, maintaining their original order; Even-numbered rows are rearranged to the lower half of the image, maintaining their original order; Generate a reconstructed image F'(n), which has the same width and height as the original image F(n).

3. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 2, characterized in that: The parity column reorganization mode is: Extract all odd-numbered and even-numbered pixels from the original image F(n) respectively; The odd-numbered columns are rearranged to the left half of the image, maintaining the original order; Even-numbered columns are rearranged to the right half of the image, maintaining their original order; Generate a reconstructed image F'(n), which has the same width and height as the original image F(n).

4. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 3 is characterized in that: S2 is specifically: The single encoder uses a standard algorithm based on block coding to encode the reconstructed entire image F'(n).

5. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 4 is characterized in that: A single encoder uses a standard block-based encoding algorithm including any one of H.264, H.265, and AV1.

6. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 5, characterized in that: S3 specifically: For even-odd row reorganization mode: TS1 corresponds to all coded block data in the odd-numbered row area; TS2 corresponds to all coded block data in the even-numbered row area; For parity column reorganization mode: TS1 corresponds to all coded block data in the odd column area; TS2 corresponds to all coded block data in the even column area.

7. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 6, characterized in that: In S4, the specific process of receiving and parsing the data packet at the receiving end is as follows: S41, receiving and identifying TS1 and TS2 data streams from different transmission channels, and extracting the timestamp, sequence number, and coding block location information from each data packet; S42: Count the packet loss in TS1 and TS2 and the distribution information of the coding blocks contained therein, maintain the packet loss matrix L(x, y, t) to record the spatiotemporal packet loss distribution, and implement packet loss location at the coding block level; S43: Reassemble the received TS1 and TS2 data packets according to the original coding order to obtain a complete data stream for decoding, and mark the lost data block area.

8. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 7, characterized in that: In S4, block-level precise repair based on packet loss includes: Processing of areas without packet loss: For coding blocks with no packet loss in both odd and even areas, the pixel data of the corresponding areas are directly extracted from the decoded reconstructed image. Unilateral packet loss area repair: When only odd-numbered or even-numbered areas have lost code blocks, the lost areas are filled using inter-row / inter-column interpolation algorithms. Double-sided packet loss area repair: When the coding blocks at the same position in two areas are lost, the traditional packet loss repair solution is used.

9. The video packet loss repair method based on image parity row / column reorganization and block coding according to claim 8, characterized in that: S5 is specifically: For even-odd row reorganization mode: According to the spatial position relationship of the original image, the pixels in the odd-numbered row area and the even-numbered row area are reorganized in an interlaced manner: The pixels in the odd-numbered row area are placed back to their original odd-numbered row positions; The pixels in the even-numbered row area are placed back to their original even-numbered row positions; For parity column reorganization mode: According to the spatial position relationship of the original image, the pixels in the odd-numbered column area and the even-numbered column area are reorganized in an interleaved manner: The pixels in the odd column area are placed back to the original odd column position; The pixels in the even column area are placed back to their original even column positions.

Images