Error resilient video coding method and system based on channel importance-aware redundancy allocation
By evaluating the channel importance of the neural video codec and adaptively adjusting the redundancy allocation, the problem of unutilized channel importance differences in existing technologies is solved, achieving video quality preservation and robust transmission under high packet loss rates.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-03-25
- Publication Date
- 2026-06-09
AI Technical Summary
Existing error-resistant video coding methods fail to effectively utilize the differences in importance between neural video codec channels, resulting in critical channels not being adequately protected under high packet loss conditions, leading to a decline in video reconstruction quality.
By assessing channel importance based on the distribution area length criterion, adaptively adjusting redundancy allocation, prioritizing the protection of important channels, and utilizing redundant copies to recover lost data, differentiated protection is achieved.
Without increasing the total transmission bit rate, it significantly improves the reception reliability of critical channels under high packet loss conditions, thereby enhancing the robustness and quality of video transmission.
Smart Images
Figure CN122179565A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video coding and communication, and more specifically, to a robust video coding method and system based on channel importance-aware redundancy allocation. Background Technology
[0002] Real-time video communication has become increasingly widespread in recent years, accounting for a growing proportion of internet traffic. Despite the continuous evolution of communication networks, real-time video communication services still face the problem of data packet loss during video stream transmission, leading to frequent video quality degradation and video freezing. To ensure reliable video stream transmission, retransmitting lost data packets is a direct and widely used strategy in video communication. However, retransmission strategies can introduce significant transmission delays when the network round-trip time (RTT) is large, causing end-to-end video transmission latency to fail to meet real-time requirements (typically less than 1 second). Therefore, packet loss-resistant coding techniques are receiving increasing attention in real-time video communication scenarios.
[0003] Forward error correction (FEC) channel coding, a typical method for combating packet loss, includes traditional block codes (such as Reed-Solomon codes) and low-latency stream codes. It corrects bit errors in the received video bitstream by adding extra parity check codes to the output bitstream of the video source codec (such as H.264 and H.265). However, FEC-based error-resistant coding methods have a fixed redundancy ratio. When the actual packet loss rate exceeds the preset FEC ratio, it triggers a "cliff effect," causing the entire frame to become undecodeable. Conversely, when the packet loss rate is lower than the FEC ratio, it wastes additional bandwidth resources, making it difficult to adapt to dynamically changing network packet loss conditions.
[0004] In contrast, error concealment methods at the decoder end recover the original video content by designing a reconstruction mechanism on the decoder side to address the received incomplete information. Existing work either introduces manual intervention strategies in traditional video codecs such as H.264 / AVC for motion vector interpolation compensation, or uses neural networks to estimate and recover lost motion vectors and residual information. However, all of these methods divide video frames into multiple independent processing units, disrupting the overall coding structure between frames and thus reducing compression efficiency. Furthermore, when packet loss leads to severe loss of source information, the prior knowledge that the decoder can rely on is extremely limited, making it difficult to guarantee reconstruction quality.
[0005] In recent years, some studies have proposed end-to-end jointly optimized loss-resilient neural video coding methods. Cheng et al., in their paper "Grace: Loss-resilient real-time video through neural codecs" presented at the USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) in 2024, proposed an end-to-end optimized error-resistant neural video codec (NVC) framework that jointly trains the video encoder and decoder under packet loss channel conditions. Unlike traditional methods, this framework trains both the encoder and decoder simultaneously to adapt to the packet loss patterns of the channel, resulting in a smoother and more gradual performance degradation of video quality as the packet loss rate increases. However, existing error-resistant NVC frameworks neglect the inherent statistical properties of the coding feature channels. Different feature channels in the coding tensor contribute significantly to the reconstruction quality at the receiver. Applying an equal-weighted packet loss protection strategy to each channel leads to insufficient protection of critical channels under high packet loss rates, resulting in a significant decrease in video reconstruction quality.
[0006] Currently, there is no systematic research on adaptive redundancy allocation based on the differences in channel importance in neural video codecs. How to improve video transmission quality under extreme packet loss conditions by performing differentiated protection based on channel importance within a limited transmission bitrate budget is a key technical problem that needs to be solved. Summary of the Invention
[0007] In view of the deficiencies in the prior art, the purpose of this application is to provide a robust video coding method and system based on channel importance-aware redundancy allocation.
[0008] A first aspect of this application provides a robust video coding method based on channel importance-aware redundancy allocation, comprising the following steps: The input video frames are encoded using an anti-packet loss neural video encoder to generate quantized feature tensors corresponding to motion vector feature maps and prediction residual feature maps; The importance of each channel of the quantized feature tensor is evaluated based on the distribution region length criterion. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of the channel. All channels are sorted from largest to smallest according to the importance index. Based on the current channel packet loss rate, the threshold for the number of important channels is adaptively determined using the cumulative importance summation function. Low-importance channels below the threshold are pruned, and the released bit rate budget is redistributed to important channels that reach or exceed the threshold using redundant copying. The channels are then packaged and transmitted through the packet loss channel. The receiving end sets the elements of the lost channels and the pruned channels to zero, uses multiple redundant copies to complement and recover the complete data of the important channels, and inputs the restored feature tensor into the anti-packet loss neural video decoder to reconstruct the video frames.
[0009] Optionally, the importance evaluation of each channel of the quantized feature tensor based on the distribution region length criterion, using the difference between the upper and lower bounds of the latent representation of each channel on the training dataset as the importance index of that channel, and sorting all channels in descending order of importance index, includes: traversing the training dataset. Statistical analysis of all training samples. The supremum of the potential representation of each channel serves as the upper bound of the region within the channel distribution. The statistical infimum serves as the lower bound of the region within the channel distribution. The importance index of the channel is defined by the difference between the upper and lower bounds. ; Rank all N channels according to their importance index Arranged from largest to smallest.
[0010] Optionally, the step of adaptively determining the threshold for the number of important channels using a cumulative importance summation function based on the current channel packet loss rate includes: defining a cumulative importance summation function. For the front The ratio of the sum of the importance indicators of each channel to the total sum of the importance indicators of all channels. ; Set to satisfy of The smallest channel index that is equal to 1 minus the current packet loss rate (PLR). As the packet loss rate increases, the number of protected important channels decreases, and the number of redundant copies allocated to each important channel increases to ensure reliable reception of important channels under high packet loss rates.
[0011] Optionally, the step of pruning low-importance channels below the threshold number involves redistributing the released bit rate budget to important channels that reach or exceed the threshold number using redundant copies. This includes removing low-importance channels with lower importance rankings (below the threshold number) from the feature tensor. While maintaining the total transmission bit rate budget unchanged, the saved bit rate is used to perform additional redundant copies and distribute the packets for transmission of important channels that reach or exceed the threshold number.
[0012] Optionally, the redundant copy prioritizes the protection of important channels corresponding to the motion vector feature map; in the case of packet loss, the motion vector determines the basic quality of the reconstructed frame, and the prediction residual further enhances the reconstruction quality; priority is given to ensuring the complete reception of important channels in the motion vector feature map, and no additional redundant protection may be applied to the prediction residual feature map.
[0013] Optionally, the power scaling factor Different values were set for the motion vector feature map and the prediction residual feature map to accommodate the differences in the channel statistical characteristics of the two types of feature maps.
[0014] Optionally, the packet loss resistant neural video codec adopts an end-to-end joint training method. During the training phase, a packet loss channel is simulated by randomly setting zeros, and the encoder and decoder are jointly optimized with the rate-distortion loss function as the optimization objective.
[0015] A second aspect of this application provides a channel importance-aware redundancy allocation system for loss-resistant video coding in neural video codecs, comprising: Encoding module: The input video frame is encoded using a packet loss resistant neural video encoder to generate a quantized feature tensor corresponding to the motion vector feature map and the prediction residual feature map; the encoding module also makes redundant copies of important channels based on the output of the adaptive redundancy allocation module, and sends them through the packet loss channel after being distributed and packaged. Channel importance assessment module: Based on the distribution region length criterion, the importance of each channel of the quantized feature tensor is assessed. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of the channel. All channels are sorted from largest to smallest according to the importance index. Adaptive Redundancy Allocation Module: Based on the current channel packet loss rate, it adaptively determines the threshold number of important channels using the cumulative importance summation function, prunes low-importance channels below the threshold number, redistributes the released bit rate budget to important channels that reach or exceed the threshold number using redundant copying, and sends the redistribution result to the encoding module. Decoding module: Receives data packets from the lost channel, sets the missing channel elements and pruned channel elements to zero, uses multiple redundant copies to complement and recover the complete data of important channels, inputs the restored feature tensor into the anti-packet loss neural video decoder, and reconstructs video frames.
[0016] Compared with the prior art, the embodiments of this application have at least one of the following beneficial effects: (1) This application introduces a channel importance assessment method based on the distribution region length criterion, which reveals that there are significant differences in the contribution of different channels to the anti-packet loss performance in the latent representation of neural video codecs.
[0017] (2) The heuristic redundancy allocation method proposed in this application can adaptively adjust the number and redundancy of protected channels according to the real-time packet loss rate, and has more robust transmission performance in dynamic packet loss environment.
[0018] (3) The superiority of the proposed method has been verified on a variety of public video coding datasets (HEVC series, UVG, etc.), especially in extreme scenarios with packet loss rate exceeding 60%, where it has significant performance advantages.
[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description
[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary embodiment illustrating a fault-resistant video coding method based on channel importance-aware redundancy allocation; Figure 2 A flowchart of a robust video coding method based on channel importance-aware redundancy allocation, according to an exemplary embodiment, is shown at the transmitting end. Figure 3 A flowchart of a receiver for a fault-resistant video coding method based on channel importance-aware redundancy allocation, according to an exemplary embodiment; Figure 4 This is a structural diagram of a fault-resistant video coding system based on channel importance-aware redundancy allocation, according to an exemplary embodiment. Figure 5 This is a schematic diagram illustrating the sending and receiving processes according to an exemplary embodiment. Detailed Implementation
[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.
[0022] Reference Figure 1 As shown, in one embodiment of this application, a fault-resistant video coding method based on channel importance-aware redundancy allocation is provided. This method may include steps S1-S4: S1, the input video frames are encoded using a neural video encoder to generate a quantized feature tensor; In this step, the quantization feature tensor includes the first quantization tensor corresponding to the motion vector feature map and the second quantization tensor corresponding to the prediction residual feature map. The first quantization tensor and the second quantization tensor are independently evaluated for channel importance and redundancy allocation.
[0023] S2, based on the importance assessment of each channel, sort the channels and prune low-importance channels; In this step, the importance of each channel of the quantized feature tensor is evaluated based on the distribution region length criterion. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of that channel, and all channels are sorted from largest to smallest according to the importance index.
[0024] S3 redistributes the released bit rate budget to important channels in a redundant copy manner, packages it, and sends it to the channels; In this step, based on the current channel packet loss rate, a threshold for the number of important channels is adaptively determined using a cumulative importance summation function. Low-importance channels below this threshold are pruned, and the released bit rate budget is redistributed to important channels that reach or exceed the threshold using redundant copies. These channels are then packaged and transmitted via the packet loss channel. For example, during packet loss channel transmission, data packets are transmitted through a random packet loss channel following a Bernoulli distribution, and each data packet is lost independently and randomly at a packet loss rate (PLR).
[0025] The operations in steps S1-S3 above can be processed at the sending end.
[0026] For example, the above-mentioned packaging may include: encapsulating multiple redundant copies of the same important channel into different data packets, and assigning different transmission priorities to different data packets, with the core redundant copy (the first allocated copy) assigned the highest transmission priority.
[0027] S4 sets the elements of the channel corresponding to the lost data packet and the elements of the pruned channel to zero. At the same time, it uses multiple redundant copies to complement and restore the complete data of the important channels. The restored feature tensor is input into the error-resistant neural video decoder to reconstruct the current video frame.
[0028] Setting the elements of the pruned channel to zero means setting all feature elements of the pruned channel to zero uniformly, without distinguishing whether the channel has packet loss.
[0029] The operation in step S4 above can be reconstructed at the receiving end.
[0030] The embodiments described above in this application significantly improve the reception reliability of important channels under high packet loss conditions by performing channel importance-aware differentiated redundancy allocation on the quantization feature tensor of the neural video codec without increasing the total transmission bit rate, thereby achieving more robust error-resistant video transmission.
[0031] Reference Figure 2 , Figure 5 As shown, to achieve channel importance-aware redundancy allocation, in one embodiment of this application, the following operations can be performed at the transmitting end (encoding end): S110, Packet Loss Resistant Neural Video Coding: Neural video encoding is performed on the video frames to be sent to generate quantized feature tensors corresponding to motion vector feature maps and prediction residual feature maps. The quantized feature tensors are then grouped and packaged for transmission.
[0032] Specifically, during the training phase, the encoder and decoder simulate a packet loss channel by randomly setting the values to zero, and perform end-to-end joint optimization with the rate-distortion loss function as the optimization objective, so that the encoder and decoder can jointly adapt to the packet loss channel conditions.
[0033] For example, a packet loss-resistant neural video codec employs an end-to-end joint training approach. During the training phase, it simulates a packet loss channel by randomly zeroing the quantized feature tensor, using a rate-distortion loss function. To optimize the target, the encoder and decoder are jointly optimized, where λ is the balance coefficient, D is the distortion between the reconstructed image and the original image, and R is the coding bit rate.
[0034] S120, Channel Importance Assessment: Based on the distribution region length criterion, the importance of each channel of the quantized feature tensor is evaluated and ranked using the training dataset.
[0035] Specifically, traverse the training dataset Statistical analysis of all training samples. The supremacy of the potential representation of each channel As the upper boundary of the area within the distribution of this channel Statistical definition As the lower realm The importance index of the channel is defined by the difference between the upper and lower bounds. : All Each channel is ranked by importance index Arranged from largest to smallest.
[0036] It should be noted that S120 is an offline pre-computation step, which only needs to be executed once after the model training is completed, and does not need to be recalculated during each frame encoding.
[0037] S130, Adaptive Channel Selection: Based on the current channel packet loss rate (PLR), the cumulative importance summation function is used. Adaptive determination of the threshold for the number of important channels .
[0038] Specifically, Defined as before The ratio of the sum of the importance indicators of each channel to the total sum of all channel importance indicators: Setting to meet Minimum channel index As an important threshold for the number of channels, among which The scaling factor is used to determine the number of important channels: in Represents the cumulative sum function The inverse function of .
[0039] It should be noted that the encoding end can obtain the current channel packet loss rate (PLR) through feedback from the receiving end (decoding end) or through timing prediction. Different power scaling factors are set for the motion vector feature map and the predicted residual feature map. To accommodate the differences in the importance distribution characteristics of the two types of feature map channels, the number of protected important channels decreases as the packet loss rate increases, while the number of redundant copies allocated to each important channel increases to ensure reliable reception of important channels under high packet loss rates.
[0040] The above-mentioned timing prediction method for obtaining the current channel packet loss rate (PLR) can be implemented using existing technologies.
[0041] S140, Channel Rate Redistribution: Ranked by threshold Subsequent low-importance channels are pruned from the feature tensor; while keeping the total transmission bit rate budget unchanged, the bit rate freed up by pruning is redistributed to those arranged at the threshold. Previously important channels were copied redundantly and packaged separately.
[0042] It should be noted that in this embodiment, redundancy protection is prioritized for important channels of the motion vector feature map, while no additional redundancy protection is applied to the prediction residual feature map.
[0043] For example, the allocation rule for redundant copies can be: the higher the channel importance index and the higher the current channel packet loss rate, the more redundant copies are allocated, and the bit rate occupied by each redundant copy does not exceed the set threshold of the original bit rate of the channel, such as 50%.
[0044] S150, Data packet transmission: Each element of the encoded tensor is randomly sent to different data packets, and the packaged data packets are sent to the receiving end (decoding end) via the packet loss channel.
[0045] The embodiments described above in this application combine offline channel importance ranking with online adaptive bit rate reallocation to achieve differentiated protection of key characteristic channels without increasing the total transmission bit rate.
[0046] The error-resistant video coding method and system based on channel importance-aware redundancy allocation in the above embodiments of this application, at the encoding end, uses a neural video codec to encode the motion vector feature map and prediction residual feature map of the current frame to obtain a quantized feature tensor; based on the in-distribution (IND) region length criterion, the importance of each channel of the quantized feature tensor is evaluated, and the importance index of each channel is calculated; according to the packet loss rate of the current channel, the number of important channels that need to be protected is adaptively determined using a cumulative importance summation function; important channels are retained, unimportant channels are eliminated, and the bit rate resources released by eliminating unimportant channels are reallocated to important channels, improving the packet loss resistance of important channels through multiple copies; at the decoding end, the lost channel elements are set to zero, and the video frame is reconstructed using the error-resistant decoder. This application has significant performance advantages over existing methods in extreme packet loss scenarios (packet loss rate exceeding 60%), demonstrating superior robustness and video quality preservation capabilities.
[0047] Reference Figure 3 , Figure 5 As shown in one embodiment of this application, the following operations can be performed at the receiving end: S210, Data Packet Reception and Channel Restoration: Receive data packets from the lost packet channel and restore the data for each channel according to the channel index.
[0048] Specifically, for channels that fail to receive any copies due to packet loss, the corresponding feature tensor elements are set to zero; for low-importance channels pruned by the sender, the corresponding elements are also set to zero to maintain the complete dimensional structure of the feature tensor for subsequent decoder processing.
[0049] S220, redundant replica complementary recovery: For critical channels protected by redundancy, multiple redundant copies are used for complementary recovery to restore complete channel data.
[0050] Specifically, for multiple redundant copies of the same important channel, each copy experiences independent random packet loss during transmission, resulting in different positions of the unlost elements in different copies. The receiving end traverses each copy, taking the corresponding element from the first successfully received copy as the recovery value for each element position, thereby complementing and merging multiple incomplete copies to recover the complete channel data.
[0051] It should be noted that if a certain element is missing in all replicas due to packet loss, that element will still be set to zero.
[0052] S230, anti-packet loss neural video decoder: The feature tensor restored after processing by S210 and S220 is input into the anti-packet loss neural video decoder to reconstruct the current video frame.
[0053] Specifically, the anti-packet loss neural video decoder has undergone end-to-end joint optimization through simulated packet loss channels during the training phase, enabling it to adaptively reconstruct video frames based on incomplete feature tensors and output the reconstructed video frames as the final result.
[0054] The above-described receiver embodiment of this application effectively ensures the quality of reconstructed video frames under high packet loss conditions through the collaborative processing of redundant copy complementary recovery and anti-packet loss neural video decoder.
[0055] Reference Figure 4 As shown, based on the same technical concept, other embodiments of this application also provide a robust video coding system based on channel importance-aware redundancy allocation, used to implement the robust video coding method based on channel importance-aware redundancy allocation in any of the above embodiments. Specifically, the system includes: Encoding module: The input video frame is encoded using a packet loss resistant neural video encoder to generate a quantized feature tensor corresponding to the motion vector feature map and the prediction residual feature map; the encoding module also makes redundant copies of important channels based on the output of the adaptive redundancy allocation module, and sends them through the packet loss channel after being distributed and packaged. Channel importance assessment module: Based on the distribution region length criterion, the importance of each channel of the quantized feature tensor is assessed. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of the channel. All channels are sorted from largest to smallest according to the importance index. Adaptive Redundancy Allocation Module: Based on the current channel packet loss rate, it adaptively determines the threshold number of important channels using the cumulative importance summation function, prunes low-importance channels below the threshold number, redistributes the released bit rate budget to important channels that reach or exceed the threshold number using redundant copying, and outputs the redistribution result to the encoding module. Decoding module: Receives data packets from the lost channel, sets the missing channel elements and pruned channel elements to zero, uses multiple redundant copies to complement and recover the complete data of important channels, inputs the restored feature tensor into the anti-packet loss neural video decoder, and reconstructs video frames.
[0056] In the above embodiments, the channel importance assessment module sorts all channels in descending order of importance index and generates a channel importance index table; the adaptive redundancy allocation module receives the channel importance index table and the current channel packet loss rate output by the channel importance assessment module, adaptively determines the threshold for the number of important channels using the cumulative importance summation function, prunes low-importance channels, and redistributes the released bit rate budget to important channels in a redundant copy manner, outputting the redistribution result to the encoding module; the encoding module is set at the transmitting end, performs packet loss-resistant neural video coding on the input video frame, and generates a quantized feature tensor; based on the output result of the adaptive redundancy allocation module, redundant copies are made for important channels and distributed and packaged, and sent to the receiving end through the packet loss channel, and then the decoding module reconstructs the video frame.
[0057] The specific implementation of each module in the above embodiments of this application can refer to the implementation technology of the corresponding steps of the error-resistant video coding method based on channel importance-aware redundancy allocation in the above embodiments, and will not be repeated here.
[0058] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.
[0059] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.
[0060] The channel importance characteristics of a pre-trained packet loss-resistant neural video codec model were evaluated using the Vimeo-90K dataset. Offline channel importance statistics were performed on the motion vector feature map and the prediction residual feature map, and different Lagrange multipliers were generated. The channel importance index table is set under the following conditions. Power scaling factors are applied to the motion vector feature map and the predicted residual feature map. The values were set to 4.0 and 2.0 respectively. The channel model adopted a completely random and independent Bernoulli packet loss model.
[0061] The test datasets used HEVC series (Class B 1080P, Class C 480P, Class D 240P), UVG (1080P), FVC (1080P), and 45 randomly selected 10-second video clips (720P, 360P) from the Kinetics dataset. The encoding bitrate was set to 0.20 bpp, and PSNR and SSIM were used as evaluation metrics for video reconstruction quality.
[0062] Under the experimental settings described above, the proposed method exhibits a smooth and gradual performance degradation trend with increasing packet loss rate on all test datasets, without a "cliff effect." When the packet loss rate increases from 0 to 90%, the SSIM / PSNR of the proposed method only decreases by an average of 1.9 / 4.7 dB. Even in extreme packet loss scenarios with a packet loss rate exceeding 60%, it can still maintain high video reconstruction quality, verifying the effectiveness and robustness of the proposed method.
[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0067] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.
Claims
1. A robust video coding method based on channel importance-aware redundancy allocation, characterized in that, include: The input video frames are encoded using an anti-packet loss neural video encoder to generate quantized feature tensors corresponding to motion vector feature maps and prediction residual feature maps; The importance of each channel of the quantized feature tensor is evaluated based on the distribution region length criterion. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of the channel. All channels are sorted from largest to smallest according to the importance index. Based on the current channel packet loss rate, the threshold for the number of important channels is adaptively determined using the cumulative importance summation function. Low-importance channels below the threshold are pruned, and the released bit rate budget is redistributed to important channels that reach or exceed the threshold using redundant copying. The channels are then packaged and transmitted through the packet loss channel. The receiving end sets the elements of the lost channels and the pruned channels to zero, uses multiple redundant copies to complement and recover the complete data of the important channels, and inputs the restored feature tensor into the anti-packet loss neural video decoder to reconstruct the video frames.
2. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 1, characterized in that, The importance of each channel of the quantized feature tensor is evaluated based on the in-distribution region length criterion. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of that channel. All channels are sorted from largest to smallest according to the importance index, including: Traversing the training dataset Statistical analysis of all training samples. The supremum of the potential representation of each channel serves as the upper bound of the region within the channel distribution. The statistical infimum serves as the lower bound of the region within the channel distribution. ; The importance index of the channel is defined by the difference between the upper and lower bounds. ; All N channels are ranked by importance. Arranged from largest to smallest.
3. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 1, characterized in that, The step of adaptively determining the threshold for the number of important channels based on the current channel packet loss rate using a cumulative importance summation function includes: Define a cumulative importance summation function For the front The ratio of the sum of the importance indicators of each channel to the total sum of the importance indicators of all channels. ,in This represents the importance index of the i-th channel; Setting to meet Minimum channel index As an important threshold for the number of channels, among which PLR is the power scaling factor, and PLR is the current channel packet loss rate. As the packet loss rate increases, the threshold for the number of critical channels increases. As the number of redundant copies decreases, the number of redundant copies obtained by each important channel increases accordingly.
4. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 3, characterized in that, The power scaling factor Different values were set for the motion vector feature map and the prediction residual feature map to accommodate the differences in the channel importance distribution characteristics of the two types of feature maps.
5. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 1, characterized in that, The redundant copies are preferentially allocated to the important channels corresponding to the motion vector feature map, and additional redundancy protection is applied or not applied to the prediction residual feature map.
6. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 1, characterized in that, The method of using multiple redundant copies to complementarily restore the complete data of important channels includes: For multiple redundant copies of the same important channel, traverse each copy and take the corresponding element in the first successfully received copy as the recovery value for each element position. If an element is missing in all replicas due to packet loss, then that element is set to zero.
7. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 1, characterized in that, The packet loss-resistant neural video codec employs an end-to-end joint training approach. During the training phase, it simulates a packet loss channel by randomly zeroing the quantization feature tensor, using a rate-distortion loss function. To optimize the target, the encoder and decoder are jointly optimized, where λ is the balance coefficient, D is the distortion between the reconstructed image and the original image, and R is the coding bit rate.
8. The error-resistant video coding method based on channel importance-aware redundancy allocation according to claim 1, characterized in that, The current channel packet loss rate is obtained from feedback from the receiver or based on timing prediction.
9. A robust video coding system based on channel importance-aware redundancy allocation, characterized in that, include: Encoding module: The input video frames are encoded using a packet loss-resistant neural video encoder to generate quantized feature tensors corresponding to motion vector feature maps and prediction residual feature maps; The encoding module also creates redundant copies of important channels based on the output of the adaptive redundancy allocation module, and then distributes and packages them before sending them through the packet loss channel. Channel importance assessment module: Based on the distribution region length criterion, the importance of each channel of the quantized feature tensor is assessed. The difference between the upper and lower bounds of the latent representation of each channel on the training dataset is used as the importance index of the channel. All channels are sorted from largest to smallest according to the importance index. Adaptive Redundancy Allocation Module: Based on the current channel packet loss rate, it adaptively determines the threshold number of important channels using the cumulative importance summation function, prunes low-importance channels below the threshold number, redistributes the released bit rate budget to important channels that reach or exceed the threshold number using redundant copying, and sends the redistribution result to the encoding module. Decoding module: Receives data packets from the lost channel, sets the missing channel elements and pruned channel elements to zero, uses multiple redundant copies to complement and recover the complete data of important channels, inputs the restored feature tensor into the anti-packet loss neural video decoder, and reconstructs video frames.
10. The error-resistant video coding system based on channel importance-aware redundancy allocation according to claim 9, characterized in that, The adaptive redundancy allocation module prioritizes redundancy protection for important channels corresponding to motion vector feature maps, and applies or does not apply additional redundancy protection to the prediction residual feature maps.