Neural network-based packet loss video key region reconstruction method

Abstract

The application relates to a neural network-based key area reconstruction method for a packet loss video, which comprises the following steps: acquiring a current frame to be reconstructed, at least one time-adjacent reference frame and a loss identifier; scoring the key areas of the frame to be reconstructed according to the loss identifier and the reference frame, and selecting the key areas whose scores meet a preset threshold; inputting the available pixels in the key areas, the time context of the corresponding areas of the reference frame and the loss identifier into a reconstruction neural network, and outputting at least two reconstruction candidates; logically judging the candidates, wherein the judgment includes boundary transition continuity, cross-frame change consistency and subject structure integrity judgment, and the candidates meeting the preset threshold are passed; selecting the passed candidates as the key area output, and generating a reconstruction result according to a fallback strategy if none of the candidates is passed; fusing the key area output and non-key areas to obtain a reconstructed frame; and repeatedly performing the acquisition, determination, generation, judgment and fusion steps on the subsequent frames to be reconstructed in a video frame sequence, and outputting a reconstructed video sequence.

Description

Technical Field

[0001] This invention relates to the field of digital video processing technology, specifically to a method for reconstructing key regions of lost video based on neural networks. Background Technology

[0002] Current technologies, such as those disclosed in CN115103188A, show significant shortcomings in neural network-based methods for reconstructing key regions in lost videos. From the overall design perspective, the core of this approach still revolves around scalable video coding systems, focusing primarily on overall image recovery in scenarios with packet loss at the enhancement layer. It fails to introduce a mechanism to differentiate between semantic or perceptual levels of video content. Furthermore, the method does not distinguish between the subjective perceptual differences between faces, text, main targets, and background areas during reconstruction; all regions are treated uniformly in the error-hiding stage. This can easily lead to a waste of computational resources in low-perceptual-value areas in practical applications, and may still fail to achieve sufficient reconstruction quality improvement when packet loss occurs in key visual regions. From the perspective of neural network input and modeling information, this scheme mainly relies on the offset parameters and residual information between the base layer and enhancement layer of the previous frame. These information are essentially still derived from the inter-layer relationships within the coding structure. Although a convolutional neural network is introduced for learning, its feature sources have strong coding correlation and structural dependence, which means that the method needs to be re-adapted to the model when facing different coding configurations, different GOP structures, or different layering strategies, thus limiting its versatility and transferability.

[0003] Regarding temporal consistency, this document primarily utilizes information from the previous frame for error compensation and residual fusion. However, it lacks explicit cross-frame stability constraints or quantitative evaluation mechanisms, and also lacks comparison and selection strategies for multiple candidate results. When the network prediction results exhibit slight instability between adjacent frames, flickering, jitter, or texture drift issues can easily occur, especially under high packet loss ratios or complex motion scenes, where these problems are further amplified. From a motion modeling perspective, although this scheme uses offsets and deformable convolutions to enhance spatial adaptability, it does not classify or determine the reliability of motion information, and lacks a backoff mechanism for abnormal motion estimation results. Once the offset estimation itself is affected by noise or packet loss, erroneous information may propagate through the network and be further amplified, ultimately reducing the final reconstruction quality.

[0004] This document does not propose a clear fallback reconstruction strategy when processing failures or insufficient reconstruction quality, and lacks a fallback scheme based on a combination of reference frame temporal compensation and boundary transition suppression. This poses a significant risk in real-time video scenarios with frequent network fluctuations, as the system lacks a stable and reliable backup output once the neural network output fails. Regarding region fusion and boundary processing, existing methods focus more on overall image-level restoration, lacking specific design for boundary transitions between critical and non-critical regions, brightness continuity, and high-frequency artifact suppression, easily leading to noticeable visual breaks at region transitions. From an engineering implementation and real-time perspective, this scheme requires multi-level convolutional networks to work collaboratively and is highly dependent on encoding-side information. In real-time video conferencing or weak network environments, the complexity, latency control, and computational power allocation at the decoding end face significant pressure, making dynamic adjustment based on the current network state and content importance difficult. Summary of the Invention

[0005] The purpose of this invention is to provide a method for reconstructing key regions of lost video based on neural networks, thereby addressing some of the drawbacks and shortcomings pointed out in the background art.

[0006] The present invention adopts the following technical solution to solve the above-mentioned technical problems: a method for reconstructing key regions of packet-lost video based on neural networks, comprising: acquiring the current frame to be reconstructed, at least one temporally adjacent reference frame and a missing identifier; performing key scoring on each region of the frame to be reconstructed based on the missing identifier and the reference frame, and selecting key regions whose scores meet a preset threshold.

[0007] The available pixels in the key area, the temporal context of the corresponding area of ​​the reference frame, and the missing identifier are input into the reconstruction neural network, which outputs at least two reconstruction candidates. Logical adjudication is performed on the candidates, which includes judgment of boundary transition continuity, cross-frame change consistency, and main structure integrity. Those that meet the preset threshold are passed.

[0008] Candidates that pass the adjudication are selected as key regions for output. If none pass, a reconstruction result is generated according to the backoff strategy. The key region output is fused with the non-key region to obtain the reconstructed frame. The acquisition, determination, generation, adjudication and fusion steps are repeated for subsequent frames to be reconstructed in the video frame sequence to output the reconstructed video sequence.

[0009] Furthermore, the criticality score reflects both the impact of packet loss and the viewing perception sensitivity, wherein the viewing perception sensitivity is determined based on at least one of the face region, text region, or main target region; when the total area of ​​the critical region exceeds the area limit corresponding to the preset calculation budget, the critical regions are selected from high to low according to the criticality score until the area limit is met.

[0010] Furthermore, the at least two reconstruction candidates are generated by the reconstruction neural network with different reconstruction intensities, the different reconstruction intensities including at least different detail enhancement intensities or different temporal constraint intensities; in the cross-frame change consistency judgment, the logical decision selects the reconstruction candidate with the smallest cross-frame change amplitude and which passes the boundary transition continuity judgment as the key region output.

[0011] Furthermore, when all reconstruction candidates fail the logical decision, the backoff strategy includes: performing temporal compensation on the key region based on the region corresponding to the reference frame to generate a backoff reconstruction result, and performing transition suppression processing on the backoff reconstruction result at the boundary of the key region to meet the boundary transition continuity judgment; the fusion sets a transition band at the boundary of the key region, and performs fusion within the transition band according to the weights of the key region output and the non-key region.

[0012] Furthermore, the different reconstruction intensities are generated by the reconstruction neural network using at least two sets of preset inference parameters under the same input conditions. The preset inference parameters include at least detail enhancement intensity parameters and temporal constraint intensity parameters. The values ​​of the detail enhancement intensity parameters and temporal constraint intensity parameters are determined based on the missing range indicated by the missing identifier and the availability of adjacent reference frames.

[0013] Further, obtaining the cross-frame change amplitude includes: aligning the regions corresponding to the key regions in the previous frame or reference frame based on motion information, and calculating the difference between the aligned results and each reconstruction candidate within the key regions to obtain the corresponding cross-frame change amplitude; wherein, for the first... The cross-frame variation magnitude of each reconstruction candidate is denoted as ,satisfy:

[0014]

[0015] in, The set of pixels in the key region. The number of pixels in the pixel set. The pixel positions within the key region; The first frame to be reconstructed in the current frame One reconstruction candidate at pixel location Pixel value at; The region in the previous frame or reference frame that corresponds to the key region; An alignment operator based on the motion information, used to... Align with the current frame to be rebuilt; This is a gradient operator used to characterize boundary and texture changes; For the penalty function, Preset to positive numbers; Preset weighting coefficients are used to balance pixel difference terms and gradient difference terms; and the logical decision introduces a stability threshold in the cross-frame change consistency judgment, when the... When the value is below the stability threshold, the reconstruction intensity corresponding to the selected candidate in the previous frame is maintained first to suppress flicker.

[0016] Furthermore, the timing compensation includes: calculating or obtaining motion information based on the region corresponding to the key region in the reference frame, and aligning the corresponding region to the key region of the current frame to be reconstructed based on the motion information to generate the backtracking reconstruction result; wherein, when there are multiple reference frame corresponding regions within the key region, the reference frame corresponding region with the smallest time interval to the current frame to be reconstructed and the highest available pixel ratio is selected to perform the timing compensation.

[0017] Furthermore, the transition suppression process includes performing boundary artifact suppression on the backtracking reconstruction results within a preset width range of the critical region boundary. The boundary artifact suppression includes at least: attenuating the high-frequency detail components at the boundary and maintaining the continuity of brightness and chromaticity changes on both sides of the boundary; and the preset width is adjusted according to the size of the packet loss range indicated by the missing identifier.

[0018] Furthermore, the motion information includes motion vectors obtained from the decoding end, optical flow information calculated from adjacent frames, or a combination of the two; when both motion vectors and optical flow information are available, coarse alignment is first performed based on the motion vectors, and then fine alignment is performed based on the optical flow information to generate the backtracking reconstruction result.

[0019] Furthermore, the optical flow information is subjected to a credibility determination in the key region. The credibility determination includes at least a consistency judgment of alignment residuals and a continuity judgment of boundary motion. When the optical flow information fails the credibility determination, the fine alignment is stopped and only the coarse alignment result based on motion vectors is used as the backtracking reconstruction result, or the sub-regions that fail the credibility determination are finely aligned using neighborhood smoothing optical flow substitution values.

[0020] The beneficial effects of this invention are as follows: By combining missing identifiers and reference frame information, key regions of video frames are scored, and key regions are adaptively selected under computational budget constraints. Multi-candidate, high-precision neural network reconstruction is performed only on key regions, while fusion completion is used for non-key regions. This effectively reduces the overall computational complexity while ensuring the reconstruction quality of face, text, and subject target perception sensitive regions, and improves the applicability of the method in real-time or resource-constrained scenarios.

[0021] Multiple candidate results with different reconstruction intensities are generated, and a logical adjudication mechanism including boundary transition continuity, cross-frame change consistency, and main structure integrity is introduced. This mechanism can adaptively select the reconstruction result with the most stable temporal sequence and reasonable structure, significantly suppressing reconstruction flicker and structural distortion. When none of the candidates meet the adjudication conditions, a backoff strategy combining temporal compensation of motion information and boundary transition suppression is adopted to ensure that the reconstruction result still has good temporal continuity and visual consistency under extreme packet loss conditions, thereby improving the overall stability, robustness, and viewing experience of key area reconstruction in packet-loss videos. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the multi-candidate logic decision-making process for reconstructing key regions of lost video in this invention.

[0023] Figure 2 This invention provides a fusion graph for the multi-intensity candidate consistency decision and backoff fusion in the key region scoring budget screening.

[0024] Figure 3 This is a functional relationship diagram of the key area rollback and reconstruction of the present invention.

[0025] Figure 4 This is a schematic diagram of packet loss area statistics and key area screening in Embodiment 1 of the present invention.

[0026] Figure 5 This is the spatial decision diagram of the reconstructed intensity parameters in Embodiment 1 of the present invention.

[0027] Figure 6 This is a schematic diagram of cross-frame variation amplitude decomposition and stability threshold determination in Embodiment 1 of the present invention.

[0028] Figure 7 This is a diagram illustrating the evolution of the multi-reference frame filtering ranking in Embodiment 2 of the present invention.

[0029] Figure 8 This is a comparison diagram of the continuity of phase space scatter cloud brightness and high-frequency artifact energy in the transition suppression treatment of Embodiment 2 of the present invention.

[0030] Figure 9 This is a schematic diagram illustrating the fusion of coarse alignment of motion vectors and fine alignment of optical flow in Embodiment 2 of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] Combined with appendix Figure 1This invention relates to a method for reconstructing key regions of lost video based on neural networks. The method obtains the current frame to be reconstructed from the lost video at the receiving end, and at the same time obtains at least one reference frame that is temporally adjacent to the current frame to be reconstructed. The reference frame is a video frame that has been successfully decoded and can be used for timing compensation or content inference. Simultaneously, the method obtains missing identification information to indicate the location of missing pixels in the current frame to be reconstructed due to packet loss. The missing identification information is used to clearly identify the range and distribution of the area affected by packet loss.

[0033] By combining missing pixel information and reference frames, multiple regions in the current frame to be reconstructed are analyzed and processed separately. A key score is generated by comprehensively evaluating the impact of packet loss and the perceived importance of each region. The impact of packet loss reflects the number of missing pixels, the continuity of their distribution, and the degree of disruption to the reference relationship within the region. The perceived importance reflects whether the region contains content that significantly impacts the user's viewing experience. Subsequently, the key scores of each region are compared with pre-set thresholds. Regions whose key scores meet the pre-set thresholds are selected as key regions for subsequent focused reconstruction processing.

[0034] Unaffected by packet loss, usable pixel information is extracted from the key region. Simultaneously, regions corresponding to the spatial location of the key region are extracted from at least one temporally adjacent reference frame as temporal context information. This information, combined with a missing marker indicating the location of the missing data, is then uniformly input into a pre-trained reconstruction neural network. Based on the input spatial and temporal information, the reconstruction neural network infers the missing content within the key region and generates at least two distinct reconstruction candidate results under the same input conditions. These candidate results differ in detail intensity or temporal constraint level, providing multiple reconstruction schemes.

[0035] After generating reconstruction candidate results, a logical adjudication process is performed on each candidate result. This adjudication includes: judging the continuity of pixel transitions between the candidate result and surrounding areas at the critical region boundaries to avoid obvious seams or edge breaks; simultaneously, judging the consistency of cross-frame changes between the candidate result and the corresponding region in the reference frame over time to suppress unreasonable jumps in the reconstruction result between adjacent frames; and also judging the integrity of the main structure of the critical region in the candidate result to avoid deformation of the main outline or structural loss. Only when a reconstruction candidate result meets the preset threshold conditions in all of the above judgments is it determined to have passed the logical adjudication and used for subsequent critical region reconstruction output.

[0036] At least one reconstruction candidate that passes the logical decision is selected as the final reconstruction output for the key region. If multiple reconstruction candidates pass the logical decision simultaneously, the candidate with smaller cross-frame variation and higher structural stability is selected first. If none of the reconstruction candidates pass the logical decision, a preset backoff strategy is triggered. The key region is temporally compensated based on the region corresponding to the key region in the reference frame to generate the reconstruction result. Transition suppression processing is performed at the boundary of the key region to avoid obvious discontinuities between the reconstruction result and the surrounding region at the edge.

[0037] The key region output is fused with the non-key region in the current frame to be reconstructed. A transition region is set at the boundary between the key and non-key regions, and pixels within this transition region are weighted and fused according to the weights of the key and non-key region outputs to achieve a smooth and natural image transition, ultimately yielding the reconstructed frame of the current frame to be reconstructed. Subsequently, the processing steps of obtaining the frame to be reconstructed and the reference frame, determining the key region, generating reconstruction candidates, performing logical decisions, and fusing the reconstruction results are repeated for subsequent frames in the video frame sequence, outputting the reconstructed video frames frame by frame to form a complete reconstructed video sequence.

[0038] Combined with appendix Figure 2 For each candidate region, a key score is calculated. This key score reflects both the degree to which the region is affected by packet loss and its perceived sensitivity during viewing. The degree of packet loss impact characterizes the number of missing pixels within the region, the continuity of the missing pixel distribution, and the extent to which packet loss disrupts the integrity of the region's content. Perceived viewing sensitivity characterizes the importance of the region's content to the user's viewing experience. Perceived viewing sensitivity is determined based on whether the region contains at least one of the following: a face region, a text region, or a main target area, thus giving regions containing key visual information a higher weight in the score.

[0039] After completing the criticality scoring of each region, all regions whose scores meet the preset conditions are aggregated as candidates for critical regions, and the overall size of the critical regions is constrained by a pre-set computing budget. When the total area of ​​the critical regions exceeds the area limit corresponding to the computing budget, the candidates for critical regions are sorted from high to low according to their criticality scores, and regions with higher scores are selected as the final critical regions in turn, until the total area of ​​the selected critical regions does not exceed the area limit.

[0040] When multiple reconstruction candidate results generated for a key region fail to meet the preset logical decision conditions, a preset backoff strategy is triggered to generate a reconstruction result for the key region. In the backoff strategy, a region corresponding to the spatial location of the key region is obtained from at least one temporally adjacent reference frame. Temporal compensation processing is then performed based on this corresponding region to align the content in the reference frame to the key region position in the current frame to be reconstructed, thereby generating a backoff reconstruction result. Even when the reconstruction neural network cannot provide reliable candidates, it can still utilize effective information in the temporal dimension to recover the basic structure and content of the key region.

[0041] After generating the reconstructed results, transition suppression processing is performed at the boundaries of key regions. This processing weakens abrupt changes in brightness, chroma, or texture between the key regions and surrounding areas, preventing noticeable artifacts at the boundaries and ensuring the reconstructed results meet the requirements for boundary transition continuity. Subsequently, when blending key and non-key regions, a transition band is set at the boundaries of the key regions. Within this transition band, pixels are weighted and blended according to the weights corresponding to the key and non-key regions, resulting in a smooth and natural spatial transition in the image.

[0042] Reconstruction intensity is used to characterize the degree of enhancement of detail information and the degree of constraint on temporal consistency during the reconstruction process. Different reconstruction intensities are reflected at least in different detail enhancement intensities or different temporal constraint intensities, which makes each reconstruction candidate make different trade-offs between the degree of texture detail preservation and temporal stability, providing multiple reconstruction schemes for subsequent selection.

[0043] Different reconstruction intensities are achieved by configuring at least two sets of preset inference parameters for the reconstruction neural network. These inference parameters include at least a detail enhancement intensity parameter and a temporal constraint intensity parameter. The detail enhancement intensity parameter controls the enhancement magnitude of high-frequency textures and details in the reconstruction result, while the temporal constraint intensity parameter controls the degree of change in the reconstruction result between adjacent frames. The values ​​of the inference parameters are determined based on the size of the missing range indicated by the missing marker and the availability of adjacent reference frames. When the missing range is large or the availability of reference frames is low, the detail enhancement intensity is reduced and the temporal constraint intensity is increased to ensure structural stability. When the missing range is small and the availability of reference frames is high, the detail enhancement intensity is increased to improve image clarity.

[0044] After generating multiple reconstruction candidate results, a logical decision is made on each reconstruction candidate result. In the cross-frame change consistency judgment, by comparing the cross-frame change magnitude of each reconstruction candidate relative to the reference frame in the key region, the reconstruction candidate with the smallest cross-frame change magnitude and which also passes the boundary transition continuity judgment is selected as the final output of the key region.

[0045] When assessing the consistency of cross-frame changes in the reconstruction candidate results, a reference frame, either the previous frame or at least one temporally adjacent frame, is obtained. The region spatially corresponding to the key region of the current frame to be reconstructed is extracted from the reference frame. Subsequently, the corresponding region of the reference frame is aligned based on motion information, mapping it to the coordinate space of the current frame to eliminate positional deviations caused by object or camera motion. After alignment, the difference between the aligned reference region and each reconstruction candidate within the key region is calculated to quantify the degree of change of each reconstruction candidate in the temporal dimension, thereby obtaining the corresponding cross-frame change magnitude.

[0046] For the first The cross-frame variation magnitude of each reconstruction candidate is denoted as The calculation method is as follows:

[0047]

[0048] in, Represents the set of pixels within the key region. This indicates the number of pixels contained in the pixel set. This indicates the pixel position within the critical area. Indicates the first frame in the current frame to be reconstructed. One reconstruction candidate at pixel location The pixel value or pixel vector at that location. This indicates the region in the previous frame or reference frame that corresponds to the key region. This represents a motion-based alignment operator used to align corresponding regions in a reference frame to the current frame to be reconstructed, enabling comparison between the two within the same coordinate space. (Symbol) This represents the gradient operator, used to extract spatial variation features of pixels to characterize boundary and texture information and enhance constraints on structural consistency.

[0049] In the calculation of dissimilarity, the function Let be the penalty function, where This is a preset positive number used to avoid numerical instability when the gradient is zero, and to suppress abnormal difference values, thereby improving the robustness of cross-frame variation calculation. Parameter The preset weighting coefficients are used to balance the influence of pixel difference terms and gradient difference terms on the overall cross-frame variation range, so that the cross-frame variation range reflects both brightness or color changes and boundary and texture changes.

[0050] After calculating the cross-frame change magnitude for each reconstruction candidate, a stability threshold is introduced into the cross-frame change consistency judgment in the logical decision. This threshold is set when the cross-frame change magnitude corresponding to a given reconstruction candidate... When the value is below the stability threshold, the reconstruction candidate is considered to have good stability in the time dimension. Under the premise of satisfying the boundary transition continuity judgment, the reconstruction intensity is given priority to be the same as that of the candidate selected in the previous frame.

[0051] Combined with appendix Figure 3 The process involves extracting regions spatially corresponding to the key regions of the current frame to be reconstructed from at least one temporally adjacent reference frame, and calculating or directly acquiring motion information for these corresponding regions. This motion information characterizes the temporal displacement relationship between the reference frame and the current frame to be reconstructed. Subsequently, alignment processing is performed on the corresponding regions in the reference frames based on the motion information, mapping them to the key region positions of the current frame to be reconstructed. Missing pixels in the key regions are then compensated using reliable temporal content from the reference frames to generate the backtracking reconstruction result.

[0052] When a critical region corresponds to multiple available reference frame regions on the timeline, the regions corresponding to the multiple reference frames are filtered, and the region corresponding to the reference frame with the smallest time interval to the current frame to be reconstructed is selected first to reduce the risk of content changes due to excessive time span; when the time intervals are the same or similar, the proportion of available pixels in the regions corresponding to each reference frame is compared, and the region corresponding to the reference frame with the highest proportion of available pixels is selected first to perform timing compensation.

[0053] To address the visual discontinuity between the key region and its surrounding areas at the spatial boundary, transition suppression processing is applied to the regressed reconstruction results. This transition suppression processing is performed within a preset width extending inwards or outwards from the key region boundary. The preset width defines the spatial area where boundary suppression needs to be performed, avoiding unnecessary impact on the effective details within the key region.

[0054] Within a preset width range, boundary artifact suppression is performed on the reconstructed results. Boundary artifact suppression includes attenuation of high-frequency detail components at the boundaries to reduce edge ringing and texture abruptness caused by reconstruction uncertainties or timing compensation errors. Simultaneously, brightness and chromaticity variations on both sides of the boundary are smoothed to maintain continuity in brightness and color transitions between critical and non-critical areas, eliminating obvious seams. The preset width is adjusted based on the packet loss range indicated by the missing data marker. When the packet loss range is large, the preset width is appropriately increased to enhance the smoothness of the boundary transition; when the packet loss range is small, the preset width is correspondingly decreased to avoid over-smoothing leading to detail loss.

[0055] Motion information includes at least motion vector information obtained directly from the video decoder and optical flow information calculated from adjacent video frames. Motion vectors reflect block-level or macroblock-level motion relationships obtained during the encoding stage and have the characteristics of high stability and low computational overhead. Optical flow information reflects pixel-level motion changes and can describe local motion details more precisely. The two types of motion information can be used alone or in combination to complement each other's advantages.

[0056] When both motion vectors and optical flow information are available, the motion vectors are used first to perform coarse alignment on the corresponding region of the reference frame. This ensures that the overall position of the reference frame region is basically consistent with the key region of the current frame to be reconstructed, quickly eliminating large-scale displacement errors. After coarse alignment, fine alignment is then performed on the coarse alignment result based on the optical flow information. This finely adjusts the local pixel positions to compensate for subtle deviations caused by complex motion or non-rigid changes, generating a more accurate backtracking reconstruction result.

[0057] The credibility determination includes at least alignment residual consistency judgment and boundary motion continuity judgment. The alignment residual consistency judgment is used to evaluate whether the residual distribution between the reference region after optical flow alignment and the current frame to be reconstructed is stable in the key region, and to determine whether there are local anomalies in the optical flow estimation. The boundary motion continuity judgment is used to evaluate whether the motion changes at the boundary of the key region are smooth and coherent, so as to prevent abrupt motion at the boundary of the region.

[0058] When optical flow information fails the aforementioned confidence assessment within a critical region, fine alignment based on optical flow is not performed. Instead, a coarse alignment result directly based on motion vectors is used as the regression reconstruction result, prioritizing the overall structural and temporal stability. Alternatively, if only some sub-regions within the critical region fail the confidence assessment, fine alignment is performed on these sub-regions using neighborhood-smoothed optical flow substitution values. This allows the motion information of the sub-regions to be constrained by their surrounding reliable regions, completing local alignment without introducing significant errors.

[0059] Example 1:

[0060] In an online video conferencing scenario, the video resolution is 1920×1080, and the frame rate is 30 frames per second. Due to network fluctuations, data loss occurs at frame 125, resulting in approximately 18% pixel data loss. The system obtains the current frame to be reconstructed and the previous frame as reference frames, and retrieves missing pixel information from the decoding end to identify the missing locations. Based on the missing pixel information, the system divides the current frame into multiple fixed-size regions, each 120×120 pixels, and performs statistical analysis on the impact of packet loss in each region. In one region, the missing pixel percentage reaches 35%, while in another background region, the missing pixel percentage is only 8%. Figure 4 As shown.

[0061] In this meeting scenario, through object detection and region semantic analysis, the system detected a face region located in the center of the image. This region covers approximately 260,000 pixels and contains clear facial contours and expression information, thus it was identified as a face region. The system assigned a high viewing sensitivity weight of 0.9 to this face region; while the conference table background area, which does not contain faces, text, or prominent subjects, was assigned a lower viewing sensitivity weight of 0.2. The system comprehensively calculates the impact of packet loss within the region and the viewing sensitivity, resulting in the following criticality score for the face region:

[0062]

[0063] The key scores for the background area are:

[0064]

[0065] Based on a preset criticality scoring threshold of 0.1, only facial areas and a small number of screen-shared areas containing text content were selected as critical areas.

[0066] The system's computational budget was set to cover a critical area of ​​25% of the total image size, approximately 518,400 pixels. In the initial screening phase, the total area of ​​the selected critical areas was 620,000 pixels, exceeding the computational budget limit. Based on this, the system sorted the critical areas according to their criticality scores from highest to lowest, prioritizing the retention of face areas (0.315) and text areas (0.18), while discarding lower-scoring, less important areas. This ensured that the total area of ​​the critical areas was ultimately kept below 490,000 pixels, prioritizing the reconstruction of areas with the greatest impact on the viewing experience within the constraints of limited computing power.

[0067] In the subsequent reconstruction process, the reconstruction neural network generated two reconstruction candidate results for key regions. However, under extreme packet loss conditions, due to the continuous missing area exceeding 60% within the face region, both reconstruction candidates failed the logical decision-making in terms of boundary transition continuity and cross-frame change consistency. The system then triggered a backtracking strategy, extracting the region corresponding to the face area from the previous frame and performing temporal compensation based on motion vectors and optical flow information. Setting the usable pixel ratio of the face region in the previous frame to 92% and the time interval between the previous frame and the current frame to 33ms, the system aligned this region to the current frame position to generate a backtracked reconstruction result, restoring the basic contour and expression structure of the face.

[0068] At the boundary of the reconstructed results, the system performs approximately 30% attenuation on high-frequency detail components and applies smoothing constraints to brightness and chromaticity variations, ensuring visual continuity between the key region and the surrounding background. Subsequently, a transition band is established at the boundary of the key region, and fusion calculations are performed within this band using a weight of 0.7 for the key region and 0.3 for the non-key region. The brightness value of a pixel is obtained by weighted fusion of 180 from the key region result and 160 from the non-key region result.

[0069]

[0070] Effectively eliminate boundary seam phenomenon, such as Figure 4 As shown in the image.

[0071] After the system completes the key region screening, it initiates neural network-based reconstruction processing for the facial key region located in the center of the image. Since the packet loss rate in this facial region is approximately 35%, and the usable pixel rate of the corresponding region in adjacent reference frames is 92%, the system determines that this key region needs to recover a significant amount of detail while also maintaining temporal stability. Therefore, the reconstruction neural network generates at least two reconstruction candidate results using two different sets of preset inference parameters under the same input conditions, such as... Figure 5 As shown.

[0072] The first set of inference parameters corresponds to a higher detail enhancement intensity, with a detail enhancement intensity parameter set to 0.8 and a temporal constraint intensity parameter set to 0.3, used to restore the texture details and clarity of the face region. The second set of inference parameters corresponds to a higher temporal constraint intensity, with a detail enhancement intensity parameter set to 0.4 and a temporal constraint intensity parameter set to 0.7, used to prioritize ensuring the stability of the reconstruction results between adjacent frames.

[0073] After generating two reconstruction candidates, the system performs logical adjudication on each candidate. In the cross-frame change consistency judgment, the system aligns the selected face region from the previous frame with the two reconstruction candidates in the current frame and calculates the difference. The cross-frame change amplitude of the first reconstruction candidate is 0.28, and that of the second reconstruction candidate is 0.14. Subsequently, the system detects the transition continuity between the two at the boundary of the key region. The detection results show that both reconstruction candidates meet the boundary transition continuity judgment condition. Based on the principle of minimizing the cross-frame change amplitude, the system selects the second reconstruction candidate as the final reconstruction output for the key region.

[0074] In subsequent video frame processing, as the network gradually stabilized, the proportion of missing face regions decreased to 12%. The system accordingly adjusted the reconstruction intensity parameters, increasing the detail enhancement intensity parameter to 0.6 while decreasing the temporal constraint intensity parameter to 0.4, to further improve image clarity while ensuring temporal stability. Figure 5 The changes in the mid-parameter trajectory are shown.

[0075] After generating multiple reconstruction candidates for the facial key regions in frame 126, the system needs to quantitatively evaluate the temporal stability of each candidate to determine the most suitable reconstruction result for the current frame output. The set of pixels corresponding to the facial key regions is denoted as... Its area is Each pixel. The system selects frame 125 as the reference frame, and records the region in the reference frame corresponding to the key facial region as... An alignment operator is constructed based on the motion vectors and optical flow information provided by the decoding end. Align the reference region to the coordinate space of frame 126 to obtain the aligned reference region. This is to eliminate positional deviations caused by slight head movements.

[0076] In frame 126, the reconstruction neural network generated two reconstruction candidates, denoted as the first and second reconstruction candidates, at pixel positions... The pixel values ​​at each location are denoted as follows: and The system then calculates the cross-frame variation magnitude for each reconstruction candidate. The calculation formula is as follows:

[0077]

[0078] The penalty function is defined as follows:

[0079]

[0080] System retrieves:

[0081]

[0082] To ensure numerical stability, the weighting coefficients are set as follows:

[0083]

[0084] Used to balance the effects of pixel difference terms and gradient difference terms.

[0085] During the calculation process, the system calculates the average absolute value of the pixel difference between the first reconstruction candidate and the aligned reference region within the key region as follows:

[0086]

[0087] The corresponding average gradient difference is:

[0088]

[0089] Substituting the penalty function and weighting, we get:

[0090]

[0091] Similarly, for the second reconstruction candidate, the average pixel difference is 0.11 and the average gradient difference is 0.09. Substituting these values ​​into the above formula yields:

[0092]

[0093] The system pre-sets the stability threshold for cross-frame variation consistency as follows:

[0094]

[0095] The comparison shows the cross-frame variation of the first reconstruction candidate:

[0096]

[0097] The cross-frame variation of the second reconstruction candidate is above the stability threshold:

[0098]

[0099] The value being below the stability threshold indicates that the second reconstruction candidate maintains a higher consistency with the previous frame in the temporal dimension.

[0100] Therefore, under the premise of satisfying the boundary transition continuity judgment, the system selects the second reconstruction candidate as the final output of the key facial region in frame 126. Meanwhile, because:

[0101]

[0102] The system prioritizes maintaining the selected reconstruction intensity parameter combination from the previous frame, ensuring that the texture details and brightness changes in the face region remain consistent between frames 125 and 126, effectively suppressing flickering between consecutive frames. Figure 6 As shown.

[0103] Example 2:

[0104] Based on Example 1, such as Figure 7As shown, the system obtains multiple reference frames that are temporally adjacent to the current frame to be reconstructed from the decoding end and the video buffer. These reference frames include those that have been fully decoded before frames 123, 124, and 125. The system then locates the region corresponding to the spatial position of the current key facial region within these reference frames. For each region corresponding to a reference frame, the system calculates the percentage of usable pixels. The usable pixel percentage for the region corresponding to frame 123 is 88%, for frame 124 it is 92%, and for the region corresponding to the previous complete frame before frame 125 it is 95%. Simultaneously, the system records the time interval between each reference frame and the current frame to be reconstructed. The time interval for frame 123 is 66ms, for frame 124 it is 33ms, and for frame 125 it is 33ms.

[0105] After obtaining the above information, the system filters according to a priority rule that combines the time interval from small to large and the available pixel ratio from high to low. Figure 7 The two-stage selection logic is intuitively illustrated using a ranking evolution approach. In the first stage, candidate reference frames are sorted according to time intervals, with the 124th and 125th frames, which have the smallest time intervals, retained for the next stage. In the second stage, the usable pixel ratio is further compared only among candidate frames with the same time interval. Since both the 124th and 125th frames have a time interval of 33ms, but the usable pixel ratio of the corresponding area of ​​the 125th frame (95%) is higher than that of the 124th frame (92%), the system ultimately selects the 125th frame as the reference frame source for timing compensation, maximizing the amount of usable information while reducing structural errors caused by timing offsets.

[0106] After selecting a reference frame, the system calculates or acquires motion information for the regions in the reference frame that correspond to the key facial regions. The system prioritizes reading the motion vector information provided by the decoder to describe the overall translation trend of the facial region between adjacent frames, and combines it with the optical flow information calculated based on adjacent frames to describe the subtle non-rigid movements of the local areas of the eyes and mouth.

[0107] After completing the alignment process, the system outputs the aligned reference region as the back-reconstruction result to fill in the missing pixels of key facial regions in the current frame to be reconstructed. Since the selected reference frame has a small temporal interval with the current frame and a high proportion of available pixels, the back-reconstruction result can better maintain the continuity of facial contours, facial feature positions, and expression changes, and still maintain basic visual acceptability under extreme packet loss conditions.

[0108] like Figure 8As shown, after temporal compensation, the internal contours and facial expression structures of the key facial regions have been largely restored. However, due to differences in generation methods and information sources between this region and the surrounding background region, high-frequency noise accumulation and brightness abrupt changes are prone to occur at the boundaries of the key regions. The system determines the preset width of the key region boundaries based on the packet loss range recorded in the missing identifier. The continuous missing range of the face region accounts for approximately 60% of the region's area. Based on this, the system sets the preset width to 12 pixels and performs transition suppression processing within this width range.

[0109] The system attenuates high-frequency detail components at the boundary. By applying an attenuation coefficient of 0.7 to the high-frequency components, the intensity of fine textures and noise at the boundary location is reduced by approximately 30%. Figure 8 As shown in the energy distribution of mid-to-high frequency artifacts, at a certain boundary pixel, the amplitude of the mid-to-high frequency component in the original backtracking reconstruction result is 20, which is adjusted to 14 after attenuation, effectively reducing the fine artifacts introduced by time-series compensation and region switching.

[0110] The brightness value of a boundary pixel within a critical region is 178, while the brightness value of the corresponding pixel in an adjacent non-critical region is 162. The system introduces a smooth interpolation method within the transition zone to ensure that the brightness value transitions continuously within the boundary width according to a linear interpolation relationship. For example... Figure 8 As shown, at a position 6 pixels from the boundary of the key area, the brightness value is adjusted to 170 according to a linear transition relationship to avoid abrupt brightness jumps. For the chroma component, the system adopts the same smoothing strategy to keep color changes spatially consistent and prevent obvious color banding.

[0111] When the packet loss range is small, the system reduces the preset width accordingly. When the packet loss ratio in the face area drops to 12% in subsequent frames, the system reduces the preset width to 5 pixels and performs transition suppression processing only within a narrower boundary range, so as to preserve the detail clarity of key areas while ensuring visual continuity.

[0112] like Figure 9 As shown, the system obtains motion vector information corresponding to the key facial regions in frames 125 and 126 from the decoding end. The motion vectors describe the overall displacement of the region in macroblocks. In this conference scene, the facial region is shifted approximately 2 pixels to the right and 1 pixel downwards. The system uses this motion vector information to perform coarse alignment processing on the regions corresponding to the key facial regions in the reference frame, ensuring that the reference region is spatially consistent with the key regions in the current frame to be reconstructed. This quickly eliminates large-scale displacement errors and avoids estimation instability caused by optical flow directly acting on unaligned regions.

[0113] Optical flow information describes local motion changes at the pixel level, such as Figure 9As shown in the streamline distribution, the system detected non-rigid motion changes ranging from 0.3 to 0.6 pixels in the eye and mouth regions. By introducing optical flow fine alignment processing on top of the coarse alignment results, the system can more accurately recover facial expression changes and local structural details, improving the local detail representation of the backtracking reconstruction results without introducing significant structural deviations.

[0114] The reliability assessment includes at least two aspects: alignment residual consistency and boundary motion continuity. The system statistically analyzes the residual distribution between the reference region and the current frame to be reconstructed within the key region after optical flow alignment. When the mean residual is stable at 0.12 and the variance is less than 0.03, the optical flow information in that region is considered to have high reliability. Simultaneously, the system detects motion changes at the boundaries of the key region. When the difference in motion amplitude between the inside and outside of the boundary is less than 0.5 pixels, the boundary motion continuity is considered to meet the requirements.

[0115] When optical flow information fails the reliability assessment in a critical region, the system immediately halts fine alignment processing based on optical flow and uses only coarse alignment results based on motion vectors as the back-reconstruction result to prioritize overall structural stability. In one frame, rapid head turning caused the mean local optical flow estimation residual to rise to 0.35 and the boundary motion difference to reach 1.2 pixels. The system determined that the optical flow information was unreliable and directly used the coarse alignment result to complete the back-reconstruction, avoiding stretching or misalignment of the facial structure.

[0116] When optical flow information fails the confidence assessment only in certain sub-regions within the critical region, the system performs fine alignment processing on these sub-regions using neighborhood-smoothed optical flow substitution values. For example... Figure 9 As shown, for high-contrast regions at the edge of the glasses, the system uses the mean of the optical flow vectors in its surrounding 5×5 pixel neighborhood as a substitute value, so that the motion information of this sub-region is constrained by the adjacent reliable regions, thereby completing local alignment without introducing significant errors.

[0117] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for reconstructing key regions in packet-loss video based on neural networks, characterized in that... include: Obtain the current frame to be reconstructed, at least one temporally adjacent reference frame, and the missing identifier; Based on the missing identifier and the reference frame, key scores are performed on each region of the frame to be reconstructed, and key regions whose scores meet the preset threshold are selected. The available pixels in the key area, the temporal context of the corresponding area of ​​the reference frame, and the missing identifier are input into the reconstruction neural network, which outputs at least two reconstruction candidates. The candidates are logically adjudicated, and the adjudication includes the judgment of boundary transition continuity, cross-frame change consistency and main structure integrity. Those that meet the preset threshold are passed. Candidates that pass the adjudication are selected as key regions for output. If none pass, a reconstruction result is generated according to the backoff strategy. The key region output is fused with the non-key region to obtain the reconstructed frame. The acquisition, determination, generation, adjudication and fusion steps are repeated for subsequent frames to be reconstructed in the video frame sequence to output the reconstructed video sequence.

2. The method for reconstructing key regions of packet-lost video based on neural networks according to claim 1, characterized in that... The criticality score reflects both the impact of packet loss and the sensitivity of viewing perception. The sensitivity of viewing perception is determined based on at least one of the face region, text region, or main target region. When the total area of ​​the critical region exceeds the area limit corresponding to the preset calculation budget, the critical regions are selected from high to low according to the criticality score until the area limit is met.

3. The method for reconstructing key regions of packet-lost video based on neural networks according to claim 1, characterized in that... The at least two reconstruction candidates are generated by the reconstruction neural network with different reconstruction intensities, the different reconstruction intensities including at least different detail enhancement intensities or different temporal constraint intensities; the logical decision selects the reconstruction candidate with the smallest cross-frame change amplitude and which passes the boundary transition continuity judgment as the key region output in the cross-frame change consistency judgment.

4. The method for reconstructing key regions of packet-loss video based on neural networks according to claim 1, characterized in that... When all reconstruction candidates fail the logical decision, the backoff strategy includes: performing temporal compensation on the key region based on the region corresponding to the reference frame to generate a backoff reconstruction result, and performing transition suppression processing on the backoff reconstruction result at the boundary of the key region to meet the boundary transition continuity judgment; the fusion sets a transition band at the boundary of the key region, and performs fusion within the transition band according to the weights of the key region output and the non-key region.

5. The method for reconstructing key regions of packet-lost video based on neural networks according to claim 3, characterized in that... The different reconstruction intensities are generated by the reconstruction neural network using at least two sets of preset inference parameters under the same input conditions. The preset inference parameters include at least a detail enhancement intensity parameter and a temporal constraint intensity parameter. The values ​​of the detail enhancement intensity parameter and the temporal constraint intensity parameter are determined based on the missing range indicated by the missing identifier and the availability of adjacent reference frames.

6. The method for reconstructing key regions of packet-lost video based on neural networks according to claim 3, characterized in that... The cross-frame change amplitude is obtained by: aligning the region corresponding to the key region in the previous frame or reference frame to the current frame to be reconstructed based on motion information, and calculating the difference between the alignment result and each reconstruction candidate in the key region to obtain the corresponding cross-frame change amplitude; and the logical decision introduces a stability threshold in the cross-frame change consistency judgment, and when the cross-frame change amplitude is lower than the stability threshold, the reconstruction intensity corresponding to the selected candidate in the previous frame is maintained first to suppress flicker.

7. The method for reconstructing key regions of packet-lost video based on neural networks according to claim 4, characterized in that... The timing compensation includes: calculating or obtaining motion information based on the region corresponding to the key region in the reference frame, and aligning the corresponding region to the key region of the current frame to be reconstructed based on the motion information to generate the backtracking reconstruction result; wherein, when there are multiple reference frame corresponding regions in the key region, the timing compensation is performed on the reference frame corresponding region with the smallest time interval to the current frame to be reconstructed and the highest available pixel ratio.

8. The method for reconstructing key regions of packet-lost video based on neural networks according to claim 4, characterized in that... The transition suppression process includes performing boundary artifact suppression on the backtracking reconstruction results within a preset width range of the critical region boundary. The boundary artifact suppression includes at least: attenuating the high-frequency detail components at the boundary and maintaining the continuity of brightness and chromaticity changes on both sides of the boundary; and the preset width is adjusted according to the size of the packet loss range indicated by the missing identifier.

9. The method for reconstructing key regions of packet-loss video based on neural networks according to claim 7, characterized in that... The motion information includes motion vectors obtained from the decoding end, optical flow information calculated from adjacent frames, or a combination of the two. When both motion vectors and optical flow information are available, coarse alignment is first performed based on the motion vectors, and then fine alignment is performed based on the optical flow information to generate the backtracking reconstruction result.

10. The method for reconstructing key regions of packet-loss video based on neural networks according to claim 9, characterized in that... The optical flow information is evaluated for credibility in key regions. The credibility evaluation includes at least alignment residual consistency evaluation and boundary motion continuity evaluation. When the optical flow information fails the credibility evaluation, the fine alignment is stopped and only the coarse alignment result based on motion vector is used as the backtracking reconstruction result. Alternatively, fine alignment is performed on the sub-regions that fail the credibility evaluation using neighborhood smoothing optical flow substitution values.

Images