Face video quality enhancement and deblurring processing method and device

Abstract

The present disclosure relates to a face video quality enhancement and de-shake processing method and device, and relates to the technical field of video processing. The method comprises: acquiring an initial video sequence containing multiple video frames, wherein the initial video sequence is a video sequence with pixel-level distortion and / or time sequence jitter caused by unstable operation of a collection device; performing enhancement processing on the details of the initial video sequence to generate an enhanced frame; determining the operation trajectory of the collection device that collected the initial video sequence based on the enhanced frame; performing smoothing processing on the operation offset data related to the collection device in the enhanced frame based on the motion trajectory to generate a stable frame; performing completion processing on the missing data in the stable frame to generate a full-view frame; integrating the full-view frame to generate a target video sequence, wherein the target video sequence is a video sequence with complete view and stable picture operation. The present disclosure solves the problem that face video quality enhancement and de-shake cannot be jointly processed in the prior art.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a method and apparatus for enhancing and de-shaking facial video quality. Background Technology

[0002] In recent years, with the rapid development of social media applications, facial content has undergone a significant shift from static images to dynamic video formats. This evolution has driven the continuous development of video quality enhancement and stabilization technologies. In the field of video quality enhancement, technical approaches are mainly divided into single-frame and multi-frame methods. Single-frame methods reduce compression artifacts by optimizing network structure or residual learning, while multi-frame methods utilize inter-frame temporal dependencies and improve reconstruction quality through cross-frame alignment and feature fusion. Some enhancement methods for facial videos also introduce prior information such as facial parsing to optimize facial detail restoration. In the field of video stabilization, traditional methods achieve stabilization based on feature point tracking, motion trajectory smoothing, and frame distortion. Deep learning-based methods are gradually becoming mainstream, directly learning motion representations and smoothing processes through network training to achieve end-to-end image stabilization.

[0003] Existing technologies have significant limitations: most video quality enhancement methods assume smooth motion transitions, making it difficult to generate satisfactory visual results when faced with shaky and low-quality facial videos. Single-frame methods also generally ignore inter-frame temporal information. Video stabilization methods experience significant performance degradation when facial details are severely degraded. Traditional stabilization methods have limited stability in complex motion or fast-shaking scenarios and often rely on additional hardware sensors to obtain 3D motion information, limiting their universality. More importantly, existing quality enhancement and stabilization methods typically develop along two independent paths, employing a task decoupling design, which cannot effectively address the intertwining of pixel-level distortion and motion shaking in real-world scenarios.

[0004] In summary, the existing technology cannot jointly process facial video quality enhancement and shaking reduction. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a method and apparatus for enhancing and stabilizing facial video quality.

[0006] This disclosure provides a method for enhancing and de-shaking facial video quality, the method comprising: Obtain an initial video sequence containing multiple video frames, wherein the initial video sequence is a video sequence with pixel-level distortion and / or timing jitter caused by unstable operation of the acquisition device; The details of the initial video sequence are enhanced to generate enhanced frames; Based on the enhanced frame, determine the operating trajectory of the acquisition device that acquired the initial video sequence; Based on the motion trajectory, the running offset data related to the acquisition device in the enhanced frame is smoothed to generate a stable frame; The missing data in the stable frame is filled in to generate a full-view frame; The full field-of-view frames are integrated to generate a target video sequence, which is a video sequence with a complete field of view and stable image operation.

[0007] Optionally, the step of enhancing the details of the initial video sequence to generate enhanced frames includes: Extract the current video frame of the initial video sequence, the facial key point features in the current video frame, and the enhanced stabilization result of the reference frame, where the reference frame is the frame preceding the current video frame; Based on global dynamic information, the reference frame and the current video frame are globally aligned. The global dynamic information is used to characterize the motion trends of multiple video frames in the initial video sequence related to the acquisition device. Based on the globally aligned reference frame, the video frames of the initial video sequence are reconstructed using an encoder-decoder architecture combined with a residual attention mechanism to generate enhanced frames.

[0008] Optionally, before reconstructing each video frame of the initial video sequence based on the globally aligned reference frame using an encoder-decoder architecture combined with a residual attention mechanism to generate the enhanced frame, the method further includes: The global dynamic information and the hybrid dynamic information are decoupled to obtain local dynamic information. The local dynamic information is used to characterize the motion differences or motion changes in local regions in the reference frame and the current video frame. The hybrid dynamic information is used to characterize the dynamic features associated with pixel-level details in the video frame. Based on the local dynamic information, local alignment and residual alignment processing are performed on the reference frame and the current video frame.

[0009] Optionally, the step of smoothing the motion offset data related to the acquisition device in the enhanced frame based on the motion trajectory to generate a stable frame includes: A distortion field is generated based on the motion trajectory, and the distortion field is used to characterize the set of geometric transformation parameters for geometric adjustment of the enhanced frame; Based on the distortion field, the running offset data related to the acquisition device in the enhanced frame is smoothed to generate the stable frame.

[0010] Optionally, the step of completing the missing data in the stable frame to generate a full-view frame includes: Based on the distortion field, an original mask is generated, which is used to characterize the distribution of valid content regions and blank missing regions in the stable frame. Multiple sampling points are selected at the boundary of the stable frame, and the position of the sampling points is corrected using the neighborhood information of each sampling point to obtain a coarse motion mask. A refined mask is generated by fusing the shared content region of the original mask of the stable frame and the coarse motion mask. Based on the refined mask, missing data in the stable frame is filled in to generate the full-view frame.

[0011] Optionally, before smoothing the running offset data related to the acquisition device in the enhanced frame based on the motion trajectory to generate a stable frame, the method further includes: adding a safety boundary to each video frame, the safety boundary being used to add an additional content area outside the original edge of the video frame.

[0012] Optionally, before integrating the full-view frames to generate the target video sequence, the method further includes: The full-view frame is cropped to the same frame size as the video frames in the initial video sequence.

[0013] A facial video quality enhancement and shaking reduction processing device, the device comprising: The acquisition module is used to acquire an initial video sequence containing multiple video frames, wherein the initial video sequence is a video sequence with pixel-level distortion and / or timing jitter caused by unstable operation of the acquisition device; The enhancement module is used to enhance the details of the initial video sequence and generate enhanced frames; The trajectory estimation module is used to determine the operating trajectory of the acquisition device that acquires the initial video sequence based on the enhanced frame; A smoothing module is used to smooth the running offset data related to the acquisition device in the enhanced frame based on the motion trajectory, and generate a stable frame; The completion module is used to complete the missing data in the stable frame and generate a full-view frame; The generation module is used to integrate the full field-of-view frames to generate a target video sequence, which is a video sequence with a complete field of view and stable image operation.

[0014] An electronic device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the face video quality enhancement and de-shaking processing method as described in any of the preceding claims.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the face video quality enhancement and de-shaking processing method as described in any of the preceding claims.

[0016] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: by sequentially performing detail enhancement, determination of the acquisition device's operating trajectory, smoothing of operating offset data, and completion of missing data, it can specifically solve the problems of pixel-level distortion and timing jitter caused by unstable acquisition devices in the initial video sequence. It not only effectively improves the detail quality and visual clarity of the video image, but also achieves a smooth transition of the video operating state. At the same time, by completing the missing data, it ensures the integrity of the field of view of the target video sequence, and finally generates a target video with high-quality visual effects, stable operating state, and complete field of view, improving the user's viewing experience. It can also provide high-quality video data support for subsequent face-related visual tasks (such as key point detection), and has a wide range of applications and strong practicality. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of the face video quality enhancement and shaking removal processing method described in the embodiments of this disclosure; Figure 2 This is a schematic diagram of the structure of a face video quality enhancement and stabilization network provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of a unified recurrent network for face video quality enhancement and shaking provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a face video quality enhancement and shaking removal processing device provided in an embodiment of this disclosure. Detailed Implementation

[0020] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0022] Figure 1 This is a schematic flowchart illustrating the face video quality enhancement and shaking reduction method according to an embodiment of this disclosure, as shown below. Figure 1 As shown, the method for enhancing and stabilizing facial video quality includes the following steps: S11. Obtain the initial video sequence containing multiple video frames.

[0023] The initial video sequence is a video sequence with pixel-level distortion and / or timing jitter caused by unstable operation of the acquisition equipment.

[0024] Specifically, firstly, based on the actual application scenarios of facial videos (such as social media shooting, daily selfies, etc.), video sequences with typical quality defects caused by non-professional shooting are selected. The core problems of these sequences are either pixel-level distortion (such as compression noise, blockiness, blur) or temporal jitter caused by unstable acquisition equipment (such as handheld shaking, mobile shooting), or both. Secondly, all video frames of the sequence are loaded through the video reading interface, preserving the original temporal relationship and basic attributes of the frames (such as size, frame rate), ensuring that the inter-frame temporal logic is not disordered during subsequent processing, and providing a prerequisite for cross-frame information interaction (such as dynamic information transmission, inter-frame alignment).

[0025] For example, a short video of a user's face taken while walking with a mobile phone may have frequent shaking of the face position between video frames due to unstable handholding (timing jitter), while the image may be blurred and blocky artifacts may appear in some areas due to network transmission compression (pixel-level distortion); another example is a video of a face taken in low light conditions, which has obvious image noise (pixel-level distortion), but the motion between frames is relatively stable. Both of these types of video sequences belong to the initial video sequences obtained in this step.

[0026] In this embodiment, the core input object of the processing task is clearly defined, and the face video sequence with pixel-level distortion and / or temporal jitter is locked. This provides a clear target carrier for the subsequent quality enhancement, jitter reduction, missing data completion and other full-process processing, ensuring that the subsequent technical means can specifically solve the core quality problems of videos in real-world scenarios, avoid resource waste or effect deviation caused by ambiguous processing scope, and lay the foundation for the final generation of high-quality and stable videos.

[0027] S12. Enhance the details of the initial video sequence to generate enhanced frames.

[0028] In this embodiment, the focus is on the pixel-level distortion problem of the initial video. By enhancing details, key facial details (such as facial texture and edge contours) are restored and strengthened, eliminating quality defects such as blur, noise, and blockiness, and improving the visual clarity and detail integrity of the video frames. At the same time, the generated enhanced frames contain richer high-frequency details in the pixel domain, providing high-quality dynamic information support for subsequent accurate estimation of the operating trajectory of the acquisition device, thus laying the groundwork for both "enhancing details" and "assisting in shaking".

[0029] Specifically, S12 above can be further refined into steps 121 to 123 as follows: Step 121: Extract the current video frame of the initial video sequence, the facial key point features in the current video frame, and the enhanced stabilization result of the reference frame.

[0030] The reference frame is the frame preceding the current video frame.

[0031] Specifically, firstly, the current video frame to be processed (e.g., frame t) is sequentially locked from the initial video sequence, while the enhanced and stabilized result of the previous frame (frame t-1) after pre-enhancement and stabilization processing is extracted as a reference frame, and cross-frame reference is constructed using the inter-frame temporal correlation. Secondly, the feature information of key facial positions is extracted from the current frame through a facial landmark detection algorithm to clarify the spatial distribution of the core facial region. Finally, the facial features of the current frame and the enhanced and stabilized result of the reference frame are extracted respectively through a shared feature extractor to ensure that the dimensions and representation logic of the two types of features are consistent, providing a matching feature basis for subsequent inter-frame alignment.

[0032] In this embodiment, the core data required for subsequent alignment and enhancement are captured, providing a basic carrier and key prior information for global alignment, local alignment and final frame reconstruction. The current frame is the direct object of enhancement processing, and facial key point features can lock the core facial region (such as facial features and contours) to ensure targeted enhancement. The enhancement stability results of the reference frame provide a high-quality cross-frame reference benchmark. The combination of the three avoids alignment deviation or enhancement distortion caused by missing data, laying a core data foundation for generating enhancement frames with clear details and coherent inter-frame connections.

[0033] Step 122: Based on global dynamic information, perform global alignment processing between the reference frame and the current video frame.

[0034] Among them, global dynamic information is used to characterize the motion trends between multiple video frames of the initial video sequence and related to the acquisition device.

[0035] Specifically, firstly, global dynamic information (low-frequency components in the motion domain) from the shaking stabilization network (LDS-Net) is acquired, which characterizes the overall motion trend of the acquisition device during the shooting process. Secondly, based on this global dynamic information, the features of the reference frame are initially geometrically adjusted (e.g., translation, rotation) through feature warping operations to simulate the global motion trajectory of the device, so that the features of the reference frame are initially adapted to the spatial position of the current frame. Subsequently, the facial features of the current frame are fused with the extracted facial keypoint features, and the three types of features are progressively fused through convolutional layers to predict the global offset. Finally, deformable convolution is used to optimize the features of the reference frame to obtain global alignment features that match the global position of the current frame, thus completing the global alignment process.

[0036] In this embodiment, global dynamic information is used to eliminate the overall positional offset between the reference frame and the current frame caused by the macroscopic movement of the acquisition device (such as translation, rotation, and scaling), ensuring that the two frames are consistent in global layout. This avoids the failure of subsequent local detail alignment due to excessive overall misalignment between frames. At the same time, it lays the foundation for global consistency for the subsequent application of hybrid dynamic information, and improves the inter-frame coherence and detail recovery accuracy of the enhanced frame.

[0037] Step 123: Based on the globally aligned reference frame, the encoder-decoder architecture combined with the residual attention mechanism is used to reconstruct each video frame of the initial video sequence to generate the enhanced frame.

[0038] Specifically, firstly, the features of the globally aligned reference frame and the current frame are collected and fused together to fully utilize the complementary information across frames (high-quality details of the reference frame and the original content of the current frame). Secondly, the fused features are extracted at multiple scales through an encoder architecture to gradually mine deep semantic features and shallow detail features, providing rich feature support for reconstruction. Subsequently, a residual attention mechanism is introduced to assign weights to the extracted features, strengthening the feature weights of core facial regions (such as facial features and skin texture) and weakening the interference from irrelevant regions such as the background. Finally, the processed features are progressively upsampled through a decoder architecture, and residual learning is used to compensate for the loss of detail during feature transmission, ultimately reconstructing an enhanced frame with clear details and no pixel distortion.

[0039] In one specific embodiment, before reconstructing each video frame of the initial video sequence using an encoder-decoder architecture combined with a residual attention mechanism based on a globally aligned reference frame to generate an enhanced frame, the method further includes: decoupling global dynamic information from hybrid dynamic information to obtain local dynamic information, which is used to characterize the motion differences or motion changes in local regions between the reference frame and the current video frame; and performing local alignment and residual alignment processing on the reference frame and the current video frame based on the local dynamic information.

[0040] Specifically, firstly, based on a two-way information interaction mechanism, hybrid dynamic information (high-frequency details in the pixel domain) from the enhancement task and global dynamic information from the dodge task are obtained. Through feature decoupling operations (separating high-frequency local dynamics and low-frequency global dynamics), local dynamic information (characterizing motion differences in local regions between frames) is obtained. Secondly, based on the local dynamic information, the offset of local feature alignment is predicted, and the features of the reference frame after global alignment and the features of the current frame are finely adjusted to correct the subtle misalignment of local regions (such as the corners of the eyes and mouth), thus completing the local alignment. Finally, a residual alignment module is constructed through convolutional layers and the LeakyReLU activation function to predict the residual feature alignment offset. The locally aligned features are further refined and corrected to eliminate residual small offsets, and the aligned features are used for subsequent reconstruction.

[0041] In this embodiment, through an efficient feature extraction and reconstruction architecture, the cross-frame feature information after global alignment is fully utilized to restore the pixel-level details of the video frame (such as eliminating blur, noise, and blockiness). At the same time, the residual attention mechanism can focus on the core facial region and avoid irrelevant background interference, ensuring that the enhanced frame is significantly better than the initial frame in terms of visual clarity and detail integrity. Furthermore, the inter-frame feature fusion makes the enhancement result more temporally coherent, providing high-quality hybrid dynamic information support for subsequent shaking tasks.

[0042] For example, Figure 2 This is a schematic diagram of a face video quality enhancement and de-shaking network provided in this embodiment. It enhances the details of the initial video sequence through a keypoint-guided progressive alignment and enhancement network (LPE-Net) to generate enhanced frames. The core objective of LPE-Net is to enhance the details of the initial video sequence through a progressive feature alignment mechanism. This network combines global dynamic information from the de-shaking network with prior knowledge of facial keypoints, constructing an alignment process with clear physical meaning in the pixel domain. Given the first... Frame input image Enhanced stabilization results compared to the previous frame First, facial features are obtained through a shared feature extractor. and At the same time from the input frame Extract 68 facial landmark features Global dynamics from the de-jitter network. As prior motion information, it is introduced to support the three-stage progressive alignment. The first stage is the global alignment stage, which incorporates features from the reference frame. According to global dynamics Perform initial warping and fuse current frame features. and key features Predict global offset : (1) in, This represents a feature warping operation based on optical flow. Facial features and keypoint features are progressively fused through convolutional layers. Global alignment features are obtained through deformable convolutions. : (2) and It uses feasible variable convolution and LeakyReLU activation functions. Based on global alignment, it introduces local dynamics. Fine-grained alignment is performed. Local dynamics are obtained by decoupling global dynamics and hybrid dynamics. (3) (4) in, The offset represents the alignment of local features. Features are aligned globally and locally. Finally, features are further refined through residual alignment. : (5) (6) in, It is the offset for residual feature alignment. It consists of convolutional layers and the LeakyReLU activation function. It's a splicing operation.

[0043] The reconstruction module fuses the alignment features of multiple reference frames and reconstructs the enhanced frame using an encoder-decoder architecture combined with a residual attention mechanism. (7) (8) In the formula above, and One or more convolutional layers It is the first One residual attention module (sampling factor is) ); , , This represents the number of residual modules with different sampling factors; It is the enhanced frame.

[0044] S13. Based on the enhanced frame, determine the operating trajectory of the acquisition device that acquires the initial video sequence.

[0045] Specifically, firstly, hybrid dynamic information (high-frequency details in the pixel domain, including fine facial textures and dynamic features related to local pixel changes) is extracted from the enhanced frames. This hybrid dynamic information is used to characterize the dynamic features associated with pixel-level details in the video frames. This type of information is more recognizable due to detail enhancement and can reflect subtle motion differences between frames. Secondly, the hybrid dynamic information is input into the dynamic information estimation module. Combined with the coarsely estimated initial dynamic information, invalid interference information is eliminated through the local dynamic perception branch, along with upsampling, downsampling operations, and effective area mask screening, thereby accurately predicting the motion offset. Finally, based on the optimized dynamic information, the complete operating trajectory of the acquisition device during the shooting process (including irregular motion components caused by shaking) is reconstructed.

[0046] For example, the Local Dynamics Awareness (LDS-Net) stabilization network determines the complete trajectory of the acquisition device during shooting. LDS-Net is designed to smooth camera motion while preserving high-frequency detail information from the enhancement network. This network achieves joint modeling of pixel-level and motion-level features in the motion domain by introducing a local dynamics awareness mechanism. LDS-Net includes a dynamics information estimation module and a stabilization module.

[0047] In the dynamic information estimation module, to obtain more accurate camera motion estimation, a local dynamic perception branch is added on top of the traditional global dynamic estimation. This is based on the hybrid dynamic information obtained from the augmentation network. and global dynamic estimation Predict the offset using a convolutional network: (9) in, and These are upsampling and downsampling operations. It is a local dynamic prediction branch. It is based on the hybrid dynamic information obtained from the effective area mask. This is a rough estimate of the dynamic information. In the stabilization module, based on the above global dynamic information, the stabilization module generates a smooth camera trajectory through motion smoothing.

[0048] In this embodiment, this step relies on the high-quality details of the enhanced frame to capture the real motion state of the acquisition device, generate an accurate running trajectory, avoid trajectory estimation deviation caused by the lack of details in the original low-quality video, provide reliable motion data support for subsequent smoothing processing, ensure that the shaking operation can offset the inter-frame offset caused by device jitter, and improve the motion stability of the final video.

[0049] S14. Based on the motion trajectory, smooth the running offset data related to the acquisition device in the enhanced frame to generate a stable frame.

[0050] In this embodiment, irregular jitter components in the trajectory of the acquisition device are specifically filtered to achieve a smooth transition between video frames. At the same time, because the hybrid dynamic information of the enhanced frame is integrated, the high-quality facial details after enhancement are fully preserved during the jitter removal process, avoiding the problem of "loss of detail in jitter removal". This achieves the dual goals of "motion stability" and "visual clarity", improving the video viewing experience.

[0051] Specifically, S14 above can be further refined into steps 141 to 142 as follows: Step 141: Generate a twist field based on the motion trajectory.

[0052] The distortion field is used to characterize the set of geometric transformation parameters used to geometrically adjust the enhanced frame.

[0053] Specifically, firstly, the original motion trajectory of the acquisition device (including irregular jitter components) determined based on the enhanced frame in S13 is obtained. The trajectory is then smoothed using the LDS-Net de-jitter module to remove invalid jitter components such as sudden shaking and rapid rotation, while preserving the stable motion trend of the device, thus generating an optimized smooth motion trajectory. Secondly, based on the smoothed motion trajectory and combined with the pixel coordinate system of the video frame, the trajectory information is transformed into geometric transformation parameters (including adjustment rules such as translation, rotation, and scaling) for each pixel within the frame. Finally, the geometric transformation parameters of all pixels are integrated to form a complete distortion field, ensuring that each pixel can obtain accurate adjustment basis, thus realizing a logical closed loop of "trajectory smoothing → parameter transformation → distortion field generation".

[0054] In this embodiment, the motion trajectory of the acquisition device is transformed into a set of geometric transformation parameters that can be directly used for frame adjustment, providing core technical support for subsequent accurate offsetting of runtime offset and jitter reduction. The distortion field clarifies the adjustment direction and magnitude of each frame pixel, ensuring that the smoothing process is targeted and avoiding image distortion or incomplete jitter reduction caused by blind geometric adjustments. At the same time, it lays the foundation for collaborative work with the subsequent edge completion network (WVS-Net), ensuring the continuity of the jitter reduction and completion process.

[0055] Step 142: Based on the distortion field, smooth the running offset data related to the acquisition device in the enhanced frame to generate a stable frame.

[0056] Specifically, first, the distortion field generated in step 141 and the enhanced frame generated in step S12 are read to identify the operational offset data (such as overall face offset and background area displacement) caused by device jitter in each region of the enhanced frame. Second, the geometric transformation parameters in the distortion field are applied pixel by pixel to the enhanced frame. Following the logic of "pixel positioning → parameter matching → geometric adjustment", the position of each pixel is corrected to offset the operational offset related to the acquisition device. During this process, since the generation of the distortion field relies on the motion trajectory that integrates mixed dynamic information, the adjustment process will simultaneously adapt to the high-frequency details of the enhanced frame to avoid the details being destroyed by geometric transformation. Finally, all corrected pixels are integrated to generate a stable frame with smooth motion and complete details.

[0057] In this embodiment, by utilizing the geometric transformation parameters of the distortion field, the running offset caused by the jitter of the acquisition device in the enhanced frame is efficiently offset, achieving a smooth transition between frame motions. At the same time, because the generation of the distortion field integrates the mixed dynamic information (high-frequency details in the pixel domain) of the enhanced frame, the high-quality facial details (such as facial texture and edge contours) after enhancement are fully preserved during the shaking process, avoiding the problem of "loss of detail in shaking". This achieves the dual goals of "motion stability" and "visual clarity", significantly improving the perceptual quality of the video.

[0058] In one specific embodiment, before smoothing the motion offset data related to the acquisition device in the enhanced frame based on the motion trajectory to generate a stable frame, a safety boundary is added to each video frame. The safety boundary is used to add an additional content area outside the original edge of the video frame.

[0059] Specifically, first, the original dimensions (such as resolution and aspect ratio) of the video frames in the initial video sequence are read to determine the core content area (such as the central area where the face is located); second, based on the maximum amplitude of video jitter and the range of geometric transformations that the distortion field may bring, the required width of the safety boundary is calculated (to ensure sufficient coverage of edge loss that may be caused by distortion); finally, an additional content area is added outside the original edge of each video frame (in the four directions of up, down, left, and right) (which can be expanded based on the in-frame edge texture or filled with a similar background) to form an extended frame containing the safety boundary. This extended frame will serve as the basic input for subsequent distortion and smoothing processing to prevent the core content from being truncated due to distortion.

[0060] For example, continue to refer to Figure 2 The anti-shake module generates smooth camera trajectories through motion smoothing and calculates the corresponding distortion field. Stabilized video is generated by applying the estimated distortion field to shaky frames.

[0061] (10) The above formula describes the de-jitter process for frame t. For a distorted field, It's a function from the debouncing module. It is a stable frame.

[0062] S15. Complete the missing data in the stable frame to generate a full-view frame.

[0063] In this embodiment, the problem of boundary loss caused by frame distortion during the shaking process is solved. By filling in the missing areas, the visual integrity of the video frame is ensured, avoiding blank screens or structural deformation. This allows the final video to maintain motion stability and clear details while having a complete visual presentation effect, thus improving the video's practicality and viewing experience.

[0064] Specifically, S15 above can be further refined into steps 151 to 154 as follows: S151. Based on the distortion field, generate the original mask. The original mask is used to characterize the distribution of effective content areas and blank missing areas in the stable frame.

[0065] Specifically, firstly, the system receives the warp field sequence and the stable enhancement frame sequence from the Local Dynamic Information Aware De-shake Network (LDS-Net) to clarify the geometric transformation rules of each pixel contained in the warp field. Secondly, based on the transformation logic of the warp field, it reversely deduces which regions retain the effective content of the original video (without gaps due to warping) and which regions have meaningless blanks (i.e., missing regions) due to pixel offset and stretching after warping processing in the stable frame. Finally, the distribution of effective content regions and blank missing regions is recorded by distinguishing them using binary tags or grayscale values ​​to form an original mask, achieving preliminary locking of the target region for completion.

[0066] In this embodiment, the blank and missing areas and effective content areas generated by the anti-shake distortion operation in the stable frame are located by the distortion field. The generated original mask provides a basis for the division of the region for subsequent mask optimization and missing data completion. This avoids misoperation of effective content or omission of missing areas during the completion process, ensuring that the completion work has a clear goal orientation and laying the core foundation for the final generation of the full-view frame.

[0067] S152. Select multiple sampling points at the boundary of the stable frame, and use the neighborhood information of each sampling point to correct the position of the sampling point to obtain a coarse motion mask.

[0068] Specifically, firstly, focusing on the boundary region of the stable frame (the core region prone to structural deformation and missing parts due to jitter reduction distortion), multiple sampling points are evenly selected to cover key positions of the boundary (such as the midpoints and quarter points of the top, bottom, left, and right boundaries); secondly, key information of the neighborhood region of each sampling point is extracted (such as texture features of adjacent effective pixels, edge contour correlation, color distribution patterns, etc.), based on which it is determined whether the sampling point has a positional deviation due to structural deformation; then, the coordinates of the deviation sampling points are adjusted and corrected to return them to reasonable positions; finally, the central region of all corrected sampling points is filled and connected to form a coarse motion mask that can initially reflect the boundary structural deformation and missing parts.

[0069] In this embodiment, to address the structural deformation problem that may occur at the stable frame boundary due to jitter reduction, the marking accuracy of the mask for the region is optimized by sampling point correction. The generated coarse motion mask makes up for the limitation of the original mask, which only distinguishes between valid and missing regions and does not consider structural deformation. This provides a more accurate regional feature reference for the subsequent generation of the refined mask, and improves the accuracy of subsequent missing completion and deformation correction.

[0070] S153. Merge the shared content area of ​​the original mask of the stable frame and the coarse motion mask to generate a refined mask.

[0071] Specifically, firstly, the original mask and the coarse motion mask are read separately, and the marking information for different regions in the two types of masks is analyzed. The original mask clearly identifies the valid content and blank / missing regions, while the coarse motion mask supplements the markings for the boundary structural deformation regions. Secondly, shared content regions with consistent markings in both types of masks (i.e., regions marked as valid content by both types of masks) are selected and retained as core valid regions. Subsequently, the markings for structural deformation regions in the coarse motion mask and the markings for blank / missing regions in the original mask are integrated, and the marking differences between the two types of masks are verified and corrected (e.g., regions marked as valid in the original mask but deformed in the coarse mask are marked as such; the deformation markings in the coarse motion mask take precedence). Finally, a refined mask is formed that can distinguish between valid content regions, blank / missing regions, and structural deformation regions.

[0072] In this embodiment, by combining the advantages of the two types of masks, the generated refined mask retains the basic division of effective content and blank missing areas of the original mask, and integrates the accurate marking of boundary structural deformation areas of the coarse motion mask. This significantly improves the mask's accuracy in distinguishing regional attributes (effective content, blank missing areas, structural deformation), providing accurate positioning basis for subsequent missing data completion and structural deformation correction, and avoiding deviations and distortions in the completion process.

[0073] S154. Based on the thinned mask, the missing data in the stable frame is filled in to generate a full-view frame.

[0074] Specifically, firstly, based on a refined mask, the blank and deformed areas in the stable frame are clearly identified; secondly, an optimization model (such as a video restoration optimization model) is invoked to fully utilize the spatiotemporal dependencies between video frames (such as the continuity of background textures, consistency of facial contours, and uniformity of color tones between consecutive frames) to synthesize content in the blank areas, referring to the corresponding regional features of adjacent frames to complete effective content that is consistent with the overall picture; at the same time, the deformed areas are morphologically corrected to restore their natural geometric shape; during this process, because safety boundaries have been added to each frame in the early stage, the core content is avoided from being lost during the distortion and completion process; finally, the completed and corrected frames are cropped to the original size of the initial video to generate a full-view frame with complete field of view, no missing parts, and no deformation.

[0075] In this embodiment, by relying on the region marking of the refined mask, the problem of blanking and structural deformation caused by the jitter distortion of stable frames is solved through targeted missing completion and structural deformation correction. At the same time, combined with the safety boundary added in the early stage, it is ensured that the completed frame can retain the core content and complete field of view of the original video. Finally, a full field of view frame with complete field of view, no structural deformation, and clear details is generated, which significantly improves the visual integrity and practicality of the video.

[0076] For example, a motion mask-guided edge completion network (WVS-Net) processes stable frames to generate full-view frames, which are then used to render high-quality full-view frames. The input to WVS-Net is a warp field sequence from LDS-Net. and stable enhanced frame sequence First, based on the twisted field Obtain the original mask Considering that video stabilization often introduces structural distortion, especially at frame boundaries, information-rich neighborhood regions are used as cues to refine the motion mask. Multiple sampling points are taken at the video frame boundaries, and by utilizing the key information from the neighborhood region of each sampling point, the corrected sampling point positions can be obtained. A coarse motion mask can then be obtained by filling in the center region of the connection points. Then, the original mask was applied. and coarse mask The shared region in the image is regarded as the content region, and a refined mask is generated from it. In the mask sequence Based on this, frame sequences are synthesized by optimizing the model [5]. The missing regions are used to obtain a high-quality stable frame sequence. In this process, to avoid content loss during frame warping in the decluttering task, safety boundaries are added to each frame before LDS-Net processing. This allows WVS-Net to fully capture and utilize rich spatiotemporal dependencies for view synthesis. Accordingly, the synthesized result is cropped back to its original size to obtain the final high-quality frame sequence covering the entire view. .

[0077] S16. Integrate the full field of view frames to generate the target video sequence, which is a video sequence with a complete field of view and stable image operation.

[0078] Specifically, first, the cropped full-view frames are sorted according to the frame time sequence of the initial video sequence to ensure that the temporal logic between frames is consistent with the original video and to avoid frame order disorder; second, the transition between frames is checked for consistency (such as background texture, light brightness, and color tone) to ensure that there are no obvious splicing marks or abrupt changes; finally, the sorted full-view frames are packaged into a complete video sequence according to the frame rate parameters of the initial video to generate the final target video.

[0079] In one specific embodiment, before integrating the full-view frames to generate the target video sequence, the full-view frames are cropped to the same frame size as the video frames in the initial video sequence.

[0080] Specifically, first, the original size parameters (such as resolution and aspect ratio) of the video frames in the initial video sequence are recorded; second, the size composition of the full-view frames is analyzed to identify the extra edge regions caused by adding safety boundaries and filling in missing areas in the early stage; finally, the full-view frames are cropped according to the size specifications of the initial frames, retaining the core content areas (such as faces and the main background) and removing extra edges to ensure that the cropped frames are completely consistent with the initial video in terms of size and proportion.

[0081] In this embodiment, by integrating full-view frames, the single-frame processing effects such as detail enhancement, shaking, missing data completion, and size calibration are transformed into a coherent video sequence, ultimately generating a target video with "high-quality details, motion stability, complete field of view, and consistent specifications." This solves the core quality problem of the initial video, meets the practical application needs of social media dissemination and facial vision tasks (such as key point detection), and improves the perceptual quality and practical value of the video.

[0082] The above solution addresses pixel-level distortion and timing jitter caused by unstable acquisition equipment in the initial video sequence by sequentially performing detail enhancement, acquisition device trajectory determination, offset data smoothing, and missing data completion. This not only effectively improves the detail quality and visual clarity of the video image but also ensures a smooth transition in the video's running state. Furthermore, missing data completion ensures the integrity of the target video sequence's field of view, ultimately generating a target video with high-quality visual effects, stable operation, and a complete field of view. This enhances the user's viewing experience and provides high-quality video data support for subsequent face-related visual tasks (such as key point detection). It has wide applicability and strong practicality.

[0083] The method of this disclosure will be further described below through a specific embodiment.

[0084] Figure 3 This is a schematic diagram of the structure of a unified recurrent network for face video quality enhancement and stabilization provided in an embodiment of this disclosure, as shown below. Figure 2 , Figure 3 As shown, the UniFES unified recurrent network architecture comprises three core sub-networks: a keypoint-guided progressive alignment and enhancement network, a local dynamic information-aware stabilization network, and a motion mask-guided edge completion network. These three networks are interconnected through a recurrent structure, forming a unified processing framework capable of simultaneously enhancing the quality and stabilizing face videos. In the UniFES processing flow, the input... A low-quality video sequence with jitter, in which Represents the t-th frame. This represents the total number of frames. The final output is a high-quality, stable video sequence. At the same time, maintain the original field of vision.

[0085] The method disclosed herein establishes a bidirectional information exchange mechanism between the pixel domain and the motion domain: the quality enhancement task benefits from the global dynamic information (i.e., low-frequency components in the motion domain) provided by the stabilization task, while the stabilization task integrates the mixed dynamic information (i.e., high-frequency details in the pixel domain) from the enhancement task. This bidirectional interaction is strengthened through a recursive structure, enabling continuous transfer and optimization of information between consecutive frames, thereby effectively addressing the limitation of independent quality enhancement and motion stabilization in traditional methods.

[0086] In addition, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a face video quality enhancement and shaking removal processing device 400 provided in an embodiment of this disclosure. The device includes: The acquisition module 401 is used to acquire an initial video sequence containing multiple video frames. The initial video sequence is a video sequence with pixel-level distortion and / or timing jitter caused by unstable operation of the acquisition device. Enhancement module 402 is used to enhance the details of the initial video sequence and generate enhanced frames; The trajectory estimation module 403 is used to determine the running trajectory of the acquisition device for acquiring the initial video sequence based on the enhanced frame; The smoothing module 404 is used to smooth the motion offset data related to the acquisition device in the enhanced frame based on the motion trajectory, and generate a stable frame. The completion module 405 is used to complete the missing data in the stable frame and generate a full-view frame; The generation module 406 is used to integrate the full field of view frames to generate a target video sequence, which is a video sequence with a complete field of view and stable image operation.

[0087] Optionally, the enhancement module 402 is also used for: Extract the current video frame, facial key point features in the current video frame, and the enhanced stabilization result of the reference frame from the initial video sequence. The reference frame is the frame before the current video frame. Based on global dynamic information, the reference frame and the current video frame are globally aligned. Global dynamic information is used to characterize the motion trends of multiple video frames in the initial video sequence related to the acquisition device, and hybrid dynamic information is used to characterize the dynamic features in the video frame that are associated with pixel-level details. Based on globally aligned reference frames, the encoder-decoder architecture combined with a residual attention mechanism is used to reconstruct each video frame of the initial video sequence and generate enhanced frames.

[0088] Optionally, the face video quality enhancement and stabilization processing device 400 also includes a decoupling module: The decoupling module is used to decouple global dynamic information from mixed dynamic information to obtain local dynamic information. The local dynamic information is used to characterize the motion differences or motion changes in local areas between the reference frame and the current video frame. Based on local dynamic information, local alignment and residual alignment are performed on the reference frame and the current video frame.

[0089] Optionally, the smoothing module 404 is also used for: A distortion field is generated based on the motion trajectory. The distortion field is used to characterize the set of geometric transformation parameters for geometric adjustment of the enhanced frame. Based on the distortion field, the running offset data related to the acquisition device in the enhanced frame is smoothed to generate a stable frame.

[0090] Optionally, the completion module 405 is also used for: Based on the distortion field, an original mask is generated. The original mask is used to characterize the distribution of effective content areas and blank missing areas in the stable frame. Multiple sampling points are selected at the boundary of the stable frame, and the position of the sampling points is corrected by using the neighborhood information of each sampling point to obtain a coarse motion mask. By fusing the shared content region of the original mask of the stable frame and the coarse motion mask, a refined mask is generated; Based on the refined mask, missing data in the stable frame is filled in to generate a full-view frame.

[0091] This embodiment also provides an electronic device, including a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the above-described method for enhancing and de-shaking facial video quality, thus achieving the same effect as the above-described method.

[0092] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for enhancing and de-shaking facial video quality.

[0093] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing and de-shaking facial video quality, characterized in that, The method includes: Obtain an initial video sequence containing multiple video frames, wherein the initial video sequence is a video sequence with pixel-level distortion and / or timing jitter caused by unstable operation of the acquisition device; The details of the initial video sequence are enhanced to generate enhanced frames; Based on the enhanced frame, determine the operating trajectory of the acquisition device that acquired the initial video sequence; Based on the motion trajectory, the running offset data related to the acquisition device in the enhanced frame is smoothed to generate a stable frame; The missing data in the stable frame is filled in to generate a full-view frame; The full field-of-view frames are integrated to generate a target video sequence, which is a video sequence with a complete field of view and stable image operation.

2. The processing method according to claim 1, characterized in that, The step of enhancing the details of the initial video sequence to generate enhanced frames includes: Extract the current video frame of the initial video sequence, the facial key point features in the current video frame, and the enhanced stabilization result of the reference frame, where the reference frame is the frame preceding the current video frame; Based on global dynamic information, the reference frame and the current video frame are globally aligned. The global dynamic information is used to characterize the motion trends of multiple video frames in the initial video sequence related to the acquisition device. Based on the globally aligned reference frame, the video frames of the initial video sequence are reconstructed using an encoder-decoder architecture combined with a residual attention mechanism to generate enhanced frames.

3. The processing method according to claim 2, characterized in that, Before reconstructing each video frame of the initial video sequence based on the globally aligned reference frame and generating the enhanced frame using an encoder-decoder architecture combined with a residual attention mechanism, the method further includes: The global dynamic information and the hybrid dynamic information are decoupled to obtain local dynamic information. The local dynamic information is used to characterize the motion differences or motion changes in local regions in the reference frame and the current video frame. The hybrid dynamic information is used to characterize the dynamic features associated with pixel-level details in the video frame. Based on the local dynamic information, local alignment and residual alignment processing are performed on the reference frame and the current video frame.

4. The processing method according to claim 2, characterized in that, The step of smoothing the motion offset data related to the acquisition device in the enhanced frame based on the motion trajectory to generate a stable frame includes: A distortion field is generated based on the motion trajectory, and the distortion field is used to characterize the set of geometric transformation parameters for geometric adjustment of the enhanced frame; Based on the distortion field, the running offset data related to the acquisition device in the enhanced frame is smoothed to generate the stable frame.

5. The processing method according to claim 4, characterized in that, The step of completing the missing data in the stable frame to generate a full-view frame includes: Based on the distortion field, an original mask is generated, which is used to characterize the distribution of valid content regions and blank missing regions in the stable frame. Multiple sampling points are selected at the boundary of the stable frame, and the position of the sampling points is corrected using the neighborhood information of each sampling point to obtain a coarse motion mask. A refined mask is generated by fusing the shared content region of the original mask of the stable frame and the coarse motion mask. Based on the refined mask, missing data in the stable frame is filled in to generate the full-view frame.

6. The processing method according to claim 5, characterized in that, Before smoothing the running offset data related to the acquisition device in the enhanced frame based on the motion trajectory to generate a stable frame, the method further includes: adding a safety boundary to each video frame, the safety boundary being used to add an additional content area outside the original edge of the video frame.

7. The processing method according to claim 1, characterized in that, Before integrating the full-view frames to generate the target video sequence, the method further includes: The full-view frame is cropped to the same frame size as the video frames in the initial video sequence.

8. A processing device for enhancing and stabilizing facial video quality, characterized in that, The device includes: The acquisition module is used to acquire an initial video sequence containing multiple video frames, wherein the initial video sequence is a video sequence with pixel-level distortion and / or timing jitter caused by unstable operation of the acquisition device; The enhancement module is used to enhance the details of the initial video sequence and generate enhanced frames; The trajectory estimation module is used to determine the operating trajectory of the acquisition device that acquires the initial video sequence based on the enhanced frame; A smoothing module is used to smooth the running offset data related to the acquisition device in the enhanced frame based on the motion trajectory, and generate a stable frame; The completion module is used to complete the missing data in the stable frame and generate a full-view frame; The generation module is used to integrate the full field-of-view frames to generate a target video sequence, which is a video sequence with a complete field of view and stable image operation.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the face video quality enhancement and de-shaking processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the face video quality enhancement and de-shaking processing method as described in any one of claims 1 to 7.

Images