Video editing methods, apparatus, storage media, devices and software products

Abstract

This application discloses a video editing method, apparatus, storage medium, device, and program product. The method acquires the original video target segment and determines the target state reference frame. It uses facial stable region feature points to perform geometric deformation and adaptive fusion to generate a geometrically guided frame sequence. After selective noise addition and denoising, it finally achieves precise editing of the target editing area and feature locking of the non-editing area through mask-guided local fusion technology. This enables precise editing of the open and closed state of a person's eyes in the video, while ensuring seamless connection between the target editing area and the original video in the spatiotemporal dimension. This effectively improves the visual naturalness and inter-frame continuity of the eye opening and closing state transition.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a video editing method, apparatus, storage medium, device, and program product. Background Technology

[0002] With the continuous development of video editing technology, the editing and modification of facial expressions in videos has become an important research direction. Especially in film and television production, advertising design, and social media, there is a wide range of applications for the precise control and modification of facial expressions in videos. Among these, the opening and closing of a person's eyes, as a crucial component of facial expression, makes its editing technology particularly critical.

[0003] Traditional video editing methods often rely on manual frame editing, which is not only time-consuming and labor-intensive but also struggles to guarantee the continuity and naturalness of the edited results. In recent years, with the development of deep learning technology, neural network-based video editing methods have gradually emerged. However, traditional deep learning-based video editing methods still face challenges when handling the editing of a person's eyes (open or closed), such as unnatural editing effects, loss of detail in the eye area, and discontinuity between video frames. Summary of the Invention

[0004] This application provides a video editing method, apparatus, storage medium, device, and program product. By combining geometric deformation technology and video diffusion technology, it achieves precise editing of the open and closed state of a person's eyes in a video, while ensuring the continuity and naturalness of the editing effect.

[0005] On one hand, embodiments of this application provide a video editing method, the method comprising: Obtain the target segment to be edited from the original video and determine the target state reference frame; Based on the target state reference frame and the facial stable region feature points of each target frame in the target segment, geometric deformation and adaptive fusion are performed on each target frame in the target segment to generate a geometric guide frame sequence including the geometric guide frame corresponding to each target frame. The geometric guide frame has the spatial structure of the target state in the target editing region of the corresponding target frame and is geometrically aligned with the non-editing region of the corresponding target frame. The geometric guidance frames corresponding to the middle frames of the target segment in the geometric guidance frame sequence are subjected to noise processing to obtain a noisy geometric guidance frame sequence, wherein the geometric guidance frames corresponding to the first and last frames of the target segment are kept noise-free or injected with a small amount of noise. The denoised geometric guide frame sequence is denoised to generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region, so that the generation process only updates the target editing region and locks the original features of the non-editing region. The denoised video clip is then merged with the original video to obtain the edited video.

[0006] In some embodiments, before extracting the facial stabilization region feature points of the target state reference frame and each target frame in the target segment, the method further includes: obtaining a mask of the target editing region of each target frame in the target segment, the mask being used to limit the range of geometric deformation and generation operations within the target editing region.

[0007] In some embodiments, obtaining the mask of the target editing region of each target frame in the target segment includes: constructing an initial target editing region polygon based on eyelid contour key points predicted by a face feature point detection model; and performing a morphological dilation operation on the initial target editing region polygon to obtain a mask covering the target editing region.

[0008] In some embodiments, fusing the denoised video segment with the original video to obtain an edited video includes: obtaining bounding box information of the face region in each frame of the original video; scaling the edited face image of each frame in the denoised video segment to the corresponding original bounding box size according to the bounding box information of the face region, and placing it back into the bounding box position of the corresponding frame in the original video; and reassembling all processed frames in the original time sequence to generate the edited video.

[0009] On the other hand, embodiments of this application provide a video editing apparatus, the apparatus comprising: The first processing unit is used to acquire the target segment to be edited in the original video and determine the target state reference frame; The second processing unit is used to perform geometric deformation and adaptive fusion on each target frame in the target segment based on the facial stable region feature points of the target state reference frame and each target frame in the target segment, and generate a geometric guide frame sequence including the geometric guide frame corresponding to each target frame. The geometric guide frame has the spatial structure of the target state in the target editing region of the corresponding target frame and is geometrically aligned with the non-editing region of the corresponding target frame. The noise-adding unit is used to add noise to the geometric guide frame corresponding to the middle frame of the target segment in the geometric guide frame sequence to obtain a noise-adding geometric guide frame sequence, wherein the geometric guide frames corresponding to the first and last frames of the target segment are kept noise-free or injected with a small amount of noise. A denoising unit is used to denoise the noisy geometric guide frame sequence and generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region, so that the generation process only updates the target editing region and locks the original features of the non-editing region. The editing unit is used to merge the denoised video segment with the original video to obtain the edited video.

[0010] On the other hand, an embodiment of this application provides a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the video editing method as described in any of the above embodiments.

[0011] On the other hand, an embodiment of this application provides a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the video editing method described in any of the above embodiments by calling the computer program stored in the memory.

[0012] On the other hand, an embodiment of this application provides a computer program product, including computer instructions that, when executed by a processor, implement the video editing method as described in any of the above embodiments.

[0013] This application embodiment acquires the original video target segment and determines the target state reference frame. It uses facial stable region feature points to perform geometric deformation and adaptive fusion to generate a geometric guide frame sequence. Then, it performs selective noise addition and denoising processing (preserving the stable features of the first and last frames). Finally, it uses mask-guided local fusion technology to achieve precise editing of the target editing area and feature locking of the non-editing area. This enables precise editing of the open and closed state of the person's eyes in the video, while ensuring seamless connection between the target editing area and the original video in the spatiotemporal dimension. This effectively improves the visual naturalness and inter-frame continuity of the eye opening and closing state transition. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a technical roadmap of the entire algorithm for the video editing method provided in the embodiments of this application.

[0016] Figure 2 This is a flowchart illustrating the video editing method provided in an embodiment of this application.

[0017] Figure 3 This is a sample illustration of the data processing stage provided in the embodiments of this application.

[0018] Figure 4 This is an example diagram illustrating the geometric deformation stage provided in an embodiment of this application.

[0019] Figure 5 This is a sample illustration of the denoising stage of the video diffusion model provided in the embodiments of this application.

[0020] Figure 6 This is a sample illustration of the result of cropping a video face with eyes open, provided as an embodiment of this application.

[0021] Figure 7 This is a sample illustration of the overall video face eye-opening editing result provided in an embodiment of this application.

[0022] Figure 8 This is a schematic diagram of the structure of the video editing device provided in the embodiments of this application.

[0023] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a video editing method, apparatus, storage medium, device, and program product. Exemplarily, the video editing method of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, personal computer (PC), smart vehicle terminal, etc. The terminal can also include a client, which can be a mini-program, video editing client, browser client, or instant messaging client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0026] The embodiments of this application can be applied to scenarios such as video processing, video editing, and image editing.

[0027] First, the following explanations are given for some of the nouns or terms that appear in the embodiments of this application: Geometric warping is a computer graphics operation that takes two inputs: 1) a source image, and 2) a set of feature points selected from the source image and their specified target locations. The output is a warped image in which the positions of the feature points coincide with the specified target locations, and pixels in non-feature point regions undergo spatial transformation through interpolation algorithms.

[0028] Diffusion models refer to a class of machine learning models based on generative artificial intelligence. Their core principle simulates a two-stage process: 1) A forward diffusion process, where noise is gradually added to the training data (e.g., images), disrupting its structure until it becomes pure random noise. 2) A reverse denoising process, where a neural network is trained to learn the inverse of this process, starting with pure random noise, progressively predicting and removing noise, ultimately generating new data samples that conform to the original data distribution. This model learns the recovery patterns of data under noise interference, thereby gaining the ability to generate high-quality, diverse data from random noise.

[0029] Video Diffusion Model: A generative artificial intelligence (AI) model based on a denoising diffusion probability model. Starting with random noise, it gradually generates a new video that is temporally coherent and conforms to text or other conditions (such as sketches or reference images) through a multi-step denoising process. Compared to image diffusion models, its core challenge and key technology lie in modeling and maintaining temporal consistency and dynamic smoothness between video frames.

[0030] Face detection refers to the process of analyzing an input image using computer algorithms (such as feature-based methods, machine learning methods, or deep learning methods) to determine whether a face exists in it, and if so, to determine its location and extent.

[0031] Flow matching is a novel generative modeling paradigm in generative artificial intelligence. Unlike the stochastic process of diffusion models (which aim to progressively "guess" and remove noise from noise), flow matching aims to learn a deterministic, smooth path that continuously and reversibly transforms data (such as the latent representation of images and videos) from simple, known distributions (such as Gaussian noise) to complex data distributions. Its core idea is to construct a trajectory, rather than learning complex "denoising" steps, by defining an ordinary differential equation that describes a continuous, differentiable trajectory from noise to data points.

[0032] Eye Opening Coefficient: A quantitative index calculated based on the geometric relationship of key points around the eye, used to characterize the degree of eye opening and closing. This coefficient is obtained by calculating the ratio of the vertical distance between the upper and lower edges of the eyelids to the horizontal distance between the left and right corners of the eyes. Its value is strongly correlated with the actual degree of eye opening and closing; the larger the value, the greater the degree of eye opening, and the smaller the value, the greater the degree of eye closing.

[0033] Truncated noise injection: Truncated noise injection is a specific method for adding noise to data in data processing (such as video processing, image generation, etc.). Unlike ordinary noise injection, truncated noise injection sets a threshold or range for noise injection. Instead of adding noise indefinitely, the intensity or amplitude of the noise is limited to a pre-defined range. For example, in a video diffusion model, truncated noise injection is used when adding noise to a geometrically guided frame sequence, terminating early when a preset moderate noise level is reached. At this cutoff point, the injected noise is sufficient to perturb and blur imperfect texture details, but it has not completely destroyed the core spatial structure information provided by the geometric deformation, providing suitable noise interference conditions for subsequent denoising processing to generate video clips that meet expectations.

[0034] Local spatial fusion is a technique used in image or video processing to fuse specific local regions. In applications such as video editing, it's employed when processing specific regions of a video frame (e.g., the target editing region) while preserving the original features of other regions (non-editing regions). Through specific algorithms, it seamlessly merges the processed target editing region with the original non-editing region based on the mask information of the target editing region. This results in a visually natural and smooth fused video frame, free from obvious splicing artifacts or regional inconsistencies, ensuring a good overall visual effect for the edited video. Specifically, after each denoising iteration generates the current latent representation, only the latent features corresponding to the target editing region mask in the prediction result are extracted and fused into the input latent representation of the current step, replacing the corresponding parts of the masked region. For non-editing regions outside the mask, the original latent features in the input latent representation are completely preserved, ensuring that the non-editing regions remain unchanged throughout the denoising process.

[0035] Temporal anchors are reference points used to mark and locate specific points in time or time periods when processing data with time-series characteristics (such as video). For example, a temporal anchor is a keyframe in a video sequence that serves as a temporal reference. In tasks such as video editing and video generation, temporal anchors can help determine the location of keyframes, the timing of specific actions or events, etc. For instance, when generating a video, setting temporal anchors ensures that frames generated at different times maintain a certain temporal logic and coherence. In video editing, temporal anchors can be used to precisely align the timelines of different video segments, or as a reference to adjust the temporal relationships between frames when processing video frame sequences, ensuring the smoothness and accuracy of the video in the temporal dimension. In this application, the first and last frames of the target segment are set as temporal anchors, with their latent features remaining noise-free or only slightly injected. These two frames, as real references with the target state, provide reliable temporal start and end points for the entire generation process, and are key visual anchors for maintaining the temporal and identity consistency of the generated sequence.

[0036] Adaptive Weighted Fusion: A temporal distance-based weighted fusion strategy used to fuse forward and backward deformation results. In video editing and other scenarios, when it is necessary to fuse multiple video frames, video data from different processing stages, or data with different features, adaptive weighted fusion automatically assigns an appropriate weight to each data point based on factors such as quality, relevance, and importance. For example, when fusing a denoised video segment with the original video, the denoised data may be given a higher weight for the target editing area, while the original video data is tended to be retained for the non-editing area, giving it a lower weight. This adaptive weight allocation and fusion method can better balance the advantages of different data sources, generating higher quality and more natural-looking edited videos. In this application, for the t-th frame in the target segment, its fusion weight is linearly determined by the frame's temporal position in the segment: the forward deformation result (from the first reference frame) has a higher weight when it is near the beginning of the segment; the backward deformation result (from the last reference frame) has a higher weight when it is near the end of the segment. This strategy enables the creation of geometrically guided frames with smooth transitions on the timeline of scenes with closed eyes.

[0037] Facial stable region feature points refer to relatively stable feature points in a facial image that are not easily affected by changes in expression and posture. These include brow ridge contour points, bridge and tip of the nose points, and outer contour points of the cheeks and chin. In video editing, these feature points are used to achieve geometric deformation between the target frame and the reference frame, ensuring geometric consistency between the edited and non-edited areas in the edited video.

[0038] The mask for the target editing region is a binary image used to identify the target editing region and the non-editing region in a video frame. During geometric deformation, noise addition, and denoising processes, the mask limits the scope of the operation, ensuring that only the target editing region is processed, while the non-editing region retains its original features.

[0039] Thin plate spline transformation model: This is a nonlinear spatial transformation model that can achieve smooth image deformation based on the correspondence between feature points. In video editing, it is used to align the deformation of a reference frame to the pose of a target frame.

[0040] In a wide range of fields, including video content creation, film and television post-production, video communication enhancement, and virtual digital human driving, refined and high-fidelity editing of facial video expressions is a fundamental and crucial requirement. Among these, the natural editing of eye opening and closing (e.g., correcting frames with closed eyes caused by improper blinking) is a core element in enhancing content expressiveness and visual smoothness, directly impacting the final visual quality and viewing experience. To achieve more engaging video effects, natural and seamless editing of eye expressions has become an industry necessity. For example, correcting frequent blinking in interview videos can significantly enhance visual continuity and improve viewing immersion and realism through precise eye-opening editing.

[0041] Traditional video editing methods face a series of inherent challenges and limitations when handling human-eye editing tasks, mainly in the following three aspects: 1. Direct transfer of static image processing methods leads to temporal inconsistencies. Common diffusion-based image editing techniques primarily optimize single-frame images, lacking modeling for the temporal continuity of video. When directly applied to video sequences, the independent processing of each frame results in random differences in detail, causing flickering, jitter, and inconsistent changes in eye morphology (such as pupil position, eye shape, and gaze direction) in the edited video over time, severely compromising visual smoothness and realism. In contrast, this application deeply integrates the temporal modeling capabilities of video diffusion models, explicitly constraining inter-frame consistency during generation. Through temporal weighted fusion and temporally aware denoising, it ensures a smooth temporal transition in the generated open-eye sequence, achieving a highly coherent and natural video editing effect.

[0042] 2. Insufficient protection of non-editable areas can easily lead to distortion of identity information and background tampering. Common diffusion-based editing methods, when performing local editing (such as opening someone's eyes), typically require global noise processing and reconstruction of the entire image. This process easily "contaminates" non-editable areas that should remain unchanged (such as skin texture, hairstyle, accessories, and video background), causing facial features to drift and background details to be accidentally modified, thus compromising the authenticity and consistency of the video content. This application precisely limits the editing scope through a strategy combining localized conditional generation and geometric guidance: during denoising, the model only updates the target eye region, while other parts of the face and the background are strictly locked and protected as strong conditional inputs. This achieves the effect of "high-fidelity generation of editable areas and zero-distortion preservation of non-editable areas," ensuring the complete preservation of the person's identity and scene information.

[0043] 3. Weak spatial guidance leads to inaccurate matching of edited results with facial pose. Common methods typically rely on text descriptions or coarse spatial cues (such as semantic segmentation maps) for content generation control. In tasks requiring high-precision spatial alignment, the lack of fine-grained guidance results in edited results (such as generated eyes) that are inconsistent with the overall facial pose and expression in terms of position, shape, and perspective, producing a "floating" or "distorted" appearance. This application introduces geometric deformation as a pixel-level spatial guide. By calculating the precise spatial transformation from the reference frames to the target frame, it provides the diffusion model with explicit prior information on the eye position, contour, and 3D pose. This strong geometric condition enables the generated eye structure to strictly match facial muscle movements and head rotation, achieving sub-pixel-level spatial alignment and improving the accuracy and naturalness of the edited results in the spatial dimension.

[0044] Please see Figure 1 , Figure 1 This is a technical roadmap of the entire algorithm for the video editing method provided in this application embodiment. This application embodiment designs a video human eye-opening editing algorithm based on a geometric deformation and diffusion model. This algorithm provides high-precision spatial guidance through geometric deformation, ensures smooth and coherent inter-frame dynamics using a video diffusion model, and employs a localized editing strategy. While achieving realistic eye-opening editing, it strictly protects non-editing areas, ensuring a high degree of consistency between the person's identity and visual characteristics. The technical roadmap of the entire algorithm is shown in... Figure 1 The process mainly includes inputting the original video, data processing, geometric deformation, video diffusion model denoising, and post-processing to output the edited results. This application can solve the following problems: 1. Solving the problem of inconsistent timing in video editing: By introducing the timing modeling capability of the video diffusion model and the frame alignment mechanism based on geometric deformation, the problem of screen flickering, jittering and discontinuous eye dynamics generated in video sequences by traditional single-frame image processing methods is effectively overcome, and the editing of open-eye state with smooth timing and natural dynamics is achieved.

[0045] 2. Solve the problem of non-editable areas being easily tampered with: By truncating noise injection and using localized editing strategies, while ensuring high-quality generation of the eye area, other areas of the face and background content are strictly protected from being affected, thus fully preserving the character's identity features and scene information, and avoiding identity distortion and background tampering during the editing process.

[0046] 3. Solving the problem of insufficient spatial alignment accuracy in editing results: By using pixel-level spatial guidance information generated by geometric deformation, the diffusion model is provided with precise eye position, contour and pose constraints, so that the generated eye structure is strictly matched with facial expression and head movement in three-dimensional space, solving the problems of spatial misalignment and unnatural deformation that are common in general methods.

[0047] This application significantly reduces the cost and barrier to entry for professional production. By transforming the traditional "open-eye" operation, which requires experienced editors to manually repair frame by frame, into a fully automated, one-click intelligent process, the professional skill requirements for operators are greatly reduced. Efficiency and cost reduction: Manual retouching work that might have taken hours or even days is shortened to minutes or even seconds, greatly improving the efficiency of post-production processes such as film and television production and short video production, directly saving manpower and time costs.

[0048] This application's embodiments guarantee and improve the quality and consistency of content production. Regardless of the type of video material processed (such as interviews, live broadcasts, films, etc.), the algorithm can output restoration results that meet broadcast-grade or commercial-grade visual standards, avoiding quality fluctuations that may occur during manual processing. Stylistic consistency: In series of content (such as multi-episode courses, TV series, etc.), it can ensure that the eye expressions of the same person in different segments remain consistent, maintaining the overall professional quality of the work.

[0049] This application's embodiments activate and expand the possibilities of creative expression. It provides content creators with unprecedented controllable editing capabilities, allowing them to creatively fine-tune or reshape a person's gaze (such as creating special catchlights, adjusting the direction of gaze, etc.), opening up new forms of artistic expression. Material rescue and value enhancement: It can "rescue" valuable footage that was originally discarded due to a person blinking (such as precious interviews, beautiful shots, etc.), turning waste into treasure, directly increasing the value and reuse rate of the material library.

[0050] In practice, after the user selects the closed-eye segment and submits it to the algorithm backend, the algorithm automatically edits the face and outputs the open-eye result.

[0051] The solutions provided in this application relate to technologies such as video editing, and are specifically illustrated through the following embodiments. Detailed descriptions are provided below. It should be noted that the order of description of the following embodiments is not intended to limit the priority of the embodiments.

[0052] This application provides a video editing method, which can be executed by a terminal or a server, or by both a terminal and a server. This application uses the example of a video editing method executed by a terminal for illustration.

[0053] Please see Figures 2 to 7 , Figure 2 This is a flowchart illustrating the video editing method provided in an embodiment of this application. Figures 3 to 7 All the images shown are examples provided in the embodiments of this application. The method may include the following steps: Step 110: Obtain the target segment to be edited from the original video and determine the target state reference frame.

[0054] like Figure 1 and Figure 3 The data processing stage shown here, after obtaining the original video input, first executes an automated preprocessing workflow. Its core objective is to accurately extract facial dynamic information from the video and provide a complete data foundation for subsequent geometric deformation and editing. This stage does not modify any content; instead, it completes the localization, detection, and analysis of the target object (face). This data processing stage mainly includes the following core steps: face detection and cropping, eye region attribute analysis, determining the first and last frames of closed-eye segments, and video segment cropping.

[0055] This data processing stage uses cascaded face detection and feature point analysis technology, combined with temporal smoothing and state machine judgment, to automatically locate the closed-eye segments (such as frames 12 to 17) that need to be edited from the original video, and extracts the stable open-eye frames before and after as references.

[0056] In some embodiments, step 110 can be implemented by the following steps 111 to 114, specifically including: Step 111: Perform face detection and cropping on the original video to obtain an aligned face sequence.

[0057] In some embodiments, the step of performing face detection and cropping on the original video to obtain an aligned face sequence includes: performing face detection on each frame of the original video to obtain bounding box information of the face region, the bounding box information including bounding box coordinates; performing exponential motion smoothing on the bounding box coordinates to obtain a stable face trajectory across frames; and cropping each frame of the original video according to the face trajectory to obtain an aligned face sequence centered on the face and with normalized dimensions.

[0058] In face detection and cropping, pre-trained face detection models can be used to detect each frame of the input raw video in real time. These models are trained on a large amount of data and can quickly and accurately locate face regions and output bounding box information for all face regions. The bounding box information includes bounding box coordinates (usually the coordinates of the top left and bottom right corners or the center point coordinates and width and height information).

[0059] Then, the bounding box coordinates are subjected to exponential motion smoothing. Exponential motion smoothing is a time series smoothing method that effectively reduces bounding box jitter caused by single-frame detection errors by assigning different weights to data at different times (e.g., recent data has a larger weight), thereby obtaining stable face trajectories across frames.

[0060] Subsequently, based on the obtained face trajectories, each frame in the original video is cropped. Based on the bounding box information determined by the face trajectories, each frame is cropped with the face as the center and according to preset size specifications (such as fixed width and height), resulting in an aligned face sequence centered on each face and with standardized dimensions.

[0061] This step ensures that subsequent processing can focus on the target region (face), avoiding interference from irrelevant areas and improving processing efficiency. Simultaneously, it provides standardized input for spatial alignment in face analysis, ensuring consistent face position and size across different frames, laying the foundation for subsequent accurate eye region attribute analysis and other operations.

[0062] Step 112: Perform target editing region attribute analysis on the aligned face sequence to determine the target segment that does not have a target state in the target editing region.

[0063] In some embodiments, the target editing region is the eye region, and the target state is an open-eye state. The step of performing target editing region attribute analysis on the aligned face sequence to determine the target segment whose target editing region does not possess the target state includes: performing target editing region attribute analysis on the aligned face sequence, calculating the open-eye coefficient for each frame in the aligned face sequence to obtain an open-eye coefficient sequence; determining a closed-eye segment based on the temporal variation of each open-eye coefficient in the open-eye coefficient sequence, and using the closed-eye segment as the target segment.

[0064] In some embodiments, the step of performing target editing region attribute analysis on the aligned face sequence and calculating the eye-opening coefficient of each frame in the aligned face sequence includes: performing facial feature point detection on each frame in the aligned face sequence to obtain the coordinates of key eye points; calculating the vertical distance between the upper and lower edges of the eyelids and the horizontal distance between the left and right corners of the eyes based on the coordinates of the key eye points; and using the ratio obtained by dividing the vertical distance by the horizontal distance as the eye-opening coefficient of each frame in the aligned face sequence.

[0065] In some embodiments, the key points of the eye include at least the highest point of the upper eyelid, the lowest point of the lower eyelid, the left corner of the eye, and the right corner of the eye.

[0066] In the eye region attribute analysis, based on the aligned face sequence obtained in step 111, a facial feature point detection model can be used to locate key points in each frame of the face image, obtaining the coordinates of dozens of facial feature points, including eyelid contours and pupil centers. For example, several sets of key feature points in the left and right eye regions can be extracted. These key eye points include at least the coordinates of the highest point of the upper eyelid, the lowest point of the lower eyelid, the left corner of the eye, and the right corner of the eye. Based on the coordinates of the key eye points, the vertical distance between the upper and lower edges of the eyelids (by calculating the vertical distance between the highest point of the upper eyelid and the lowest point of the lower eyelid) and the horizontal distance between the left and right corners of the eyes (by calculating the horizontal distance between the left and right corners of the eye) are calculated. The ratio of the vertical distance to the horizontal distance is used as the eye-opening coefficient for each frame in the aligned face sequence. The core idea of ​​this eye-opening coefficient is to quantify the degree of eye opening and closing by the ratio of vertical distance to horizontal distance. The calculation formula is as follows: calculate the vertical distance by measuring the distance between two pairs of key points (the upper and lower edges of the eyelids), and calculate the horizontal distance by measuring the distance between the left and right corners of the eyes. The eye-opening coefficient is defined as the vertical distance divided by the horizontal distance. This ratio is positively correlated with the actual degree of eye opening and closing; the ratio is larger when the eyes are fully open and approaches zero when they are fully closed. The value of this eye-opening coefficient is strongly correlated with the degree of eye opening and closing, thus achieving a continuous and quantitative description of the human eye state in each frame.

[0067] In some embodiments, determining the closed-eye segment based on the temporal variation of the eye-opening coefficient includes: performing temporal smoothing filtering on the eye-opening coefficient sequence to obtain a smoothed eye-opening coefficient sequence; determining the open / closed state of each frame corresponding to each smoothed eye-opening coefficient in the smoothed eye-opening coefficient sequence according to preset eye-opening and eye-closed thresholds; identifying a complete event from an open-eye state to a closed-eye state and back to an open-eye state by detecting state transitions on the time axis; taking the frame where the smoothed eye-opening coefficient is first lower than the closed-eye threshold and lasts for more than a first preset number of frames as the first frame of the closed-eye segment, taking the frame where the smoothed eye-opening coefficient is again higher than the open-eye threshold and lasts for more than a second preset number of frames as the last frame of the closed-eye segment, and determining the continuous frame sequence between the first frame and the last frame as the closed-eye segment.

[0068] In determining the first and last frames of a closed-eye segment, a closed-eye condition judgment logic based on temporal analysis can be designed to automatically and robustly identify the closed-eye segment. Specifically, the open-eye coefficient sequence is subjected to temporal smoothing filtering to suppress noise, for example, using methods such as moving average filtering, to obtain a smoothed open-eye coefficient sequence. Moving average filtering smooths the data by calculating the average value of the data within a certain window, effectively suppressing noise interference and making the open-eye coefficient sequence smoother and more stable. Subsequently, based on preset dynamic thresholds (such as open-eye threshold and closed-eye threshold, which can be determined through statistical analysis of a large amount of sample data), the open-eye state (e.g., "open-eye" or "closed-eye") of each frame corresponding to each smoothed open-eye coefficient in the smoothed open-eye coefficient sequence is determined. For example, when the open-eye coefficient is greater than the open-eye threshold, it is judged as an open-eye state, and when it is less than the closed-eye threshold, it is judged as a closed-eye state. By detecting state transitions on the time axis, the complete event of transitioning from an open-eye state to a closed-eye state and then back to an open-eye state is identified. Specifically, this can be achieved by traversing the sequence of open and closed eye states to find the starting point where the state changes from "open" to "closed" and the ending point where the state changes back from "closed" to "open". The starting frame of a closed-eye segment is defined as the frame where the open-eye coefficient first falls below the closed-eye threshold and its duration exceeds a first preset number of frames (e.g., ≥2 frames); the ending frame is defined as the frame where the open-eye coefficient rises above the open-eye threshold again and remains stable, and this stable frame is defined as the frame whose duration exceeds a second preset number of frames (e.g., ≥2 frames). The continuous frame sequence between the first and last frames is then defined as the closed-eye segment.

[0069] By accurately calculating the eye-opening coefficient and analyzing temporal variations, the system can automatically and robustly identify closed-eye segments, effectively filtering out interference factors such as occasional jitter and accurately defining the closed-eye intervals to be edited. Simultaneously, it records the nearest stable open-eye frames before and after the closed-eye segment as reference frames, providing accurate reference data for subsequent editing operations and improving the accuracy and reliability of the editing process.

[0070] Step 113: Based on the time information of the target segment, the original video is cropped to obtain the target segment.

[0071] In some embodiments, cropping the original video according to the time information of the target segment to obtain the target segment includes: extracting a continuous sub-video sequence from the first frame to the last frame from the original video according to the timestamps of the first and last frames of the target segment; and cropping the face region of each frame in the continuous sub-video sequence according to the bounding box information of the face trajectory within the corresponding time period of the continuous sub-video sequence to obtain the spatiotemporally aligned target segment.

[0072] When cropping video segments, the original video stream is cropped based on the timestamps of the first and last frames (start and end frames) of the target segment (closed-eyes segment) and the obtained aligned face trajectory. The cropping operation operates in two dimensions, as detailed below: In the time dimension, a continuous sub-video sequence is extracted from the original video, consisting of the first frame (start frame) and the last frame (end frame) of the target segment (closed-eye segment). For example, the original video can be read using a video processing library, and the corresponding sub-video sequence can be accurately extracted based on the timestamp information.

[0073] In the spatial dimension, based on the bounding box information of the face trajectory within the corresponding time period of the continuous sub-video sequence, the face region is cropped from each frame in the continuous sub-video sequence. Similarly, based on the bounding box determined by the face trajectory, each frame is cropped using the same cropping method as in step 111 to obtain the spatiotemporally aligned target segment (closed-eye segment).

[0074] The cropping operation is performed in both time and space dimensions, ensuring that the extracted target segment is a continuous closed-eye interval in time and that the face position and size remain consistent in space, thus achieving temporal-spatial alignment. This cropping result will serve as direct input for subsequent geometric deformation and generative editing, ensuring the consistency and stability of the processed object in terms of temporal continuity and spatial alignment. This lays the data foundation for high-precision editing and avoids poor editing results caused by temporal-spatial inconsistencies.

[0075] Step 114: Based on the time information of the target segment, determine the frames that have the target state before and after the target segment as the target state reference frames.

[0076] In some embodiments, determining the frames with the target state before and after the target segment as the target state reference frames based on the time information of the target segment includes: taking the frame with the target state in one or more frames before the first frame of the target segment as the front reference frame; taking the frame with the target state in one or more frames after the last frame of the target segment as the back reference frame; and determining the front reference frame and the back reference frame as the target state reference frames.

[0077] Specifically, the frame containing the target state (open-eye state) one or more frames preceding the first frame of the target segment (closed-eye segment) is used as the preceding reference frame. This can be achieved by traversing the frames preceding the first frame of the target segment and finding a matching open-eye frame as the preceding reference frame based on the open / closed-eye state determination result (such as the method for determining open / closed-eye state in step 112).

[0078] Specifically, the frame containing the target state (open-eyed state) one or more frames following the last frame of the target segment (closed-eye segment) is used as the subsequent reference frame. Similarly, the frames following the last frame of the target segment are traversed to find open-eye frames that meet the conditions and are used as subsequent reference frames.

[0079] Then, the previous reference frame and the next reference frame are determined as the target state reference frames.

[0080] By recording the nearest stable open-eye frame before and after the closed-eye segment as a reference frame, a reliable target state reference is provided for subsequent geometric deformation, ensuring that the editing result can naturally transition to the open-eye state before and after the closed-eye segment.

[0081] The spatiotemporal aligned target fragment and the before and after eye-opening reference frames output by this data processing stage fully preserve the facial dynamic information and contextual information required for editing, laying a solid data foundation for the subsequent generation of geometric deformation and diffusion models, and ensuring the high fidelity and temporal consistency of the final editing effect.

[0082] Step 120: Based on the target state reference frame and the facial stable region feature points of each target frame in the target segment, perform geometric deformation and adaptive fusion on each target frame in the target segment to generate a geometric guide frame sequence including the geometric guide frames corresponding to each target frame. The geometric guide frames have the spatial structure of the target state in the target editing region of the corresponding target frame and are geometrically aligned with the non-editing region of the corresponding target frame.

[0083] Each target frame refers to every frame of the target segment that requires geometric deformation and adaptive fusion processing, including the first frame, the last frame, and all intermediate frames between the first and last frames of the target segment.

[0084] like Figure 1 and Figure 4 The geometric deformation stage, as shown, takes the spatiotemporally aligned target segment (closed-eye segment) and its corresponding target state reference frames (open-eye reference frames at the beginning and end) output from the previous stage as input. Its core objective is to construct an initial eye state guide with roughly correct spatial layout and temporal smoothness for the subsequent generative model through robust geometric deformation technology. Specifically, based on the geometric structure of the stable facial region, this stage synthesizes a geometric guidance frame for each target frame in the target segment (i.e., each frame from the first to the last frame of the target segment, where the target frame is a closed-eye frame), where the eye region is roughly open and precisely aligned with the current head pose. This geometric deformation stage mainly includes the following core steps: target editing region mask acquisition, first-frame geometric deformation based on local feature points, last-frame geometric deformation based on local feature points, and adaptive weighting.

[0085] This geometric deformation stage excludes eye feature points that are easily affected by eye closure, and selects only feature points in stable facial regions as the basis for deformation. It uses thin plate spline transformation to deform and align the front and back eye-opening reference frames to each target frame, and then performs adaptive weighted fusion based on time position to generate a spatially accurate and temporally smooth geometric guidance frame sequence.

[0086] In some embodiments, step 120 can be implemented by the following steps 121 to 124, specifically including: Step 121: Obtain the mask of the target editing region of each target frame in the target segment. The mask is used to limit the range of geometric deformation and generation operations within the target editing region.

[0087] In some embodiments, obtaining the mask of the target editing region of each target frame in the target segment includes: constructing an initial target editing region polygon based on eyelid contour key points predicted by a face feature point detection model; and performing a morphological dilation operation on the initial target editing region polygon to obtain a mask covering the target editing region.

[0088] In some embodiments, the morphological dilation operation employs an elliptical structural element, and the pixel radius of the dilation is adaptively determined based on the size of the target editing region.

[0089] In obtaining the target editing region mask (such as an eye mask), to precisely define the editing range, an initial target editing region polygon (such as an eye region polygon) is first constructed based on key points predicted by the facial feature point detection model and located on the contours of the upper and lower eyelids. To avoid missing detailed areas of the eyelid edges during deformation and generation operations, a morphological dilation operation is performed on this initial target editing region polygon (such as an eye region polygon). Elliptical structuring elements are preferably used to expand the pixel radius, which can be adaptively adjusted according to the size of the eye region to obtain a mask (such as a soft mask) that completely covers the eyelids, eyelashes, and surrounding skin. This mask ensures that all subsequent geometric deformations and generation operations are strictly confined within the target area, while non-edited areas always retain their original features.

[0090] Step 122: Extract facial stable region feature points from the target state reference frame and each target frame in the target segment. The facial stable region feature points include at least one of the following: brow bone contour points, bridge of nose and tip of nose points, and outer contour points of cheeks and chin.

[0091] Specifically, when extracting feature points from stable facial regions, the mask obtained in step 121 is used to exclude feature points located within the target editing area. Specifically, for all initially detected facial feature points, only those located outside the mask area are retained as the basis for subsequent deformation. That is, only stable feature points located in non-editing areas, such as brow bone contour points, bridge and tip of the nose, and outer contour points of the cheeks and chin, are selected. This avoids interference from blurred or incorrect feature point positioning within the target editing area caused by closed eyes, ensuring the robustness and accuracy of subsequent deformation transformations.

[0092] Step 123: Based on the feature points of the facial stable region, perform geometric deformation on the front reference frame and the back reference frame in the target state reference frame respectively to obtain the forward deformation result and the backward deformation result.

[0093] During geometric deformation, the mask obtained in step 121 is used to limit the scope of interest for the deformation operation. Specifically, when calculating the spatial transformation model, the loss function of the deformation algorithm is optimized only based on the feature point correspondence error in the non-edited region (i.e., outside the masked region), so that the deformation transformation mainly ensures the alignment accuracy of the non-edited region; for the target edited region (i.e. within the masked region), its deformed content is completely inherited from the corresponding region of the reference frame, and is not affected by the missing or incorrect feature points inside the masked region, ensuring that the deformed non-edited region is accurately aligned with the target frame, while the target edited region naturally presents the target state structure of the reference frame.

[0094] In some embodiments, the step of performing geometric deformation on the front reference frame and the back reference frame in the target state reference frame based on the facial stable region feature points to obtain forward deformation results and backward deformation results includes: for each target frame in the target segment, calculating a first spatial transformation model based on the correspondence between the facial stable region feature points in the front reference frame and the target frame, and deforming and aligning the front reference frame to the pose of the target frame through the first spatial transformation model to generate a forward deformation result; calculating a second spatial transformation model based on the correspondence between the facial stable region feature points in the back reference frame and the target frame, and deforming and aligning the back reference frame to the pose of the target frame through the second spatial transformation model to generate a backward deformation result.

[0095] In some embodiments, the first spatial transformation model and the second spatial transformation model are thin plate spline transformation models.

[0096] In performing the first-frame geometric deformation based on local feature points, to avoid ambiguity or mispositioning of eye feature points caused by the closed-eye state, this application excludes feature points of the eye region itself and selects only stable and reliable facial stable region feature points from the facial geometry as the basis for deformation. These include, but are not limited to: brow bone contour points (reflecting the relationship between brow bone shape and eye position), bridge and tip of the nose points (reflecting the facial midline posture), and outer contour points of the cheeks and chin (reflecting the overall facial contour and orientation). These feature points are minimally affected by the closed-eye state and can still be accurately located even when the target frame is in the closed-eye state, providing a reliable correspondence for subsequent deformation. For each target frame in the target segment, a global, smooth first spatial transformation model is calculated using the correspondence between these stable facial region feature points in the previous reference frame (the first open-eye reference frame) and the current target frame in the closed-eye segment. This application preferably uses thin-plate spline transformation because it can achieve smooth interpolation based on discrete point correspondences, minimizing overall bending energy. This transformation warps and aligns the previous reference frame (the first frame with open eyes) to the pose of the current target frame in the closed-eye segment, generating a forward deformation result. The eye structure of this forward deformation result is inherited from the previous reference frame (the first frame with open eyes) and is precisely aligned with the facial geometry (except for the eyes) of the current target frame.

[0097] Specifically, when performing tail-frame geometric deformation based on local feature points, the same strategy and feature point set as the "first-frame geometric deformation based on local feature points" are used. The thin-plate spline transformation from the rear reference frame (tail-frame open-eye reference frame) to the same target frame in the closed-eye segment is calculated. The rear reference frame (tail-frame open-eye reference frame) is then warped and aligned to the pose of the current target frame in the closed-eye segment, generating the backward deformation result. The eye structure of this backward deformation result is inherited from the rear reference frame (tail-frame open-eye reference frame).

[0098] Step 124: Based on the time position of each target frame in the target segment, the forward deformation result and the backward deformation result are adaptively weighted and fused to generate the geometric guidance frame.

[0099] In some embodiments, the step of adaptively weighting and fusing the forward deformation result and the backward deformation result according to the time position of each target frame in the target segment to generate the geometric guide frame includes: determining a fusion weight according to the time position of each target frame in the target segment, wherein the fusion weight is positively correlated with the time distance of the target frame from the first frame of the target segment and negatively correlated with the time distance of the target frame from the last frame of the target segment; and weighting and fusing the forward deformation result and the backward deformation result according to the fusion weight to generate the geometric guide frame.

[0100] In the adaptive weighting process, to obtain a smooth transition geometric guide frame on the timeline of the target segment (closed-eye segment), the forward deformation results and backward deformation results are adaptively weighted and fused based on time distance. For the t-th target frame in the target segment (closed-eye segment), let the time of the first frame (starting frame) of the target segment be t. s The time of the last frame (end frame) of the target segment is t. e The fusion weight w(t) is linearly determined by the temporal position of the target frame t: if it is closer to the beginning of the target segment (first frame), the forward deformation result (from the first frame) has a larger weight; if it is closer to the end of the target segment (last frame), the backward deformation result (from the last frame) has a larger weight.

[0101] In some embodiments, the fusion weights are determined by the following formula: w(t) = (tt s ) / (t e -t s ); Where t is the time index of the target frame, t s t is the time index of the first frame of the target segment. e The time index of the last frame of the target segment; The geometric guide frame is determined by the following formula: I guide (t) = (1-w(t))·I warpfront (t) + w(t)·I warpback (t); Among them, I guide (t) is the geometric guide frame corresponding to the t-th frame, I warpfront (t) represents the forward deformation result, I warpback (t) represents the back deformation result.

[0102] After the above processing, the generated geometric guide frame has the following key characteristics: (1) The target editing area (eyes) has the approximate spatial structure of the open-eye state (such as position, outline, pose, etc.), but the texture details may be rough or unnatural; (2) The non-editing area is precisely geometrically aligned with the corresponding target frame, keeping the original facial features unchanged; (3) The entire segment sequence transitions smoothly in the time dimension, with no sense of jump between adjacent frames.

[0103] The geometric deformation technology designed in this application gives the target editing region (eyes) the spatial structure of the target state (open eyes), while the non-editing region is geometrically aligned with the target frame. This provides precise guidance to the subsequent video diffusion model on "where and in what shape" the eyes should be generated, ensuring that the final open-eye shape accurately matches facial expressions and head movements. After geometric deformation, the target editing region (eye region) has a roughly correct spatial position, outline, and posture, but the texture content is rough or unnatural. In subsequent stages, the geometrically deformed video frames are used as conditional input to the video diffusion model, strictly constraining the spatial layout of the generated content, ensuring the accurate matching of the final open-eye shape with facial expressions and head movements, and achieving geometric correctness and high controllability in video editing. Adaptive weighted fusion enables a smooth transition of the geometrically guided frame sequence in the time dimension, avoiding visual discontinuity caused by state jumps and improving the naturalness of the editing effect. The target editing region mask ensures that the deformation and generation operations are strictly limited to the target area, avoiding unnecessary interference to the non-editing region and maintaining the original features unchanged. By employing facial stable region feature points and thin plate spline transformation model, feature point positioning errors caused by closed eyes are effectively avoided, ensuring the geometric correctness and high controllability of deformation results, and laying a reliable foundation for subsequent generation stages.

[0104] Step 130: Add noise to the geometric guide frames corresponding to the middle frames of the target segment in the geometric guide frame sequence to obtain a noisy geometric guide frame sequence, wherein the geometric guide frames corresponding to the first and last frames of the target segment remain noise-free or are injected with a small amount of noise.

[0105] like Figure 1 and Figure 5 The video diffusion model denoising stage shown takes the guide frame sequence output from the geometric deformation stage and the corresponding eye region mask as input. Its core lies in utilizing a large-scale video diffusion model (such as Wan2.1-Fun-14B-InP) specifically trained for the inpainting task. Under precise spatiotemporal constraints, it performs controllable local generation, transforming the geometric guide frames into a visually realistic and temporally coherent final eye-opening sequence. This stage ensures high-quality and accurate editing through a process of "conditional noise injection - localized generation - mask fusion." The video diffusion model denoising stage mainly includes the following core steps: step 130, which involves adding noise to the video and truncating the noise injection; and step 140, which involves video diffusion model denoising and mask-based video fusion.

[0106] Step 130 transforms the geometric guide frame sequence into the latent feature space, keeps the first and last frames clear as temporal anchors, and injects truncated noise into the middle frames, so that the noisy sequence is in an intermediate state of "structure still exists, details need to be repaired", providing an ideal starting point for the subsequent denoising process.

[0107] In some embodiments, the step of adding noise to the geometric guide frames corresponding to the middle frames of the target segment in the geometric guide frame sequence to obtain a noisy geometric guide frame sequence includes: converting the geometric guide frame sequence to a latent feature space; in the latent feature space, keeping the latent features of the geometric guide frames corresponding to the first and last frames of the target segment noise-free or injecting a small amount of noise as temporal anchors; and injecting noise into the latent features of the geometric guide frames corresponding to the middle frames of the target segment to obtain a noisy geometric guide frame sequence.

[0108] In some embodiments, the step of converting the geometric guide frame sequence to a latent feature space includes: encoding the geometric guide frame sequence from a pixel space to a low-dimensional latent feature space using an encoder of a pre-trained variational autoencoder.

[0109] When adding noise to the video, the input guide frame sequence (RGB pixel space) is first transformed into a low-dimensional latent feature space by a pre-trained variational autoencoder (VAE). This transformation compresses the high-dimensional image data into a compact latent representation, which reduces the computational complexity of the subsequent diffusion model while preserving the semantic structure and spatial layout information of the image.

[0110] Within the latent feature space, structured noise injection is performed to initiate the diffusion generation process. For the geometric guide frames corresponding to the first and last frames of the target segment (closed-eye segment), their latent features are kept essentially noise-free or only a very small amount of noise is injected. These two frames, as the true open-eye state, provide reliable temporal start and end references for the entire generation process, and are key visual anchors for maintaining the temporal and identity consistency of the generated sequence. In the subsequent denoising process, the model can "see" clear start and end points, thereby constraining the generation path of intermediate frames and ensuring the overall temporal coherence.

[0111] For all intermediate frames between the first and last frames, Gaussian noise conforming to the forward propagation process of the diffusion model is injected into their entire latent representation. The level of noise injection is controlled by a preset noise scheduling table, which aims to destroy unnatural textures and rough details in the original guide frame, while roughly preserving the basic eye spatial position and contour information conveyed by geometric deformation.

[0112] In some embodiments, injecting noise into the potential features of the geometric guide frame corresponding to the intermediate frame of the target segment includes: injecting Gaussian noise into the potential features of the geometric guide frame corresponding to the intermediate frame according to a preset noise scheduling table, and terminating the noise injection prematurely when a preset medium noise level is reached, thus forming truncated noise injection.

[0113] In some embodiments, the preset moderate noise level is lower than the maximum noise level of the pre-trained video diffusion model's forward pass, so that the texture details of the noise-added intermediate frames are disturbed and blurred while retaining the spatial structure information provided by the geometric deformation.

[0114] In this application, a truncated noise scheduling strategy is adopted to balance the relationship between "following geometric guidance" and "leveraging model generation capabilities" during truncated noise injection. Specifically, the forward noise injection process does not proceed to completely random noise (corresponding to a maximum time step T). max Instead of directly interfering with the noise level, the process terminates early at a preset, moderate noise level (e.g., time step t=600, the specific value of which is defined by the model's noise scheduling table). At this cutoff point, the injected noise is sufficient to perturb and blur imperfect details, but it does not completely destroy the core structural information provided by the geometric deformation. This strategy allows the subsequent denoising process to begin from an intermediate state of "structure still intact, details yet to be repaired," guiding the model to focus on synthesizing high-frequency, realistic visual details while inheriting the general spatial layout.

[0115] After the above processing, the noisy geometric guide frame sequence has the following key features: the first and last frames serve as clear anchor points, maintaining the original open-eye state and identity features; the latent representation of the middle frames is at a moderate noise level, preserving the basic spatial structure but losing imperfect texture details; the entire sequence provides an ideal starting point and constraints for the subsequent denoising process.

[0116] Step 140: Denoise the noise-added geometric guide frame sequence to generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region so that the generation process only updates the target editing region and locks the original features of the non-editing region.

[0117] Step 140 involves inputting the noisy geometric guide frame sequence into a pre-trained video diffusion model. In each round of denoising iteration, a mask-based local spatial fusion is performed, so that the model only iterates and updates the target region of the eye, while the non-edited region always retains its original features, ultimately generating a highly realistic and temporally coherent open-eye sequence (denoised video clip).

[0118] In some embodiments, the denoising process of the noisy geometric guide frame sequence to generate a denoised video segment, wherein local spatial fusion is performed based on the mask of the target editing region during the denoising process, so that the generation process only updates the target editing region and locks the original features of the non-editing region, includes: inputting the noisy geometric guide frame sequence into a pre-trained video diffusion model; performing local spatial fusion based on the mask of the target editing region in each denoising iteration of the video diffusion model, so that the non-editing region always maintains its original features, and only the target editing region is iteratively updated; iteratively executing the denoising process until a preset condition is met to obtain a denoised latent feature sequence; and converting the denoised latent feature sequence back to pixel space to generate a denoised video segment.

[0119] In some embodiments, performing local spatial fusion based on the mask of the target editing region includes: obtaining the input latent representation and the predicted output latent representation of the current denoising iteration step; extracting the mask region latent features corresponding to the target editing region from the predicted output latent representation; fusing the mask region latent features into the mask region corresponding to the input latent representation to replace the original latent features of the mask region; for non-editing regions outside the mask region, retaining the original latent features in the input latent representation as the input for the next round of denoising iteration, wherein the original latent features in the input latent representation originate from the latent features of the geometric guide frame sequence.

[0120] In some embodiments, inputting the noisy geometrically guided frame sequence into a pre-trained video diffusion model further includes: inputting textual prompts, mask space conditions, and complete visual context as conditional information into the video diffusion model to guide the generation process; the textual prompts are used to guide the generation of content in the target editing region at the semantic level; the mask space conditions are used to indicate the frame range that the video diffusion model needs to reconstruct and fill; and the complete visual context is used to ensure that the generated content is naturally connected to the preceding and following frames in the temporal dimension.

[0121] In some embodiments, the text prompt conditions include a text description for guiding the generation of an open-eye state in the target editing area.

[0122] In the process of denoising the video diffusion model, the potential sequence obtained above, which contains clear temporal anchors and noisy intermediate frames, along with the following condition information, is input into the video diffusion model for iterative denoising: (1) Text prompt condition: Input text descriptions such as "open eyes" and "natural, bright eyes" to strongly guide the generated target content at the semantic level. (2) Mask space condition: Provide a frame-level binary mask to clearly indicate the frame range that the video diffusion model needs to reconstruct and fill. (3) Complete visual context: In each step of denoising prediction, the video diffusion model can "see" the complete potential sequence, including clear first and last anchor frames and the current noisy intermediate frame, thereby ensuring that the generated content is naturally connected with the context (previous and subsequent frames) in terms of time and appearance; the natural connection means that the edited video meets at least one of the following objective characteristics in the time dimension: the state parameters of the target editing area between adjacent frames change continuously, the spatial position transitions smoothly, and the pixel brightness and chromaticity difference is less than a preset threshold. The video diffusion model, through its internal temporal attention, spatial attention and other mechanisms, integrates all conditions and gradually predicts and removes noise from the noisy potential representation. This process not only reconstructs realistic eye textures, lighting, and physiological details (such as iris texture and corneal reflection), but also ensures that the generated eye dynamics (such as subtle eye movements and eyelid tremors) are smooth and natural between frames.

[0123] In the mask-based video fusion process, a mask-based local spatial fusion operation is performed in each denoising iteration of the video diffusion model. Specifically, after generating the current latent representation in each denoising step, it is not directly used as the input for the next denoising step. Instead, spatially selective replacement is immediately performed: only the latent features corresponding to the mask region (eye mask region) of the target editing region in the prediction result (predicted output latent representation) are extracted and fused into the input latent representation of the current denoising iteration, replacing the corresponding part of the eye region. For the non-editing regions outside the mask (face and background regions), the original latent features corresponding to the input of the current denoising step (i.e., the encoded features from the geometric guide frame) are completely retained as the input for the next denoising iteration. The denoising process is iteratively executed until preset conditions are met (such as reaching the maximum number of denoising steps or the noise level being below a threshold), resulting in a denoised latent feature sequence. Subsequently, the latent feature sequence is converted back to pixel space by a VAE decoder to generate a denoised video segment. This strategy ensures that in each iteration of the denoising process, the non-target editing area is always strongly constrained and protected by the original geometric guidance information, thereby achieving pixel-level precise local editing effects in the final output, while maintaining the consistency between the character's identity features and the background content to the greatest extent.

[0124] This stage achieves high realism while ensuring the temporal consistency necessary for video editing. This primarily relies on the powerful spatiotemporal prior knowledge inherent in the pre-trained video diffusion model. By leveraging the model's ability to coherently model the motion and appearance changes of real-world objects, learned from a large amount of video data, and combined with the geometric deformation technology designed in this application, the generation process not only synthesizes highly realistic eye textures with rich details and natural lighting on a single frame, but also ensures that the generated continuous frame sequence maintains smoothness and coherence in dynamic details, eliminating temporal artifacts such as screen flickering and jitter, achieving a visual effect of "static realism and dynamic smoothness." Furthermore, this stage employs a "local editing, global condition" generation strategy. The model uses the complete facial image (including the unedited area that retains most of the original identity information) as the contextual condition for the generation process, performing controlled content synthesis only on the target eye area. This mechanism requires that the generated content be highly harmonious and consistent with the surrounding known identity features (such as skin texture and facial structure), thereby ensuring a high degree of consistency and authenticity between the non-eye areas and identity features of the character while achieving precise editing of the eye appearance.

[0125] Step 150: The denoised video segment is fused with the original video to obtain the edited video.

[0126] like Figure 1 , Figure 6 and Figure 7 The post-processing stage, as shown, takes the previously obtained edited cropped face sequence (denoised video clips) and, based on the frame-by-frame face bounding box information recorded in the data processing stage, seamlessly synthesizes it back into the original video through scaling and inverse transformation operations. The core objective of this stage is to completely preserve the non-face content and all dynamic information of the original video while modifying only the target editing area (eyes), ensuring that the final edited video output is visually seamless and without any editing traces.

[0127] In some embodiments, fusing the denoised video segment with the original video to obtain an edited video includes: obtaining bounding box information of the face region in each frame of the original video; scaling the edited face image of each frame in the denoised video segment to the corresponding original bounding box size according to the bounding box information of the face region, and placing it back into the bounding box position of the corresponding frame in the original video; and reassembling all processed frames in the original time sequence to generate the edited video.

[0128] In some embodiments, the face bounding box information is derived from the cross-frame stable face trajectory obtained when performing face detection and cropping on the original video.

[0129] In some embodiments, scaling each frame of the edited face image in the denoised video segment to the corresponding original bounding box size and placing it back into the bounding box position of the corresponding frame in the original video includes: adjusting each frame of the edited face image to its corresponding original bounding box size through scaling transformation; and accurately covering the bounding box area of ​​the corresponding frame in the original video with the scaled edited face image through inverse transformation.

[0130] This stage involves acquiring the face bounding box information of each frame of the original video, scaling the edited face image to its original size and accurately backfilling it into its original position, reassembling all frames according to the original time sequence, and finally outputting the complete edited video.

[0131] First, the face bounding box information for each frame of the original video is acquired and recorded during the data processing stage. This face bounding box information originates from the stable cross-frame face trajectories obtained during face detection and cropping of the original video, including the bounding box coordinates of the face region in each frame (such as the coordinates of the top left corner, width, and height). This information is persistently saved throughout the editing process, providing spatial positioning basis for accurate backfilling in the post-processing stage.

[0132] Then, each frame of the edited face image in the denoised video clip is scaled according to the size of the face bounding box of the corresponding original frame. Since the face images are cropped to a standardized size (e.g., 256×256 or 512×512) during data processing, the face bounding box size in each frame of the original video may vary (depending on the distance and angle of the face in the frame). Therefore, scaling is necessary to adjust the edited face image to a size that perfectly matches its original bounding box. Scaling algorithms can employ bilinear interpolation, bicubic interpolation, or deep learning-based super-resolution methods to ensure the quality of the scaled image.

[0133] Then, the scaled, edited face image is precisely overlaid onto the bounding box region of the corresponding frame in the original video using an inverse transform. Specifically, based on the coordinates of the original bounding box, each pixel of the edited face image is placed at the corresponding position in the original video frame. For the junction between the bounding box edge and the original background, smoothing processes such as feathering or alpha blending can be performed to eliminate any possible stitching artifacts and ensure a natural transition between the edited area and the original background.

[0134] Then, all the processed frames are reassembled in their original time sequence to generate the complete edited video. Since the editing operation only applies to frames within the target segment (the closed-eye segment), and frames outside the target segment remain unchanged, it is only necessary to replace the corresponding frames in the original video with the edited frames. Other information in the original video, such as the audio track, remains unchanged and is synchronously merged into the output video.

[0135] Optionally, after the fusion is completed, a quality check can be performed to check whether the edited face image is consistent with the original background in terms of brightness, contrast, and color. If there are obvious differences, subsequent tone matching or lighting consistency adjustments can be made to ensure the visual quality of the final output.

[0136] Figure 6 This application demonstrates a cropped video face editing result with eyes open, based on an embodiment of the present application. Figure 7 This paper demonstrates an overall video face eye-opening editing result according to an embodiment of this application. The video face eye-opening editing algorithm based on a geometric deformation and diffusion model designed in this embodiment of the application has the following significant advantages: (1) Strong universality and identity independence: This application does not rely on specific person IDs or pre-collected personalized data. By performing fully automatic face detection, feature point analysis and temporal modeling on any input video, it can be applied to faces of different identities, genders, ages and appearances, and has good generalization ability and practical applicability.

[0137] (2) High visual realism and identity consistency: Precise spatial guidance is provided through geometric deformation, and local generation is performed based on a large-scale pre-trained video diffusion model. The synthesized open-eye area presents a high degree of visual realism in terms of texture, skin texture, lighting and anatomical structure. At the same time, the non-edited area is strictly constrained to remain unchanged during the generation process, and an identity preservation mechanism is introduced to ensure that the edited face is completely consistent with the original person's identity features, without identity drift or incongruity.

[0138] (3) Excellent temporal coherence and dynamic naturalness: This application introduces a temporal weighting strategy in the inter-frame fusion stage and explicitly models temporal consistency in the video diffusion model, so that the generated eye-opening sequence transitions smoothly in the time dimension, effectively avoiding problems such as screen flickering, shaking or discontinuous eye movements, and achieving a visual performance of "static realism and dynamic smoothness".

[0139] (4) Highly controllable localized editing: Through mask-guided noise injection and local fusion mechanism, this application strictly limits the editing operation to the target area of ​​the eye. While achieving high-fidelity editing, it ensures zero modification and zero degradation of other areas of the face and video background content, and achieves precise local semantic editing at the pixel level.

[0140] (5) Enhancement of functionality in practical application scenarios: This application can be widely applied in fields such as film and television post-production, video content creation, real-time video communication enhancement, and virtual digital human driving. In film and television post-production, it can efficiently repair lens flaws caused by actors blinking improperly or closing their eyes for a long time, greatly reducing the cost of manual frame-by-frame repair; in video conferencing and live streaming, it can correct the closed-eye state in real time, improving the professional performance of speakers or anchors and the audience's immersion; in virtual digital human generation, it can generate accurate and vivid eye animations for characters, enhancing emotional expression and interactive realism; in video content re-creation, it supports free editing and artistic adjustment of the eyes of characters in existing materials, expanding creative possibilities.

[0141] This application has significant advantages in terms of technical versatility, visual quality, temporal consistency, and editing controllability, and can provide related industries with an efficient, reliable, and high-quality automated video human eye state editing solution.

[0142] The video editing method provided in this application embodiment can be applied not only to the above-mentioned eye editing scenarios (closing eyes to opening eyes), but also to lip editing scenarios (closing mouth to opening mouth), eyebrow editing scenarios (such as eyebrow shape adjustment), facial expression editing scenarios (expression intensity adjustment), skin blemish repair scenarios (such as acne scar and mole removal), and eyeglasses accessory editing scenarios (such as adding / replacing eyeglasses).

[0143] In some embodiments, the target editing region is a mouth region, and the target state is an open mouth state. The step of performing target editing region attribute analysis on the aligned face sequence to determine target segments where the target editing region does not possess the target state includes: performing mouth attribute analysis on the aligned face sequence, calculating the open mouth coefficient for each frame in the aligned face sequence to obtain an open mouth coefficient sequence; determining closed mouth segments based on the temporal changes of each open mouth coefficient in the open mouth coefficient sequence, and using the closed mouth segments as the target segments.

[0144] In some embodiments, the step of performing mouth attribute analysis on the aligned face sequence and calculating the mouth opening coefficient of each frame in the aligned face sequence includes: performing facial feature point detection on each frame in the aligned face sequence to obtain the coordinates of key mouth points, wherein the key mouth points include at least the highest point of the upper lip, the lowest point of the lower lip, the left corner of the mouth, and the right corner of the mouth; calculating the vertical distance between the upper and lower lips and the horizontal distance between the left and right corners of the mouth based on the coordinates of the key mouth points; and using the ratio obtained by dividing the vertical distance by the horizontal distance as the mouth opening coefficient of each frame in the aligned face sequence.

[0145] In some embodiments, determining the closed segment based on the temporal change of the opening coefficient includes: performing temporal smoothing filtering on the opening coefficient sequence to obtain a smoothed opening coefficient sequence; determining the opening / closing state of each frame in the smoothed opening coefficient sequence according to preset opening and closing thresholds; identifying a complete event from an open state to a closed state and back to an open state by detecting state transitions on the time axis; taking the frame where the smoothed opening coefficient is first lower than the closing threshold and lasts for more than a first preset number of frames as the first frame of the closed segment, taking the frame where the smoothed opening coefficient is again higher than the opening threshold and lasts for more than a second preset number of frames as the last frame of the closed segment, and determining the continuous frame sequence between the first frame and the last frame as the closed segment.

[0146] Stable feature points for the mouth region can be selected from the tip of the nose, the left and right nostrils, and the chin contour. The calculation logic for the mouth opening coefficient is similar to that for the eye opening coefficient, based on the ratio of vertical distance to horizontal distance. This lip-reading editing scenario includes: correcting lip-reading asynchrony caused by a person accidentally closing their mouth in a video, adjusting the rhythm of a speech, and generating lip-reading animations.

[0147] In the context of lip-shaping editing, to avoid ambiguity or errors in the positioning of lip feature points caused by a closed mouth, this application excludes the feature points of the lip region itself and selects only stable and reliable feature points in the facial geometry as the basis for deformation, including but not limited to: the tip of the nose, the left and right nostril points, the chin contour point, and the left and right cheekbone points.

[0148] Using the correspondence between these facial stable region feature points in the previous reference frame (open-mouth reference frame) and the current target frame (closed-mouth frame) in the target segment, a first spatial transformation model (preferably thin-plate spline transformation) is calculated to align the deformation of the previous reference frame to the pose of the current target frame, generating a forward deformation result. The mouth structure of this forward deformation result is inherited from the previous reference frame and is precisely aligned with the facial geometry (excluding the mouth) of the current target frame. Similarly, based on the same facial stable region feature points, a second spatial transformation model is calculated from the subsequent reference frame (open-mouth reference frame) to the current target frame, generating a backward deformation result whose mouth structure is inherited from the subsequent reference frame. For the t-th frame in the target segment (closed-mouth segment), a fusion weight is calculated based on its temporal position in the segment, and an adaptive weighted fusion is performed on the forward and backward deformation results to generate a geometric guidance frame: Fusion weight calculation: w(t) = (tt) s ) / (t e -t s Geometric guided frame fusion: I guide (t) = (1-w(t))·I warpfront (t) + w(t)·I warpback(t); The generated geometric guide frame has a general spatial structure of the mouth opening state (such as the degree of mouth opening and closing, and lip contour) in the mouth region, and is geometrically aligned with the non-edited area of ​​the current target frame, providing accurate spatial guidance for the subsequent video diffusion model.

[0149] For lip-sync editing scenarios, in the denoising stage of the video diffusion model, the sequence of geometrically guided frames for the mouth region is transformed into the latent feature space using a VAE encoder. In the latent feature space, the geometrically guided frames corresponding to the first and last frames of the target segment (closed-mouth segment) are kept noise-free or injected with a small amount of noise, serving as temporal anchors. Gaussian noise is injected into the latent features of the geometrically guided frames in the middle frames, and the noise injection terminates prematurely when it reaches a preset medium noise level, forming truncated noise injection. The denoised latent sequence, along with the following conditional information, is input into the pre-trained video diffusion model: textual prompts: input textual descriptions such as "open mouth," "naturally closed lips," and "a smile showing teeth," guiding the generated mouth content at the semantic level; mask space conditions: providing frame-level binary masks of the mouth region, explicitly indicating the frame range that the model needs to reconstruct and fill; complete visual context: the model can "see" the complete latent sequence in each denoising prediction step, ensuring that the generated content is naturally connected to the context in terms of time and appearance. In each iteration of the video diffusion model's denoising process, a mask-based local spatial fusion is performed: latent features corresponding to the mouth region mask are extracted from the predicted output latent representation and fused into the corresponding mask region in the input latent representation, replacing the features of that region; for non-edited regions outside the mask (such as the nose, cheeks, chin, etc.), the original latent features in the input latent representation (encoded features from the geometric guide frame) are completely preserved. The denoising process is iteratively executed until preset conditions are met, and the denoised latent feature sequence is converted back to pixel space through a VAE decoder to generate a denoised video clip with a naturally opening and closing mouth that is coordinated with facial expressions.

[0150] For lip-sync editing scenarios, the post-processing stage involves acquiring the bounding box information of the face region in each frame of the original video. Each frame of the edited face image (with the mouth edited to an open state) in the denoised video clip is scaled and adjusted to the corresponding original bounding box size, and then precisely overlaid onto the bounding box region of the corresponding frame in the original video using an inverse transform. For the transition between the mouth region and the original background, smoothing processes such as alpha blending or feathering are performed to eliminate splicing artifacts. All processed frames are then reassembled in their original time sequence to generate the complete edited video, preserving all non-face content and dynamic information of the original video while only modifying the mouth state.

[0151] In some embodiments, the target editing region is an eyebrow region, and the target state is a target eyebrow shape state. The step of performing target editing region attribute analysis on the aligned face sequence to determine target segments where the target editing region does not possess the target state includes: performing eyebrow attribute analysis on the aligned face sequence, calculating the eyebrow shape feature vector for each frame in the aligned face sequence to obtain an eyebrow shape feature sequence; determining eyebrow deviation segments based on the temporal changes of each eyebrow shape feature vector in the eyebrow shape feature sequence, and using the eyebrow deviation segments as the target segments.

[0152] In some embodiments, the step of performing eyebrow attribute analysis on the aligned face sequence and calculating the eyebrow shape feature vector for each frame in the aligned face sequence includes: performing facial feature point detection on each frame of the aligned face sequence to obtain the coordinates of eyebrow key points, wherein the eyebrow key points include at least the eyebrow head point, the eyebrow peak point, and the eyebrow tail point; and calculating the eyebrow shape feature vector based on the eyebrow key point coordinates, wherein the eyebrow shape feature vector includes at least one of the eyebrow arch curvature, eyebrow peak height, and eyebrow tail angle.

[0153] Stable feature points for the eyebrow region can be selected from brow bone contour points, bridge of the nose points, forehead contour points, etc. The eyebrow shape feature vector can be compared with a preset target eyebrow shape template to determine whether editing is needed. This eyebrow editing scenario includes: adjusting eyebrow shape to match different makeup styles, correcting unnatural eyebrow shapes caused by facial expressions, and customizing eyebrow shapes for virtual characters.

[0154] For eyebrow editing scenarios, when performing geometric deformation of the eyebrow region, stable and reliable feature points in the facial geometry can be selected as the basis for deformation, including but not limited to: brow arch contour points, bridge of the nose points, forehead contour points, and left and right cheekbone points. Utilizing the correspondence between these stable facial feature points in the previous reference frame (target eyebrow shape reference frame) and the current target frame, a spatial transformation model (preferably thin plate spline transformation) is calculated to generate forward and backward deformation results. Then, based on the temporal position of the target frame in the target segment, the forward and backward deformation results are adaptively weighted and fused to generate a geometric guide frame. This geometric guide frame possesses the spatial structure of the target eyebrow shape in the eyebrow region (such as brow arch curvature, brow peak height, and brow tail angle) and is geometrically aligned with the non-editable area of ​​the current target frame.

[0155] For eyebrow editing scenarios, in the denoising stage of the video diffusion model, the geometrically guided frame sequence of the eyebrow region is transformed into a latent feature space. The first and last frames of the target segment are kept noise-free or injected with a small amount of noise as temporal anchors, while intermediate frames are truncated and noise is injected. The noisy latent sequence, along with the following conditional information, is input into the video diffusion model: textual prompts (e.g., "naturally curved eyebrows," "straight eyebrows," "high arched eyebrows"), mask space conditions (providing frame-level binary masks of the eyebrow region), and complete visual context (ensuring the generated content is consistent with facial expressions). In each iteration of the video diffusion model's denoising, mask-based local spatial fusion is performed, updating only the eyebrow mask region; non-edited regions (e.g., forehead, eyelids, bridge of the nose) retain their original features. After iterative denoising, a denoised video segment with natural eyebrow shape and reasonable hair flow direction is generated.

[0156] For eyebrow editing scenarios, in the post-processing stage, each frame of the edited face image (with the eyebrow shape adjusted to the target eyebrow shape) in the denoised video clip is scaled to the original bounding box size, precisely backfilled to the bounding box position of the corresponding frame in the original video, and reassembled according to the original time sequence to generate the edited complete video.

[0157] In some embodiments, the target editing region is a facial expression region, and the target state is a target expression state. The step of performing target editing region attribute analysis on the aligned face sequence to determine the target segment where the target editing region does not possess the target state includes: performing expression attribute analysis on the aligned face sequence, calculating the expression intensity coefficient of each frame in the aligned face sequence to obtain an expression intensity sequence; determining expression intensity deviation segments based on the temporal changes of each expression intensity coefficient in the expression intensity sequence, and using the expression intensity deviation segments as the target segments.

[0158] In some embodiments, the step of performing expression attribute analysis on the aligned face sequence and calculating the expression intensity coefficient of each frame in the aligned face sequence includes: performing facial feature point detection on each frame of the aligned face sequence to obtain the coordinates of expression-related key points defined by the facial motion coding system; and calculating the expression intensity coefficient based on the coordinates of the expression-related key points, wherein the expression intensity coefficient includes at least one of the following: jowl muscle displacement, orbicularis oculi muscle contraction degree, and intergalactor muscle compression degree.

[0159] The Facial Action Coding System (FACS) is an anatomical classification system used to objectively describe human facial expressions. It breaks down human facial expressions into a series of action units (AUs), each corresponding to a specific set of facial muscle movements.

[0160] Stable feature points for facial expression regions can be selected from the cheekbone, jawline, and bridge of the nose. The expression intensity coefficient can be quantified by the degree of difference from a neutral expression. This facial expression editing scenario includes: adjusting the speaker's expression intensity to match the emotional content, correcting unnatural expressions caused by nervousness, and driving virtual character expressions.

[0161] For facial expression editing scenarios, when performing geometric deformation of the facial expression region, stable and reliable feature points in the facial geometry can be selected as the basis for deformation, including but not limited to: cheekbone points, jaw points, bridge of the nose points, and the center point of the forehead. Utilizing the correspondence between these stable facial feature points in the previous reference frame (target expression reference frame) and the current target frame, a spatial transformation model is calculated to generate forward and backward deformation results. Adaptive weighted fusion is performed based on the temporal position of the target frame to generate a geometric guide frame. This geometric guide frame possesses the spatial structure of the target expression in the facial expression region (such as the displacement of the zygomaticus major muscle and the degree of contraction of the orbicularis oculi muscle) and is geometrically aligned with the non-editable region of the current target frame.

[0162] For facial expression editing scenarios, in the denoising stage of the video diffusion model, the geometrically guided frame sequence of the facial expression region is transformed into the latent feature space. The first and last frames of the target segment are kept noise-free or injected with a small amount of noise as temporal anchors, while intermediate frames are truncated and noise is injected. The noisy latent sequence, along with the following conditional information, is input into the video diffusion model: textual prompts (e.g., "natural smile," "calm expression," "joyful expression"); mask space conditions (providing frame-level binary masks of the facial expression region, which can be based on the Facial Action Coding System (FACS) definition); and complete visual context (ensuring the generated content is consistent with other facial regions). In each denoising iteration, mask-based local spatial fusion is performed, updating only the facial expression region, while non-edited regions (such as background, hairstyle, neck, etc.) retain their original features. After iterative denoising, a denoised video segment with appropriate expression intensity and natural dynamics is generated.

[0163] For facial expression editing scenarios, in the post-processing stage, each frame of the edited face image (with adjusted expression intensity) in the denoised video clip is scaled to the original bounding box size, precisely backfilled to the bounding box position of the corresponding frame in the original video, and reassembled according to the original time sequence to generate the edited complete video.

[0164] In some embodiments, the target editing region is a skin blemish region, and the target state is a flawless state. The step of performing target editing region attribute analysis on the aligned face sequence to determine the target segment where the target editing region does not have the target state includes: performing skin blemish detection on the aligned face sequence to determine the blemish region mask for each frame in the aligned face sequence; determining the blemish-containing segment based on the temporal change of the blemish region mask, and using the blemish-containing segment as the target segment.

[0165] In some embodiments, the step of performing skin blemish detection on the aligned face sequence and determining the blemish region mask for each frame in the aligned face sequence includes: performing semantic segmentation or object detection on each frame of the aligned face sequence to identify at least one blemish region among acne marks, moles, and scars; and generating a binary mask based on the identified blemish region as the blemish region mask.

[0166] The stable feature points for skin blemish areas can be selected from the feature points of healthy skin surrounding the blemish area. Geometric deformation can guide the texture of the surrounding healthy skin to the blemish area. This skin blemish repair scenario includes: removal of acne scars, mole removal, scar repair, and skin smoothing in videos.

[0167] For skin blemish repair scenarios, when performing geometric deformation on the blemish area, feature points from the surrounding healthy skin region can be selected as the basis for deformation. These include, but are not limited to, skin texture feature points around the blemish area and feature points from stable facial regions (such as cheekbone points, bridge of the nose points, and cheek contour points). Utilizing the correspondence between these feature points in the previous reference frame (blemish-free reference frame) and the current target frame, a spatial transformation model is calculated to guide the deformation of the corresponding healthy skin region in the previous reference frame to the blemish area. Since skin blemish areas are typically small, localized regions, a single reference frame deformation can be used without bidirectional weighted fusion to generate a geometric guide frame. This guide frame possesses the spatial structure of healthy skin within the blemish area and transitions naturally with the surrounding skin texture.

[0168] For skin blemish repair scenarios, in the denoising stage of the video diffusion model, the geometrically guided frame sequence of the skin blemish region is transformed into a latent feature space. The first and last frames of the target segment are kept noise-free or injected with a small amount of noise as temporal anchors, while intermediate frames are truncated and noise is injected. The noisy latent sequence, along with the following conditional information, is input into the video diffusion model: textual prompt conditions: input textual descriptions such as "smooth skin," "flawless skin," and "natural skin texture"; mask space conditions: providing frame-level binary masks of the skin blemish region (generated based on semantic segmentation or object detection); complete visual context: ensuring that the generated content is consistent with the surrounding skin texture, skin quality, and lighting. In each denoising iteration, mask-based local spatial fusion is performed, updating only the skin blemish region, while non-edited regions (such as healthy skin, facial features, and background) retain their original features. After iterative denoising, a denoised video segment with smooth skin and natural texture is generated.

[0169] For skin blemish repair scenarios, in the post-processing stage, each frame of the edited face image (with blemishes repaired) in the denoised video clip is scaled to the original bounding box size and precisely backfilled to the bounding box position of the corresponding frame in the original video. For the junction between the repaired area and the surrounding skin, smoothing processes such as alpha blending or feathering are performed to eliminate splicing artifacts. The edited complete video is then reassembled according to the original time sequence.

[0170] In some embodiments, the target editing region is the eye and nose bridge region, and the target state is the target accessory wearing state. The step of performing target editing region attribute analysis on the aligned face sequence to determine the target segment where the target editing region does not have the target state includes: performing accessory wearing detection on the aligned face sequence to determine the accessory wearing state of each frame in the aligned face sequence; determining the accessory missing segment based on the change of the accessory wearing state in the time domain, and taking the accessory missing segment as the target segment.

[0171] Stable feature points for the eye and nose areas can be selected from brow bone points, nose bridge points, and cheekbone points. Geometric deformation can align the reference frame with glasses on to the target frame without glasses on. This glasses accessory editing scenario includes: adding virtual glasses to a video character, changing glasses styles, and fixing lens blur caused by reflections.

[0172] For eyeglasses editing scenarios, during the geometric deformation stage, stable and reliable feature points in the facial geometry can be selected as the basis for deformation, including but not limited to: brow bone contour points, bridge of the nose points, and left and right cheekbone points. Utilizing the correspondence between these stable facial feature points in the previous reference frame (the reference frame for wearing the target glasses) and the current target frame, a spatial transformation model is calculated to generate forward and backward deformation results. Adaptive weighted fusion is performed based on the temporal position of the target frame to generate a geometric guide frame. This guide frame possesses the spatial structure of wearing glasses (such as frame position, temple angle, and lens reflection) in the eye and bridge of the nose areas and is geometrically aligned with the non-editable areas of the current target frame.

[0173] For the eyewear editing scenario, in the denoising stage of the video diffusion model, the geometrically guided frame sequence of the eyewear region is transformed into the latent feature space. The first and last frames of the target segment are kept noise-free or injected with a small amount of noise as temporal anchors, while intermediate frames are truncated and noise is injected. The noisy latent sequence, along with the following conditional information, is input into the video diffusion model: textual prompts (e.g., "metal-framed glasses," "thick-rimmed black glasses," "rimless glasses"); mask space conditions (providing frame-level binary masks for the eye and nose bridge regions); and complete visual context (ensuring the generated content is consistent with facial expressions and head posture). In each denoising iteration, mask-based local spatial fusion is performed, updating only the eyewear region, while non-edited regions (e.g., forehead, cheeks, background) retain their original features. After iterative denoising, a denoised video segment with natural-looking glasses that fits the face is generated.

[0174] For the eyewear editing scenario, in the post-processing stage, each frame of the edited face image (with added or replaced glasses) in the denoised video clip is scaled to the original bounding box size and precisely backfilled to the bounding box position of the corresponding frame in the original video. For the junction between the eyewear area and the facial skin, smoothing processes such as alpha blending are performed to ensure a natural blend. The edited complete video is then reassembled according to the original time sequence.

[0175] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0176] This application embodiment acquires the original video target segment and determines the target state reference frame. It uses facial stable region feature points to perform geometric deformation and adaptive fusion to generate a geometric guide frame sequence. Then, it performs selective noise addition and denoising processing (preserving the stable features of the first and last frames). Finally, it uses mask-guided local fusion technology to achieve precise editing of the target editing area and feature locking of the non-editing area. This enables precise editing of the open and closed state of the person's eyes in the video, while ensuring seamless connection between the target editing area and the original video in the spatiotemporal dimension. This effectively improves the visual naturalness and inter-frame continuity of the eye opening and closing state transition.

[0177] To facilitate better implementation of the video editing method of this application, this application also provides a video editing apparatus. Please refer to... Figure 8 , Figure 8 This is a schematic diagram of the structure of a video editing device provided in an embodiment of this application. The video editing device 200 may include: The first processing unit 210 is used to acquire the target segment to be edited in the original video and determine the target state reference frame; The second processing unit 220 is used to perform geometric deformation and adaptive fusion on each target frame in the target segment based on the facial stable region feature points of the target state reference frame and each target frame in the target segment, and generate a geometric guide frame sequence including the geometric guide frame corresponding to each target frame. The geometric guide frame has the spatial structure of the target state in the target editing region of the corresponding target frame and is geometrically aligned with the non-editing region of the corresponding target frame. The noise-adding unit 230 is used to add noise to the geometric guide frame corresponding to the middle frame of the target segment in the geometric guide frame sequence to obtain a noise-adding geometric guide frame sequence, wherein the geometric guide frames corresponding to the first and last frames of the target segment are kept noise-free or injected with a small amount of noise. Denoising unit 240 is used to denoise the noisy geometric guide frame sequence and generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region so that the generation process only updates the target editing region and locks the original features of the non-editing region. The editing unit 250 is used to merge the denoised video segment with the original video to obtain the edited video.

[0178] In some embodiments, the first processing unit 210 is configured to: perform face detection and cropping on the original video to obtain an aligned face sequence; perform target editing region attribute analysis on the aligned face sequence to determine a target segment whose target editing region does not have a target state; crop the original video according to the time information of the target segment to obtain the target segment; and determine, according to the time information of the target segment, frames before and after the target segment that have a target state as the target state reference frames.

[0179] In some embodiments, the target editing region is the eye region, and the target state is an open-eye state. The first processing unit 210 is configured to perform target editing region attribute analysis on the aligned face sequence to determine a target segment whose target editing region does not have a target state. This includes: performing target editing region attribute analysis on the aligned face sequence, calculating the open-eye coefficient of each frame in the aligned face sequence to obtain an open-eye coefficient sequence; determining a closed-eye segment based on the temporal change of each open-eye coefficient in the open-eye coefficient sequence, and using the closed-eye segment as the target segment.

[0180] In some embodiments, the first processing unit 210 is configured to perform target editing region attribute analysis on the aligned face sequence and calculate the eye-opening coefficient of each frame in the aligned face sequence, including: performing facial feature point detection on each frame in the aligned face sequence to obtain the coordinates of key eye points; calculating the vertical distance between the upper and lower edges of the eyelids and the horizontal distance between the left and right corners of the eyes based on the coordinates of the key eye points; and using the ratio obtained by dividing the vertical distance by the horizontal distance as the eye-opening coefficient of each frame in the aligned face sequence.

[0181] In some embodiments, the first processing unit 210 is configured to determine a closed-eye segment based on the change of the eye-opening coefficient in the time domain, including: performing temporal smoothing filtering on the eye-opening coefficient sequence to obtain a smoothed eye-opening coefficient sequence; determining the open / closed state of each frame corresponding to each smoothed eye-opening coefficient in the smoothed eye-opening coefficient sequence according to a preset eye-opening threshold and a closed-eye threshold; identifying a complete event from an open-eye state to a closed-eye state and back to an open-eye state by detecting state transitions on the time axis; taking the frame where the smoothed eye-opening coefficient is first lower than the closed-eye threshold and the duration exceeds a first preset number of frames as the first frame of the closed-eye segment, taking the frame where the smoothed eye-opening coefficient is again higher than the open-eye threshold and the duration exceeds a second preset number of frames as the last frame of the closed-eye segment, and determining the continuous frame sequence between the first frame and the last frame as the closed-eye segment.

[0182] In some embodiments, the first processing unit 210 is configured to perform face detection and cropping on the original video to obtain an aligned face sequence, including: performing face detection on each frame of the original video to obtain bounding box information of the face region, the bounding box information including bounding box coordinates; performing exponential motion smoothing on the bounding box coordinates to obtain a stable face trajectory across frames; and cropping each frame of the original video according to the face trajectory to obtain an aligned face sequence centered on the face and with normalized dimensions.

[0183] In some embodiments, the first processing unit 210 is configured to crop the original video according to the time information of the target segment to obtain the target segment, including: extracting a continuous sub-video sequence from the first frame to the last frame from the original video according to the timestamps of the first and last frames of the target segment; and cropping the face region of each frame in the continuous sub-video sequence according to the bounding box information of the face trajectory within the corresponding time period of the continuous sub-video sequence to obtain the spatiotemporally aligned target segment.

[0184] In some embodiments, the first processing unit 210 is configured to determine, based on the time information of the target segment, frames that have the target state before and after the target segment as the target state reference frames, including: taking frames that have the target state in one or more frames before the first frame of the target segment as the front reference frame; taking frames that have the target state in one or more frames after the last frame of the target segment as the back reference frame; and determining the front reference frame and the back reference frame as the target state reference frames.

[0185] In some embodiments, the second processing unit 220 is configured to: extract facial stable region feature points from the target state reference frame and each target frame in the target segment, wherein the facial stable region feature points include at least one of the following: brow bone contour points, bridge of nose and tip of nose points, and outer contour points of cheek and chin; based on the facial stable region feature points, perform geometric deformation on the previous reference frame and the next reference frame in the target state reference frame respectively to obtain forward deformation results and backward deformation results; and perform adaptive weighted fusion of the forward deformation results and backward deformation results according to the time position of each target frame in the target segment to generate the geometric guidance frame.

[0186] In some embodiments, the second processing unit 220 is configured to obtain a mask of the target editing region of each target frame in the target segment before extracting the facial stable region feature points of the target state reference frame and each target frame in the target segment, wherein the mask is used to limit the range of geometric deformation and generation operations to the target editing region.

[0187] In some embodiments, the second processing unit 220 is used to obtain the mask of the target editing region of each target frame in the target segment, including: constructing an initial target editing region polygon based on eyelid contour key points predicted by a face feature point detection model; and performing a morphological dilation operation on the initial target editing region polygon to obtain a mask covering the target editing region.

[0188] In some embodiments, the second processing unit 220 is configured to perform geometric deformation on the front reference frame and the back reference frame in the target state reference frame based on the facial stable region feature points, respectively, to obtain forward deformation results and backward deformation results, including: for each target frame in the target segment, calculating a first spatial transformation model based on the correspondence between the facial stable region feature points in the front reference frame and the target frame, and deforming and aligning the front reference frame to the pose of the target frame through the first spatial transformation model to generate a forward deformation result; calculating a second spatial transformation model based on the correspondence between the facial stable region feature points in the back reference frame and the target frame, and deforming and aligning the back reference frame to the pose of the target frame through the second spatial transformation model to generate a backward deformation result.

[0189] In some embodiments, the second processing unit 220 is configured to adaptively weight and fuse the forward deformation result and the backward deformation result according to the time position of each target frame in the target segment to generate the geometric guide frame, including: determining a fusion weight according to the time position of each target frame in the target segment, wherein the fusion weight is positively correlated with the time distance of the target frame from the first frame of the target segment and negatively correlated with the time distance of the target frame from the last frame of the target segment; and weighting and fusing the forward deformation result and the backward deformation result according to the fusion weight to generate the geometric guide frame.

[0190] In some embodiments, the noise-adding unit 230 is configured to: convert the geometric guide frame sequence to a latent feature space; in the latent feature space, keep the latent features of the geometric guide frames corresponding to the first and last frames of the target segment noise-free or inject a small amount of noise as time anchor points; inject noise into the latent features of the geometric guide frames corresponding to the middle frames of the target segment to obtain a noise-adding geometric guide frame sequence.

[0191] In some embodiments, the noise injection unit 230 is used to inject noise into the potential features of the geometric guide frame corresponding to the intermediate frame of the target segment, including: injecting Gaussian noise into the potential features of the geometric guide frame corresponding to the intermediate frame according to a preset noise scheduling table, and terminating the noise injection in advance when a preset medium noise level is reached, forming a truncated noise injection.

[0192] In some embodiments, the denoising unit 240 is configured to: input the noisy geometric guide frame sequence into a pre-trained video diffusion model; in each denoising iteration of the video diffusion model, perform local spatial fusion based on the mask of the target editing region, so that the non-editing region always retains its original features, and only the target editing region is iteratively updated; iteratively execute the denoising process until a preset condition is met to obtain a denoised latent feature sequence; and convert the denoised latent feature sequence back to pixel space to generate a denoised video segment.

[0193] In some embodiments, the denoising unit 240 is configured to perform local spatial fusion based on the mask of the target editing region, including: obtaining the input latent representation and the predicted output latent representation of the current denoising iteration step; extracting the mask region latent features corresponding to the target editing region from the predicted output latent representation; fusing the mask region latent features into the mask region corresponding to the input latent representation to replace the original latent features of the mask region; for non-editing regions outside the mask region, retaining the original latent features in the input latent representation as the input for the next round of denoising iteration, wherein the original latent features in the input latent representation originate from the latent features of the geometric guide frame sequence.

[0194] In some embodiments, the denoising unit 240, used to input the noisy geometrically guided frame sequence into a pre-trained video diffusion model, further includes: inputting textual prompts, mask space conditions, and complete visual context as conditional information into the video diffusion model to guide the generation process; the textual prompts are used to guide the generation of content in the target editing region at the semantic level; the mask space conditions are used to indicate the frame range that needs to be reconstructed and filled by the video diffusion model; and the complete visual context is used to ensure that the generated content is naturally connected to the preceding and following frames in the time dimension.

[0195] In some embodiments, the editing unit 250 is configured to: acquire bounding box information of the face region in each frame of the original video; scale the edited face image of each frame in the denoised video segment to the corresponding original bounding box size according to the bounding box information of the face region, and place it back into the bounding box position of the corresponding frame in the original video; and reassemble all processed frames in the original time sequence to generate the edited video.

[0196] It should be noted that the functions of each module in the video editing device 200 in this application embodiment can be referred to the specific implementation of any embodiment in the above method embodiments, and will not be repeated here.

[0197] Each unit in the above-described device can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0198] For example, the video editing device 200 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the video editing device 200 can be the terminal or server.

[0199] In some embodiments, this application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0200] Figure 9 A schematic diagram of the structure of the computer device provided in the embodiments of this application, such as... Figure 9 As shown, the computer device 300 may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. The communication interface 301, memory 302, and processor 303 communicate with each other via the communication bus 304. The communication interface 301 is used for data communication between the device 300 and external devices. The memory 302 can be used to store software programs and modules, and the processor 303 runs the software programs and modules stored in the memory 302, such as the software programs for the corresponding operations in the aforementioned method embodiments.

[0201] In some embodiments, the processor 303 may invoke software programs and modules stored in the memory 302 to perform the following operations: acquire the target segment to be edited in the original video and determine the target state reference frame; based on the target state reference frame and the facial stable region feature points of each target frame in the target segment, perform geometric deformation and adaptive fusion on each target frame in the target segment to generate a geometric guide frame sequence including geometric guide frames corresponding to each target frame, wherein the geometric guide frame has the spatial structure of the target state in the target editing region of the corresponding target frame and is geometrically aligned with the non-editing region of the corresponding target frame; perform noise processing on the geometric guide frames corresponding to the middle frames of the target segment in the geometric guide frame sequence to obtain a noisy geometric guide frame sequence, wherein the geometric guide frames corresponding to the first and last frames of the target segment remain noiseless or are injected with a small amount of noise; perform denoising processing on the noisy geometric guide frame sequence to generate a denoised video segment, wherein local spatial fusion is performed based on the mask of the target editing region during the denoising process, so that the generation process only updates the target editing region and locks the original features of the non-editing region; and fuse the denoised video segment with the original video to obtain the edited video.

[0202] In some embodiments, the computer device 300 may be integrated into a terminal or server that has storage and a processor and thus computing power, or the computer device 300 may be the terminal or server.

[0203] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the methods described above in the embodiments of this application; for brevity, further details are omitted here.

[0204] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0205] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described above in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0206] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0207] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0208] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0209] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0210] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0211] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0212] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0214] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0215] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video editing method, characterized in that, The method includes: Obtain the target segment to be edited from the original video and determine the target state reference frame; Based on the target state reference frame and the facial stable region feature points of each target frame in the target segment, geometric deformation and adaptive fusion are performed on each target frame in the target segment to generate a geometric guide frame sequence including the geometric guide frame corresponding to each target frame. The geometric guide frame has the spatial structure of the target state in the target editing region of the corresponding target frame and is geometrically aligned with the non-editing region of the corresponding target frame. The geometric guidance frames corresponding to the middle frames of the target segment in the geometric guidance frame sequence are subjected to noise processing to obtain a noisy geometric guidance frame sequence, wherein the geometric guidance frames corresponding to the first and last frames of the target segment are kept noise-free or injected with a small amount of noise. The denoised geometric guide frame sequence is denoised to generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region, so that the generation process only updates the target editing region and locks the original features of the non-editing region. The denoised video clip is then merged with the original video to obtain the edited video.

2. The video editing method as described in claim 1, characterized in that, The step of acquiring the target segment to be edited in the original video and determining the target state reference frame includes: The original video is subjected to face detection and cropping to obtain an aligned face sequence; Target editing region attribute analysis is performed on the aligned face sequence to determine that the target editing region does not have target segments with target states; The original video is cropped based on the time information of the target segment to obtain the target segment; Based on the time information of the target segment, frames that have the target state before and after the target segment are determined as the target state reference frames.

3. The video editing method as described in claim 2, characterized in that, The target editing region is the eye region, the target state is an open-eye state, and the step of performing target editing region attribute analysis on the aligned face sequence to determine target segments whose target editing regions do not possess the target state includes: Target editing region attribute analysis is performed on the aligned face sequence, and the eye-opening coefficient of each frame in the aligned face sequence is calculated to obtain the eye-opening coefficient sequence; Based on the change of each eye-opening coefficient in the eye-opening coefficient sequence in the time domain, a closed-eye segment is determined, and the closed-eye segment is taken as the target segment.

4. The video editing method as described in claim 3, characterized in that, The step of performing target editing region attribute analysis on the aligned face sequence and calculating the eye-opening coefficient of each frame in the aligned face sequence includes: Facial feature point detection is performed on each frame of the aligned face sequence to obtain the coordinates of key eye points; Based on the coordinates of the key eye points, calculate the vertical distance between the upper and lower edges of the eyelids and the horizontal distance between the left and right corners of the eyes; The ratio obtained by dividing the vertical distance by the horizontal distance is used as the eye-opening coefficient for each frame in the aligned face sequence.

5. The video editing method as described in claim 3, characterized in that, The step of determining the closed-eye segment based on the change of the eye-opening coefficient in the time domain includes: The eye-opening coefficient sequence is subjected to time-series smoothing filtering to obtain a smoothed eye-opening coefficient sequence; Based on the preset eye-opening threshold and eye-closing threshold, the eye-opening and eye-closing states of each frame corresponding to each smoothed eye-opening coefficient in the smoothed eye-opening coefficient sequence are judged. By detecting state transitions on the timeline, the complete event of transitioning from an open-eye state to a closed-eye state and then back to an open-eye state can be identified. The frame in which the smoothed eye-opening coefficient is first lower than the eye-closing threshold and lasts for more than a first preset number of frames is taken as the first frame of the closed-eye segment. The frame in which the smoothed eye-opening coefficient is again higher than the eye-opening threshold and lasts for more than a second preset number of frames is taken as the last frame of the closed-eye segment. The continuous frame sequence between the first frame and the last frame is determined as the closed-eye segment.

6. The video editing method as described in claim 2, characterized in that, The step of performing face detection and cropping on the original video to obtain an aligned face sequence includes: Face detection is performed on each frame of the original video to obtain the bounding box information of the face region, the bounding box information including the bounding box coordinates; The bounding box coordinates are subjected to exponential motion smoothing to obtain a stable face trajectory across frames; Based on the facial trajectory, each frame in the original video is cropped to obtain an aligned facial sequence centered on the face and with normalized dimensions.

7. The video editing method as described in claim 6, characterized in that, The step of cropping the original video based on the time information of the target segment to obtain the target segment includes: Based on the timestamps of the first and last frames of the target segment, extract a continuous sub-video sequence from the first frame to the last frame from the original video; Based on the bounding box information of the face trajectory within the corresponding time period of the continuous sub-video sequence, the face region is cropped in each frame of the continuous sub-video sequence to obtain the spatiotemporally aligned target segment.

8. The video editing method as described in claim 2, characterized in that, The step of determining, based on the time information of the target segment, frames that possess the target state before and after the target segment as the target state reference frames includes: The frame containing the target state in one or more frames preceding the first frame of the target segment is used as the previous reference frame. The frame containing the target state in one or more frames after the last frame of the target segment is used as the subsequent reference frame. The previous reference frame and the subsequent reference frame are determined as the target state reference frame.

9. The video editing method as described in claim 1, characterized in that, The step involves performing geometric deformation and adaptive fusion on each target frame in the target segment based on the facial stable region feature points of the target state reference frame and each target frame in the target segment, generating a geometric guidance frame sequence including geometric guidance frames corresponding to each target frame, including: Extract facial stable region feature points from the target state reference frame and each target frame in the target segment. The facial stable region feature points include at least one of the following: brow bone contour points, bridge of nose and tip of nose points, and outer contour points of cheeks and chin. Based on the feature points of the facial stable region, geometric deformation is performed on the front reference frame and the back reference frame in the target state reference frame to obtain the forward deformation result and the backward deformation result. Based on the temporal position of each target frame in the target segment, the forward deformation result and the backward deformation result are adaptively weighted and fused to generate the geometric guidance frame.

10. The video editing method as described in claim 9, characterized in that, Based on the feature points of the stable facial region, geometric deformation is performed on the front and back reference frames of the target state reference frame to obtain forward deformation results and backward deformation results, including: For each target frame in the target segment, based on the correspondence between the facial stable region feature points in the previous reference frame and the target frame, a first spatial transformation model is calculated, and the previous reference frame is deformed and aligned to the pose of the target frame through the first spatial transformation model to generate a forward deformation result; Based on the correspondence between the facial stable region feature points in the back reference frame and the target frame, a second spatial transformation model is calculated, and the back reference frame is deformed and aligned to the pose of the target frame through the second spatial transformation model to generate a backward deformation result.

11. The video editing method as described in claim 9, characterized in that, The step of adaptively weighting and fusing the forward deformation result and the backward deformation result according to the time position of each target frame in the target segment to generate the geometric guidance frame includes: The fusion weight is determined based on the temporal position of each target frame in the target segment, wherein the fusion weight is positively correlated with the temporal distance of the target frame from the first frame of the target segment and negatively correlated with the temporal distance of the target frame from the last frame of the target segment; The forward deformation result and the backward deformation result are weighted and fused according to the fusion weight to generate the geometric guide frame.

12. The video editing method as described in claim 1, characterized in that, The step of adding noise to the geometric guidance frame corresponding to the intermediate frame of the target segment in the geometric guidance frame sequence to obtain a noisy geometric guidance frame sequence includes: Transform the geometrically guided frame sequence into a latent feature space; In the latent feature space, the latent features of the geometric guide frames corresponding to the first and last frames of the target segment are kept noise-free or injected with a small amount of noise, serving as time anchor points; Noise is injected into the latent features of the geometric guide frame corresponding to the intermediate frame of the target segment to obtain a noisy geometric guide frame sequence.

13. The video editing method as described in claim 12, characterized in that, The latent feature injection noise into the geometric guide frame corresponding to the intermediate frame of the target segment includes: According to a preset noise scheduling table, Gaussian noise is injected into the potential features of the geometric guide frame corresponding to the intermediate frame, and the noise injection terminates prematurely when it reaches a preset medium noise level, forming a truncated noise injection.

14. The video editing method as described in claim 1, characterized in that, The denoising process is performed on the noisy geometric guide frame sequence to generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region, ensuring that the generation process only updates the target editing region and locks the original features of the non-editing region. This includes: The noisy geometric guide frame sequence is input into a pre-trained video diffusion model; In each denoising iteration of the video diffusion model, local spatial fusion is performed based on the mask of the target editing region, so that the non-editing region always retains its original features, and only the target editing region is iteratively updated; The denoising process is iteratively executed until the preset conditions are met, resulting in a denoised latent feature sequence. The denoised latent feature sequence is converted back to pixel space to generate a denoised video clip.

15. The video editing method as described in claim 14, characterized in that, The local spatial fusion based on the mask of the target editing region includes: Obtain the input latent representation and the predicted output latent representation for the current denoising iteration step; Extract the mask region latent features corresponding to the target editing region from the latent representation of the predicted output; The latent features of the masked region are fused into the corresponding masked region in the input latent representation to replace the original latent features of the masked region; For the non-editable regions outside the masked region, the original latent features in the input latent representation are retained as input for the next round of denoising iteration. The original latent features in the input latent representation are derived from the latent features of the geometric guide frame sequence.

16. The video editing method as described in claim 14, characterized in that, The step of inputting the noisy geometrically guided frame sequence into the pre-trained video diffusion model further includes: The text prompts, mask space conditions, and complete visual context are input together as conditional information into the video diffusion model to guide the generation process; The text prompts are used to guide the generation of content in the target editing area at the semantic level; Mask space conditions are used to indicate the frame range that the video diffusion model needs to reconstruct and fill; A complete visual context is used to ensure that the generated content flows naturally with the preceding and following frames in the time dimension.

17. A video editing device, characterized in that, The device includes: The first processing unit is used to acquire the target segment to be edited in the original video and determine the target state reference frame; The second processing unit is used to perform geometric deformation and adaptive fusion on each target frame in the target segment based on the facial stable region feature points of the target state reference frame and each target frame in the target segment, and generate a geometric guide frame sequence including the geometric guide frame corresponding to each target frame. The geometric guide frame has the spatial structure of the target state in the target editing region of the corresponding target frame and is geometrically aligned with the non-editing region of the corresponding target frame. The noise-adding unit is used to add noise to the geometric guide frame corresponding to the middle frame of the target segment in the geometric guide frame sequence to obtain a noise-adding geometric guide frame sequence, wherein the geometric guide frames corresponding to the first and last frames of the target segment are kept noise-free or injected with a small amount of noise. A denoising unit is used to denoise the noisy geometric guide frame sequence and generate a denoised video segment. During the denoising process, local spatial fusion is performed based on the mask of the target editing region, so that the generation process only updates the target editing region and locks the original features of the non-editing region. The editing unit is used to merge the denoised video segment with the original video to obtain the edited video.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the video editing method as described in any one of claims 1-16.

19. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the video editing method as described in any one of claims 1-16 by calling the computer program stored in the memory.

20. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the video editing method according to any one of claims 1-16.

Images