Video stabilization method, system and equipment based on diffusion model and storage medium

Abstract

The invention discloses a video stabilization method, system and device based on a diffusion model, and a storage medium, which are corresponding schemes, and in the scheme, a video stabilization network based on the diffusion model is constructed by utilizing the generation capability of the diffusion model to an image, and the video stabilization network is used for stabilizing the image. The method is used for accurately recovering boundary missing pixels caused by large camera motion, shielding or parallax and remarkably improving video stability, adjacent frame information is used as a condition to guide a denoising process of a diffusion model, a space edge compensation branch is designed, and clear and complete boundary content is generated; based on aligning each frame along a stable trajectory in a potential space, a time motion guide branch is designed, and camera shake is effectively eliminated. According to the method, on the premise that the original view is kept, artifacts are remarkably reduced, the perception quality is improved, and a plurality of quantitative performance indexes reach the industry leading level.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video stabilization method, system, device, and storage medium based on a diffusion model. Background Technology

[0002] Video stabilization, as a fundamental computer vision task, has made significant progress in recent years. It has been widely applied in handheld shooting devices, drones, and robotics. Technically, video stabilization methods typically involve three main steps: 1) estimating camera motion from unstable video, 2) smoothing camera trajectories, and 3) generating stable frames. First, in the motion estimation stage, motion can be estimated using 2D feature detection, optical flow, mesh methods, or homography transformation. Second, the camera trajectory smoothing stage aims to smooth the camera motion, predict the spatial transformation between unstable and stable frames, and stabilize each frame using homography transformation, hybrid homography transformation, or pixel-wise deformation fields. Third, in the stable frame generation stage, the predicted spatial transformation is used to synthesize a stable video. However, in video clips with fast camera movement, stable frames often exhibit large gaps in edge regions. Therefore, existing methods often require aggressive cropping strategies to maintain the rectangular shape of the frames, leading to loss of original content and reduced video resolution.

[0003] In contrast, deep learning-based methods can directly stabilize videos by inputting both stable and unstable videos, without requiring explicit motion estimation and trajectory smoothing steps. However, due to the lack of explicit motion estimation and trajectory modeling processes, the models struggle to perceive the long-term motion trends of the camera, ultimately resulting in visually lower quality videos compared to traditional methods.

[0004] Unlike two-dimensional methods, three-dimensional stabilization methods can simulate 3D camera trajectories and stabilize frames through epipolar projection. Some methods rely solely on the video itself, deforming the image through 3D reconstruction, structure-of-motion (SfM), epigeometry, stabilized rendering, or subspace projection; others introduce dedicated hardware to assist in acquiring scene structure, such as depth cameras, light field cameras, gyroscopes, and inertial measurement unit (IMU) sensors. Despite their sophisticated design, these methods still face challenges in handling scenes with multiple dynamic targets and often have limited field of view (FoV).

[0005] However, most existing methods rely on discriminative models that infer solely from observed inputs. Due to occlusion and parallax effects, boundary regions often lack reliable information, making it difficult for models to generate plausible edge content and resulting in suboptimal results at image edges.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a video stabilization method, system, device, and storage medium based on a diffusion model, which can improve the effect of video stabilization and generate clear, full-frame stabilized video frames.

[0008] The objective of this invention is achieved through the following technical solution: A video stabilization method based on a diffusion model includes: Collect training video data containing multiple data pairs, where each data pair consists of an unstable frame and its corresponding stable frame; A video stabilization neural network model based on a diffusion model is constructed, comprising a temporal motion guidance branch, a spatial edge compensation branch, and a frame reconstruction branch. The model is trained using the training video data. The steps include: generating noisy features corresponding to stable frames in each data pair using the temporal motion guidance branch, incorporating sampled random noise; transforming and encoding unstable frames in each data pair using the temporal motion guidance branch to obtain an unstable frame encoding representation; performing optical flow estimation using the transformed unstable frames and neighboring frames in the spatial edge compensation branch, and obtaining features for edge compensation based on the optical flow estimation results; and calculating noise using the noisy features corresponding to stable frames, the unstable frame encoding representation, and the features for edge compensation in the frame reconstruction branch, and constructing a loss function using the random noise sampled by the temporal motion guidance branch to train the video stabilization neural network model. Based on the trained video stabilization neural network model, the video sequence composed of unstable input frames is processed by back-diffusion to gradually obtain the corresponding encoded representation of the denoised stable frame for each unstable frame, and finally decoded to obtain the stable video sequence.

[0009] A video stabilization system based on a diffusion model, used to implement the aforementioned method, includes: The data collection unit is used to collect training video data containing multiple data pairs, where each data pair consists of an unstable frame and its corresponding stable frame. The model building and training unit is used to construct a video stabilization neural network model based on a diffusion model. It includes a temporal motion-guided branch, a spatial edge compensation branch, and a frame reconstruction branch. The model is trained using the training video data. The steps include: generating noisy features corresponding to stable frames in each data pair using the temporal motion-guided branch, incorporating sampled random noise; transforming and encoding unstable frames in each data pair using the temporal motion-guided branch to obtain an unstable frame encoding representation; performing optical flow estimation using the transformed unstable frames and neighboring frames in the spatial edge compensation branch, and obtaining features for edge compensation based on the optical flow estimation results; and calculating noise using the noisy features corresponding to stable frames, the unstable frame encoding representation, and the features for edge compensation in the frame reconstruction branch, and constructing a loss function using the random noise sampled by the temporal motion-guided branch to train the video stabilization neural network model. The video stabilization unit is used to obtain the corresponding encoded representation of the denoised stable frame for each unstable frame in a video sequence composed of input unstable frames by back-diffusion based on the trained video stabilization neural network model, and finally decodes to obtain a stable video sequence.

[0010] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0011] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0012] As can be seen from the technical solution provided by the present invention, a video stabilization network based on the diffusion model is constructed by utilizing the image generation capability of the diffusion model. This network is used to accurately recover boundary missing pixels caused by large camera movement, occlusion, or parallax, and to significantly improve video stability. Furthermore, by using information from adjacent frames as conditions to guide the denoising process of the diffusion model, a spatial edge compensation branch is designed to generate clear and complete boundary content. Based on aligning each frame along a stable trajectory in the latent space, a temporal motion guidance branch is designed to effectively eliminate camera shake. The present invention significantly reduces artifacts and improves perceptual quality while maintaining the original field of view, achieving industry-leading levels in multiple quantitative performance indicators. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a video stabilization method based on a diffusion model provided in an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of a video stabilization method based on a diffusion model provided in an embodiment of the present invention.

[0016] Figure 3 This is a schematic diagram of a video stabilization system based on a diffusion model, provided as an embodiment of the present invention.

[0017] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0020] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0021] The following provides a detailed description of a video stabilization method, system, device, and storage medium based on a diffusion model provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Unless otherwise specified, specific conditions in the embodiments of this invention are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0022] Example 1 This invention provides a video stabilization method based on a diffusion model, such as... Figure 1 As shown, the main steps include the following: Step 1: Collect training data.

[0023] In this embodiment of the invention, training video data containing multiple data pairs are collected, and each data pair consists of an unstable frame and a corresponding stable frame.

[0024] Taking the kth data pair as an example, it contains the kth unstable frame and its corresponding stable frame. Wherein, the k-th unstable video frame is represented as The corresponding target stable frame is represented as The set of frames in an unstable video is represented as The set of stable video frames is represented as M is the number of frames.

[0025] Step 2: Construct a video stabilization neural network model based on the diffusion model and train it.

[0026] In this embodiment of the invention, the video stabilization neural network model based on the diffusion model mainly includes: a temporal motion guidance branch, a spatial edge compensation branch, and a frame reconstruction branch. The training steps include: using the temporal motion guidance branch, generating noisy features corresponding to the stable frames in each data pair by combining sampled random noise; transforming and encoding the unstable frames in each data pair using the temporal motion guidance branch to obtain an unstable frame coded representation; the spatial edge compensation branch performing optical flow estimation using the transformed unstable frames and neighboring frames, and obtaining features for edge compensation by combining the optical flow estimation results; and the frame reconstruction branch calculating noise using the noisy features corresponding to the stable frames, the unstable frame coded representation, and the features for edge compensation, and constructing a loss function using the random noise sampled by the temporal motion guidance branch to train the video stabilization neural network model.

[0027] In this embodiment of the invention, the step of generating the noise-adding feature corresponding to the stable frame in each data pair by combining the sampled random noise includes: (1) For the k-th frame, the corresponding stable frame is denoted as Features are encoded by an autoencoder. (2) Sample a random noise from a standard normal distribution. ,in, The standard normal distribution is represented by 0, where 0 is the mean of the standard normal distribution. (3) Randomly select a time step t, with the corresponding predefined parameter being: The noise-adding feature is then expressed as .

[0028] In this embodiment of the invention, the step of transforming and encoding unstable frames in each data pair through time-motion guided branch to obtain an unstable frame encoded representation includes: (1) For the k-th frame, the corresponding unstable frame is denoted as Combined with the motion prior field from the unstable frame to the stable frame in the k-th frame conduct Transformation (Image Distortion Transformation): ;in, The k-th frame after transformation is an unstable frame. for Transformation; (2) Using an image encoder to transform Encode to obtain the corresponding encoded representation. .

[0029] In this embodiment of the invention, the step of using the transformed unstable frame and neighboring frames to perform optical flow estimation, and combining the optical flow estimation results to obtain features for edge compensation includes: (1) For the k-th frame, the corresponding unstable frame is denoted as The set of neighboring frame indices of the k-th frame is represented as... For each adjacent frame Optical flow estimation is performed using an optical flow algorithm to obtain the optical flow of each adjacent frame. Unstable frame after transformation (k-th frame) Optical flow field between (2) Utilizing optical flow field , each adjacent frame Align to the k-th unstable frame after transformation The space in which it is located is represented as: SA represents spatial alignment; and calculates each adjacent frame. Corresponding occlusion mask (3) The spatially aligned adjacent frames and the corresponding occlusion masks are concatenated along the channel dimension to obtain the features used for edge compensation. , represented as: .

[0030] In this embodiment of the invention, the loss function is expressed as: ; in, Let k be the loss function, and k be the k-th frame. For the random noise corresponding to the k-th frame, For the noise feature corresponding to the k-th frame, The features are obtained by encoding the stable frame corresponding to the k-th frame using an autoencoder. These are predefined parameters corresponding to time step t; The features used for edge compensation corresponding to the k-th frame, This is the encoded representation of the unstable frame at frame k. For frame reconstruction branch.

[0031] Based on the above loss function, gradient descent is used for training, and the network parameters are updated.

[0032] Step 3: Video stabilization.

[0033] In this embodiment of the invention, based on the trained video stabilization neural network model, the video sequence composed of input unstable frames is processed by back-diffusion to gradually obtain the corresponding encoded representation of the denoised stable frame for each unstable frame, and finally decoded to obtain a stable video sequence.

[0034] In this embodiment of the invention, for the r-th to Nr-th frames in the video sequence composed of unstable input frames, random noise is sampled from a standard normal distribution for initialization. Where j represents the j-th frame, j=r,…,Nr, r is the fixed window length of neighboring frames, and N is the total number of frames in the video sequence. The encoding representation for initialization, The standard normal distribution is represented by 0, where 0 is the mean of the standard normal distribution. The standard deviation of the standard normal distribution;

[0035] Starting from time step T, the process iterates until time step 1 is reached, where T is the maximum set time step. For the j-th unstable frame, the processing at time step t includes: The corresponding unstable frame coding representation is obtained using the same method as the training process. and features for edge compensation And calculate the noise ; where t=1,…,T.

[0036] Then, noise is sampled from a standard normal distribution. The following formula is used for noise reduction: ; in, The encoded representation of the input up to time step t-1, The encoded representation of the input up to time step t, when t=T, .

[0037] The process continues until time step t=1, at which point the denoised encoded representation is obtained. .

[0038] Each frame is processed using the same procedure to obtain a set of denoised encoded representations.

[0039] Those skilled in the art will understand that, due to the insufficient number of neighboring frames, the frames from frame 1 to frame r-1 and from frame N-r+1 to frame N are not processed.

[0040] In this embodiment of the invention, the stable video sequence obtained by decoding is represented as: ; in, Let be the denoised encoded representation of the j-th frame. Let J be the stable frame corresponding to the j-th frame, and N be the total number of frames in the video sequence. For decoders.

[0041] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0042] I. Overall Overview of the Plan

[0043] This invention provides a video stabilization method based on a diffusion model. This method is the first to use a diffusion model in video stabilization processing, achieving accurate recovery of missing pixels at boundaries caused by significant camera movement, occlusion, or parallax, and significantly improving video stability. Specifically, a spatial edge compensation branch is introduced, using information from adjacent frames as conditions to guide the diffusion model's denoising process, generating clear and complete boundary content. A temporal motion guidance branch is introduced, aligning frames along a stabilization trajectory in the latent space, effectively eliminating camera shake. This invention employs supervised training, integrating the diffusion model into the video stabilization network, improving the stabilization effect of the output frames. This invention demonstrates state-of-the-art performance in three video stabilization benchmarks. Experimental results show that, while maintaining the original field of view, this invention significantly reduces artifacts and improves perceptual quality, achieving industry-leading levels in multiple quantitative performance metrics.

[0044] II. Detailed introduction of the plan.

[0045] 1. Obtain training video data.

[0046] In this embodiment of the invention, the training video data includes a set of unstable video frames. With stable video frame set Among them, the k-th frame is an unstable frame. With the corresponding stable frame Constituting data pairs M is the number of frames.

[0047] Based on the above training video data, using the motion prior field from unstable frames to stable frames obtained from existing video stabilization methods, the motion prior field from the k-th unstable frame to its corresponding stable frame is denoted as... Then the set of prior fields of motion is represented as .

[0048] 2. Construct a video stabilization neural network.

[0049] In this embodiment of the invention, the video stabilization neural network includes a temporal motion guidance branch, a spatial edge compensation branch, and a frame reconstruction branch. During training, a forward diffusion mode is used, and during inference, a backward diffusion mode is used.

[0050] (1) Time motion guides the branch.

[0051] (1.1) The target stable frame of the kth frame Encoded into features by an autoencoder: And sample a random noise from a standard normal distribution. .

[0052] (1.2) Randomly select a time step Then the features after adding noise at time step t can be expressed as: ,in These are predefined parameters.

[0053] (1.3) Use the following formula to analyze the unstable video frame k. and the motion prior field from the unstable frame to the stable frame in the k-th frame conduct Transformation: .

[0054] (1.4) Utilizing a lightweight image encoder right The k-th unstable video frame after transformation Encoding is performed by the lightweight image encoder. The specific structure is as follows.

[0055] Lightweight image encoder First, remove the unstable video frames. The input is fed into two ResNet (residual network) modules with a stride of 1, increasing the number of channels from 3 to 8, and then from 8 to 16. Next, a ResNet module with a stride of 2 expands the number of channels from 16 to 32. Then, it passes through two more ResNet modules with a stride of 1, decreasing the number of channels from 32 to 16, and then from 16 to 8. Finally, a convolutional layer compresses the output to 3 channels, yielding the encoded representation used for subsequent feature alignment. .

[0056] (2) Spatial edge compensation branch.

[0057] The spatial edge compensation branch pair The k-th unstable video frame after transformation Using the RAFT optical flow algorithm, the optical flow of each adjacent frame is calculated. (in , (representing the set of neighboring frame indices of the k-th unstable video frame) to the target frame Optical flow field between Next, through Transformation will connect adjacent frames Twist Align to Target Frame In the space where it is located, the corresponding occlusion mask is calculated for each adjacent frame, where In adjacent frames, valid pixels are marked as 1, and occluded or missing pixels are marked as 0. Finally, the spatially aligned adjacent frames and their corresponding occlusion masks are concatenated along the channel dimension using the following formula to obtain the features of the adjacent frames (features used for edge compensation). .

[0058] .

[0059] (3) Frame reconstruction branch.

[0060] The frame reconstruction branch uses the ControlNet framework and leverages the features of adjacent frames. Unstable frames after motion guidance Boundary completion is performed, and then a denoising network (e.g., a U-Net network can be selected) is used to generate stable video frames with a complete field of view. The video stabilization neural network is trained using the gradient descent method of the following formula, and the loss function is calculated to update the network parameters. When the number of training iterations reaches the set number or the loss converges, the training stops, thus obtaining the optimal video stabilization model; it is used to process unstable video images to obtain the corresponding stable video images.

[0061] .

[0062] Figure 2In the middle, noise-adding features and unstable frame coding representation First, they are concatenated into a single feature, denoted as feature. And then They are all input into ControlNet, and ControlNet first processes them. Encode the code, and record the encoding result as follows: And then Encode the code, and record the encoding result as follows: Next and After addition, the result is fed into a network with the same structure as the right half of the denoising network U-Net for training. The training result is then used to initialize the right half of the denoising network U-Net. The denoising network U-Net is not trained at all; only ControlNet is trained and the weights of the right half of U-Net are updated.

[0063] The principle of the U-Net denoising network is to first predict the noise and then perform denoising processing. Its specific structure can be referred to conventional techniques. Here is a brief introduction: The U-Net denoising network is a symmetric network. The left half is a series of residual blocks that are downsampled. The right half is a series of residual blocks that progressively upsample the features extracted from the left half (downsampled) and combine them with the features of the corresponding skip connections, and finally output the noise prediction result.

[0064] 3. Video stabilization processing.

[0065] In this embodiment of the invention, the back-diffusion mode and the forward-diffusion mode share parameters. Similarly, it is implemented using a video stabilization neural network, which involves processing of the temporal motion guidance branch, the spatial edge compensation branch and the frame reconstruction branch. In addition, a denoising network (e.g., U-Net network) is also introduced.

[0066] In this embodiment of the invention, for frames r to Nr, where r is the fixed window length of neighboring frames and N is the total number of video frames, random noise is sampled from a standard normal distribution for initialization using the following formula: .

[0067] In this embodiment, time steps t are traversed sequentially from T to 1, and processing is performed from frame r to frame Nr.

[0068] Taking frame j as an example, the steps are as follows: (3.1) Use the following formula to analyze the unstable video frame j. With the sports pilot field (The motion prior field of the inference process is obtained using other models that estimate the motion prior field only through unstable frames.) Transformation: ; in, The j-th frame after transformation is an unstable frame.

[0069] (3.2) Utilizing a trained lightweight image encoder The unstable frame j after transformation is obtained by the following formula. Encode the data to obtain the encoded representation used for subsequent feature alignment: .

[0070] (3.3) Calculate the optical flow for each adjacent frame using the pre-trained RAFT optical flow algorithm. (in , (representing the set of neighboring frame indices of the j-th unstable video frame) to the target frame Optical flow field between Next, through Transformation will connect adjacent frames Twist Align to Target Frame In the given space, the corresponding occlusion mask is then calculated for each adjacent frame. The spatially aligned adjacent frames and the occlusion mask are then concatenated along the channel using the following formula: .

[0071] (3.4) In the frame reconstruction branch, the noise of the current frame at time step t is calculated using the following formula: ; in, This is the encoded representation of the output at time step t+1. If time step t+1 = T, it is obtained through the aforementioned initialization; otherwise, it is the output of the previous time step.

[0072] Then utilize the noise of the current frame at time step t Encoded representation of time step t-1 This is applied to noise calculation at time step t-1: .

[0073] (3.5) Sampling from the standard normal distribution The following formula is used to denoise the current frame at step t, resulting in the encoded representation at time step t-1. This process is repeated until t=1 to obtain the denoised encoded representation. .

[0074] In this embodiment of the invention, the remaining unstable frames in the unstable video sequence are also processed in the above manner to obtain the corresponding denoised encoded representation. Finally, the decoder is used to decode the video to obtain a stable video sequence of all frames, represented as follows: ; in, Let be the denoised encoded representation of the j-th frame. Let J be the stable frame corresponding to the j-th frame, and N be the total number of frames in the video sequence. For decoders.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0076] Example 2 This invention also provides a video stabilization system based on a diffusion model, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes: The data collection unit is used to collect training video data containing multiple data pairs, where each data pair consists of an unstable frame and its corresponding stable frame. The model building and training unit is used to construct a video stabilization neural network model based on a diffusion model. It includes a temporal motion-guided branch, a spatial edge compensation branch, and a frame reconstruction branch. The model is trained using the training video data. The steps include: generating noisy features corresponding to stable frames in each data pair using the temporal motion-guided branch, incorporating sampled random noise; transforming and encoding unstable frames in each data pair using the temporal motion-guided branch to obtain an unstable frame encoding representation; performing optical flow estimation using the transformed unstable frames and neighboring frames in the spatial edge compensation branch, and obtaining features for edge compensation based on the optical flow estimation results; and calculating noise using the noisy features corresponding to stable frames, the unstable frame encoding representation, and the features for edge compensation in the frame reconstruction branch, and constructing a loss function using the random noise sampled by the temporal motion-guided branch to train the video stabilization neural network model. The video stabilization unit is used to obtain the corresponding encoded representation of the denoised stable frame for each unstable frame in a video sequence composed of input unstable frames by back-diffusion based on the trained video stabilization neural network model, and finally decodes to obtain a stable video sequence.

[0077] Since the processing details of each unit in this system have been described in detail in previous embodiments, they will not be repeated here.

[0078] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0079] Example 3 The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0080] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0081] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0082] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0083] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0084] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A video stabilization method based on a diffusion model, characterized in that, include: Collect training video data containing multiple data pairs, where each data pair consists of an unstable frame and its corresponding stable frame; A video stabilization neural network model based on a diffusion model is constructed, comprising a temporal motion guidance branch, a spatial edge compensation branch, and a frame reconstruction branch. The model is trained using the training video data. The steps include: generating noisy features corresponding to stable frames in each data pair using the temporal motion guidance branch, incorporating sampled random noise; transforming and encoding unstable frames in each data pair using the temporal motion guidance branch to obtain an unstable frame encoding representation; performing optical flow estimation using the transformed unstable frames and neighboring frames in the spatial edge compensation branch, and obtaining features for edge compensation based on the optical flow estimation results; and calculating noise using the noisy features corresponding to stable frames, the unstable frame encoding representation, and the features for edge compensation in the frame reconstruction branch, and constructing a loss function using the random noise sampled by the temporal motion guidance branch to train the video stabilization neural network model. Based on the trained video stabilization neural network model, the video sequence composed of unstable input frames is processed by back-diffusion to gradually obtain the corresponding encoded representation of the denoised stable frame for each unstable frame, and finally decoded to obtain the stable video sequence.

2. The video stabilization method based on a diffusion model according to claim 1, characterized in that, The method of combining sampled random noise to generate the noisy features corresponding to the stable frame in each data pair includes: For the k-th frame, the corresponding stable frame is denoted as Features are encoded by an autoencoder. ; Sample random noise from a standard normal distribution. ,in, The standard normal distribution is represented by 0, where 0 is the mean of the standard normal distribution. The standard deviation of the standard normal distribution; Randomly select a time step t, the corresponding predefined parameter is The noise-adding feature is then expressed as .

3. The video stabilization method based on a diffusion model according to claim 1, characterized in that, The process of transforming and encoding unstable frames in each data pair through time-guided motion to obtain an encoded representation of the unstable frames includes: For the k-th frame, the corresponding unstable frame is denoted as Combined with the motion prior field from the unstable frame to the stable frame in the k-th frame conduct Transformation: ； in, The k-th frame after transformation is an unstable frame. for Transformation; Using an image encoder Encode to obtain the corresponding encoded representation. .

4. The video stabilization method based on a diffusion model according to claim 1, characterized in that, The step of using the transformed unstable frame and neighboring frames to perform optical flow estimation, and combining the optical flow estimation results to obtain features for edge compensation, includes: For the k-th frame, the corresponding unstable frame is denoted as The set of neighboring frame indices of the k-th frame is represented as... For each adjacent frame Optical flow estimation is performed using an optical flow algorithm to obtain the optical flow of each adjacent frame. Unstable frame after transformation (k-th frame) Optical flow field between ; Utilizing optical flow field , each adjacent frame Align to the k-th unstable frame after transformation The space in which it is located is represented as: SA represents spatial alignment; and calculates each adjacent frame. Corresponding occlusion mask ; Spatially aligned adjacent frames and their corresponding occlusion masks are concatenated along the channel dimension to obtain features for edge compensation. , represented as: .

5. The video stabilization method based on a diffusion model according to claim 1, characterized in that, The loss function is expressed as: ； in, Let k be the loss function, and k be the k-th frame. For the random noise corresponding to the k-th frame, For the noise feature corresponding to the k-th frame, The features are obtained by encoding the stable frame corresponding to the k-th frame using an autoencoder. These are predefined parameters corresponding to time step t; The features used for edge compensation corresponding to the k-th frame, This is the encoded representation of the unstable frame at frame k. For frame reconstruction branch.

6. The video stabilization method based on a diffusion model according to claim 1, characterized in that, The trained video stabilization neural network model, for the input video sequence composed of unstable frames, uses back-diffusion to progressively obtain the corresponding encoded representation of the denoised stable frame for each unstable frame, including: For the video sequence consisting of unstable input frames, frames r to Nr are initialized by sampling random noise from a standard normal distribution. Where j represents the j-th frame, j=r,…,Nr, r is the fixed window length of neighboring frames, and N is the total number of frames in the video sequence. The encoding representation for initialization, The standard normal distribution is represented by 0, where 0 is the mean of the standard normal distribution. The standard deviation of the standard normal distribution; Starting from time step T, the process iterates until time step 1 is reached, where T is the maximum set time step. For the j-th unstable frame, the processing at time step t includes: The corresponding unstable frame coding representation is obtained using the same method as the training process. and features for edge compensation And calculate the noise Where t=1,…,T; Then, noise is sampled from a standard normal distribution. The following formula is used for noise reduction: ； in, The encoded representation of the input up to time step t-1, The encoded representation of the input up to time step t, when t=T, ; The process continues until time step t=1, at which point the denoised encoded representation is obtained. ; Each frame is processed using the same procedure to obtain a set of denoised encoded representations.

7. A video stabilization method based on a diffusion model according to claim 6, characterized in that, The decoded stable video sequence is represented as: ； in, Let be the denoised encoded representation of the j-th frame. For the j-th frame, For decoders.

8. A video stabilization system based on a diffusion model, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The data collection unit is used to collect training video data containing multiple data pairs, where each data pair consists of an unstable frame and its corresponding stable frame. The model building and training unit is used to construct a video stabilization neural network model based on a diffusion model. It includes a temporal motion-guided branch, a spatial edge compensation branch, and a frame reconstruction branch. The model is trained using the training video data. The steps include: generating noisy features corresponding to stable frames in each data pair using the temporal motion-guided branch, incorporating sampled random noise; transforming and encoding unstable frames in each data pair using the temporal motion-guided branch to obtain an unstable frame encoding representation; performing optical flow estimation using the transformed unstable frames and neighboring frames in the spatial edge compensation branch, and obtaining features for edge compensation based on the optical flow estimation results; and calculating noise using the noisy features corresponding to stable frames, the unstable frame encoding representation, and the features for edge compensation in the frame reconstruction branch, and constructing a loss function using the random noise sampled by the temporal motion-guided branch to train the video stabilization neural network model. The video stabilization unit is used to obtain the corresponding encoded representation of the denoised stable frame for each unstable frame in a video sequence composed of input unstable frames by back-diffusion based on the trained video stabilization neural network model, and finally decodes to obtain a stable video sequence.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images