Video correction method, device, equipment, medium and product

Abstract

The invention discloses a video correction method and device, equipment, a medium and a product, and belongs to the related technical field of video processing. The method comprises the following steps: inputting a to-be-corrected video into a diffusion generation model, and performing correction processing operation on the to-be-corrected video by the diffusion generation model to obtain a corrected video of the to-be-corrected video; the correction processing operation comprises the following steps: for a first video frame of the to-be-corrected video, acquiring spatial layout information of the first video frame and short-time dynamic information of a subsequent video frame of the first video frame through a context path module, and generating a first condition guide vector of the first video frame based on the spatial layout information and the short-time dynamic information; the unified motion structure embedding module performs inter-frame smoothing processing on the first video frame to obtain a second condition guide vector of the first video frame; a diffusion generation module generates a corrected video frame based on the first conditional steering vector and the second conditional steering vector. According to the invention, the VR video processing effect can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video correction method, device, equipment, medium and product. BACKGROUND

[0002] With the rapid development of virtual reality (VR) technology, VR video plays a crucial role in immersive experience. However, existing VR video generation and processing technologies face many challenges in practical applications.

[0003] Existing VR video has jitter phenomenon, structural misplacement and motion inconsistency due to high-frequency motion of dynamic scenes, camera calibration errors or inaccurate inter-frame registration. Therefore, a video correction method is needed to improve the VR video processing effect.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the embodiments of the present application is to provide a video correction method, device, equipment, medium and product, which aims to improve the VR video processing effect.

[0006] In a first aspect, the embodiments of the present application provide a video correction method, which comprises:

[0007] inputting a to-be-corrected video into the diffusion generation model, performing a correction processing operation on the to-be-corrected video through the diffusion generation model, and obtaining a corrected video of the to-be-corrected video;

[0008] The correction processing operation comprises:

[0009] For a first video frame of the to-be-corrected video, the spatial layout information of the first video frame and the short-term dynamic information of the subsequent video frame of the first video frame are obtained through the context path module, and a first conditional guidance vector of the first video frame is generated based on the spatial layout information and the short-term dynamic information; the first conditional vector is used to represent the context features of the first video frame;

[0010] The first video frame is subjected to inter-frame smoothing processing through the unified motion structure embedding module, and a second conditional guidance vector of the first video frame is obtained; the second conditional guidance vector is used to represent the spatio-temporal perception ability of the first video frame;

[0011] The diffusion generation module generates a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector.

[0012] In a second aspect, an embodiment of the present application provides a video correction device, the device comprising: a correction module configured to input a to-be-corrected video into the diffusion generation model, and perform a correction processing operation on the to-be-corrected video by using the diffusion generation model to obtain a corrected video of the to-be-corrected video.

[0013] The correction processing operation comprises:

[0014] For a first video frame of the to-be-corrected video, the spatial layout information of the first video frame and the short-time dynamic information of a subsequent video frame of the first video frame are obtained by using the context path module, and a first conditional guidance vector of the first video frame is generated based on the spatial layout information and the short-time dynamic information; the first conditional guidance vector is used to represent the context feature of the first video frame.

[0015] The first video frame is subjected to inter-frame smoothing processing by using the unified motion structure embedding module to obtain a second conditional guidance vector of the first video frame; the second conditional guidance vector is used to represent the spatio-temporal perception ability of the first video frame.

[0016] The diffusion generation module is used to generate a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector.

[0017] In a third aspect, an embodiment of the present application provides a video correction device, the device comprising: a memory, a processor, and a computer processing program stored in the memory and executable on the processor, the computer processing program being configured to implement the video correction method of the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, the storage medium storing a computer processing program, the computer processing program being executed by a processor to implement the video correction method of the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, the program product being stored in a storage medium, and the program product being executed by at least one processor to implement the video correction method of the first aspect.

[0020] The application provides a video correction method, device, equipment, medium and product. The application inputs a to-be-corrected video into a diffusion generation model, performs a correction processing operation on the to-be-corrected video through the diffusion generation model, and obtains a corrected video of the to-be-corrected video. The correction processing operation includes: obtaining spatial layout information of a first video frame of the to-be-corrected video and short-time dynamic information of a subsequent video frame of the first video frame through a context path module, and generating a first conditional guidance vector of the first video frame based on the spatial layout information and the short-time dynamic information; the first conditional guidance vector is used to represent the context feature of the first video frame; performing inter-frame smoothing processing on the first video frame through a unified motion structure embedding module to obtain a second conditional guidance vector of the first video frame; the second conditional guidance vector is used to represent the spatio-temporal perception ability of the first video frame; and generating a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector through a diffusion generation module. In this way, the spatial layout information of the VR image and the short-time dynamic information of the subsequent video frame are perceived through the context path module to provide prior knowledge of static scene structure and dynamic motion for correction, solve the structure misplacement problem, and improve the spatial and temporal consistency of the generated content. At the same time, the inter-frame smoothing processing is performed on the first video frame through the unified motion structure embedding module to suppress the jitter of the generated content and ensure the motion consistency. At the model structure level, the context path module and the unified motion structure embedding module are embedded into each stage of the diffusion network of the diffusion generation module in a lightweight manner, which can reduce the inference delay while maintaining the high precision of the VR video, meet the real-time processing requirement, and improve the processing effect of the VR video. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 FIG. 1 is a flowchart of a video correction method provided by an embodiment of the application;

[0022] Figure 2 FIG. 3 is a schematic diagram of the overall implementation architecture of the diffusion generation model provided by an embodiment of the application;

[0023] Figure 3 FIG. 5 is a comparison diagram of the video correction method provided by an embodiment of the application and other methods;

[0024] Figure 4 FIG. 6 is a structural diagram of the video correction method and device provided by an embodiment of the application;

[0025] Figure 5 FIG. 7 is a structural diagram of an electronic device provided by an embodiment of the application;

[0026] Figure 6 FIG. 8 is another structural diagram of an electronic device provided by an embodiment of the application;

[0027] The objectives, functional features and advantages of the present application will be further described with reference to the embodiments and in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0029] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the front and rear associated objects.

[0030] The video correction method provided by the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings, specific embodiments and application scenarios.

[0031] The video correction method provided by the embodiments of the present application can include the following steps: Figure 1 As shown in the figure, the video correction method of the present application can include the following steps:

[0032] Step 101, inputting a to-be-corrected video into the diffusion generation model, performing a correction processing operation on the to-be-corrected video through the diffusion generation model, and obtaining a corrected video of the to-be-corrected video.

[0033] The correction processing operation includes:

[0034] For a first video frame of the to-be-corrected video, the spatial layout information of the first video frame and the short-time dynamic information of the subsequent video frame of the first video frame are obtained through the context path module, and the first conditional guidance vector of the first video frame is generated based on the spatial layout information and the short-time dynamic information; the first conditional vector is used to represent the context features of the first video frame;

[0035] The first video frame is processed by the unified motion structure embedding module for inter-frame smoothing to obtain a second conditional steering vector of the first video frame. The second conditional steering vector is used to represent the spatio-temporal perception ability of the first video frame.

[0036] The diffusion generation module generates a corrected video frame of the first video frame based on the first conditional steering vector and the second conditional steering vector.

[0037] In some embodiments, in step 101, each frame of the video to be corrected and its time sequence information are input into the diffusion generation model. Through the overall correction processing of the diffusion generation model, a more stable, clear and visually acceptable corrected video is generated.

[0038] It should be noted that the video to be corrected can be processed by multiple modules to achieve repair and optimization, and solve problems such as jitter, structural misplacement and inconsistent motion in the video to be corrected.

[0039] In some implementations, before inputting the video, the video can be pre-processed, such as adjusting the resolution, frame rate or performing preliminary noise filtering.

[0040] In some implementations, other models (such as pre-trained feature extraction models) can be combined to enhance the input features of the diffusion generation model.

[0041] The first video frame is a part of the video frame of the video to be corrected. The first video frame is only used as an example to illustrate the correction method of the video frame in the video to be corrected. Any video frame in the video to be corrected can be processed by the correction method.

[0042] The context path module can be a neural network module for extracting spatial and temporal context information of the video frame. The context path module can include a global context modeling module and a local temporal consistency modeling module. The spatial layout information can be static information such as the position, shape and distribution of objects in the video frame. The short-time dynamic information can be the motion change information between the current video frame and the subsequent several frames, which is used to capture the local motion trend. The first conditional steering vector can be a feature vector for representing the context features of the current video frame, including spatial layout and short-time dynamic information.

[0043] In some embodiments, the context path module can include a global context modeling module and a local dynamic consistency modeling module. The global context modeling module extracts the overall spatial layout and scene structure information of the input image through downsampling processing, enhances the system's perception of the overall content distribution in high-resolution images, and ensures the stability of the generated results in terms of visual composition and perspective coherence. The local dynamic consistency modeling module models the dynamic change trend in the short sequence for the time frames after the first video frame to capture the motion trajectory and local detail changes of the object, and improves the continuity of the generated image in the time dimension. The context features extracted by the above two modules are fused (first conditional guidance vector) and injected into the multi-stage generation process of the diffusion model in a conditional manner to realize joint guidance of spatial structure and temporal dynamics.

[0044] In other embodiments, the context path module can include a network and a model. The spatial layout information of the first frame is extracted using a pre-trained semantic segmentation network (such as DeepLab) or a monocular depth estimation model (such as MiDaS). The optical flow between the first frame and the subsequent N frames (such as N = 3) is calculated by a lightweight optical flow network (such as PWC-Net) to aggregate short-term dynamic information. The spatial layout information and short-term dynamic information are fused to obtain the first conditional guidance vector.

[0045] In some implementations, an attention mechanism can be introduced to weight the spatial layout information and short-term dynamic information, highlighting important feature parts.

[0046] In other implementations, multi-scale feature extraction can be used to obtain local and global spatial layout information simultaneously.

[0047] The unified motion structure embedding module can be a neural network module for processing the motion relationship between video frames to achieve inter-frame smoothing. Inter-frame smoothing can be achieved by adjusting the motion relationship between video frames to reduce jitter or unnatural transitions. The second conditional guidance vector can be a feature vector representing the spatiotemporal perception ability of the current video frame, i.e., the processing result of the inter-frame motion. The second conditional guidance vector can be used to represent the fused and encoded optical flow information and edge features.

[0048] In some embodiments, the unified motion structure embedding module can include an optical flow estimation module and a motion compensation module. First, the optical flow field between the first video frame and the previous and subsequent frames is calculated by the optical flow estimation module to represent the motion direction and size of the pixels. Then, the first video frame is adjusted by the motion compensation module to make the motion relationship with the previous and subsequent frames smoother. Finally, the processed frame features are extracted as the second conditional guidance vector.

[0049] In some implementations, the unified motion structure embedding module can further include a light flow estimation module and a motion compensation module. First, the light flow field between video frames in the first video frames is calculated by the light flow estimation module, representing the motion direction and size of the pixels. Then, the first video frames are adjusted by the motion compensation module to make the motion relationship between the front and back frames inside more smooth. Finally, the processed frame features are extracted as the second conditional guidance vector.

[0050] In some implementations, the unified motion structure embedding module can further model the first video frames as dynamic NeRF, jointly optimize geometry and motion through implicit representation. The features extracted from the optimized NeRF are rendered in time sequence consistent frames as the second conditional guidance vector.

[0051] In some implementations, the unified motion structure embedding module can first predict the inter-frame motion trajectory of the first video frames using Kalman filtering to obtain a preliminary smoothing result. The residual jitter is fine-tuned by a small CNN (such as MobileNetV3), and the corrected features are extracted as the second conditional guidance vector.

[0052] In some implementations, the motion prediction module in deep learning can be combined to predict the future frame motion, further optimizing the inter-frame smoothing effect.

[0053] In some implementations, the recurrent neural network (RNN) or Transformer structure can be introduced to enhance the modeling capability of long-time motion relationship.

[0054] The diffusion generation module can be a diffusion process-based generation module for generating corrected video frames according to the input conditional vector.

[0055] In some implementations, the output of the unified motion structure embedding module and the output of the context path module are input into the diffusion generation module as conditional guidance signals, and then the corrected video frames are generated; the output of the unified motion structure embedding module and the output of the context path module can also be reused in multiple stages (such as cross-layer sharing or feature fusion).

[0056] In some implementations, the output of the unified motion structure embedding module and the output of the context path module are input into the diffusion generation module as conditional guidance signals, and then the corrected video frames are generated; the output of the unified motion structure embedding module and the output of the context path module can also be reused in multiple stages (such as cross-layer sharing or feature fusion).

[0057] In some implementations, a multi-scale diffusion process can be introduced to correct video frames of different resolutions at the same time.

[0058] In some implementations, the quality of the generated video frame can be further improved in combination with a generative adversarial network (GAN).

[0059] It should be noted that the diffusion generative model can be a deep learning model based on a diffusion process, which generates or restores data by gradually eliminating noise. In video processing, the diffusion generative model can correct by learning the spatiotemporal features of the video. The corrected video frame can be a video frame with improved quality and optimized motion relationship after processing.

[0060] In the embodiments of the present application, the to-be-corrected video is input into the diffusion generative model, the to-be-corrected video is input into the diffusion generative model, and the diffusion generative model performs a correction processing operation on the to-be-corrected video to obtain a corrected video of the to-be-corrected video. The correction processing operation includes: for a first video frame of the to-be-corrected video, obtaining spatial layout information of the first video frame and short-term dynamic information of subsequent video frames of the first video frame through a context path module, and generating a first conditional guidance vector of the first video frame based on the spatial layout information and the short-term dynamic information. The first conditional vector is used to represent the context features of the first video frame. The first video frame is subjected to inter-frame smoothing processing through a unified motion structure embedding module to obtain a second conditional guidance vector of the first video frame. The second conditional guidance vector is used to represent the spatiotemporal perception ability of the first video frame. The diffusion generative module generates a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector. In this way, the context path module perceives the spatial layout information of the VR image and the short-term dynamic information of the subsequent video frames to provide prior knowledge of static scene structure and dynamic motion for correction, solve the structure misplacement problem, and improve the spatial and temporal consistency of the generated content. At the same time, the unified motion structure embedding module performs inter-frame smoothing processing on the first video frame to suppress the jitter of the generated content and ensure the motion consistency. At the model structure level, the context path module and the unified motion structure embedding module are embedded into each stage of the diffusion network of the diffusion generative module in a lightweight manner, which can reduce the inference delay while maintaining the high precision of the VR video, meet the real-time processing requirements, and improve the processing effect of the VR video.

[0061] In some embodiments, the diffusion generative module adopts a multi-stage U-shaped structure and performs nested diffusion denoising based on a Mamba block. The diffusion generative module generates the corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector, including:

[0062] The first condition guidance vector and the second condition guidance vector are embedded into each stage of the diffusion generation module to obtain a corrected video frame of the first video frame.

[0063] The diffusion generation module performs iterative denoising reconstruction of the image based on a multi-stage U-Net diffusion network (U-shaped hierarchical network (U-Net variant)). In each stage, a condition guidance vector embedding is introduced to improve the generation quality. Among them, Mamba block is used to replace the traditional Transformer to reduce the computational complexity. It should be noted that U-Net is a classic convolutional neural network architecture, which is composed of an encoder (downsampling) and a decoder (upsampling). The feature maps of the encoder and the decoder are connected through a skip connection, so as to capture the global structure while preserving the local details. The U-shaped structure combines the encoder-decoder structure of U-Net with the powerful modeling capability of Transformer. At a lower level, convolutional layers or Transformer blocks are used for feature extraction; at a higher level, global feature modeling is performed through Transformer blocks.

[0064] It should be noted that Mamba is a new type of neural network architecture, which belongs to the category of state space model (SSM). Mamba aims to overcome the limitations of traditional Transformer architecture in processing long-term dependencies and complex sequence data, especially in generation tasks, natural language understanding, time series modeling, etc.

[0065] The diffusion generation module can be a core image generation unit, which adopts a U-Net structure to nest a multi-step diffusion denoising process. In each stage, condition embedding from the unified motion structure embedding module (UMSE) and the context path module (GCM / PTCM) is introduced to guide the diffusion network to output video frames with clear structure and natural motion. The model supports residual connection and multi-scale fusion to ensure efficient propagation of feature information in the context. The diffusion generation module can be a condition-driven network, which relies on the guidance features (first condition guidance vector and second condition guidance vector) provided by the previous modules. In the inference process, the guidance features are concatenated in the middle and later stages of the entire system to complete the synthesis of high-quality frames.

[0066] In some implementations, in an environment with limited computing resources, an optimized version of Mamba (Selective SSM, S6) can be used to reduce the number of parameters and improve inference efficiency.

[0067] In some implementations, in a computing resource-limited environment, a depthwise separable convolution (Depthwise Separable Convolution) can be combined to reduce the amount of calculation, making it suitable for mobile VR devices.

[0068] The embodiment of the application uses a linear time complexity Mamba block to replace the traditional Transformer structure, greatly reduces the calculation cost, and increases the inference speed by 30%-50%. The consumption of computing resources is effectively reduced. The U-shaped hierarchical network (U-Net variant) is used in combination with the skip connection (Skip Connection) to enhance the feature transmission capability, so that the model can not only retain detailed information, but also maintain the integrity and consistency of the content when performing noise reduction and super-resolution processing, thereby improving the video repair quality.

[0069] In some embodiments, the context path module includes a global context modeling module and a local temporal consistency modeling module, the spatial layout information of the first video frame and the short-time dynamic information of the subsequent video frames of the first video frame are obtained, including:

[0070] The spatial distribution features of the first video frame are extracted by the global context modeling module using a down-sampling residual network to obtain the spatial layout information of the first video frame.

[0071] The short-time dynamic information of the subsequent video frames of the first video frame is obtained by processing the subsequent video frames of the first video frame using a time convolution or a sliding window attention mechanism by the local temporal consistency modeling module.

[0072] In some implementations, the context path module includes a global context modeling module and a local temporal consistency modeling module, the global context modeling module (Global Context Modeling Module, GCM) uses a down-sampling residual network to extract the spatial distribution features of the entire image, and models the background and object structure under the panoramic view. The local temporal consistency modeling module (Post-Target Consistency Modeling Module, PTCM) uses a time convolution or a sliding window attention mechanism to process a number of frames after the target frame to learn the local motion trend and change pattern.

[0073] The global context modeling module and the local temporal consistency modeling module are respectively responsible for extracting the spatial layout information of the complete picture and the short-time dynamic information of the subsequent frames of the target frame, and generating the first conditional guide vector.

[0074] In some implementations, the global context modeling module uses a feature pyramid network (FPN) to extract high-level semantic information and low-level detail information of the image, to obtain the spatial distribution features of the first video frame; and the local temporal consistency modeling module uses a bidirectional long short-term memory network (Bi-LSTM) to capture the forward and reverse temporal dependencies of the video frame sequence, to obtain the short-time dynamic information of the subsequent video frames of the first video frame.

[0075] In some other implementations, the global context modeling module uses a multi-scale convolutional network (MS-CNN) to capture local details and global structure information in the image, to extract the spatial distribution features of the first video frame. The local temporal consistency modeling module uses a temporal attention module, and the temporal attention mechanism can dynamically allocate attention weights to different video frames, to better capture the short-time dynamic information between the video frames.

[0076] It should be noted that the down-sampling residual network is a neural network structure combining down-sampling operation and residual learning. The down-sampling operation is used to reduce the spatial resolution of the feature map, so as to extract higher-level semantic features; and the residual learning solves the gradient vanishing problem in the training of the deep network by introducing residual connection.

[0077] The spatial distribution features can be the position, shape, size, and scene layout information of the objects in the video frame. These features reflect the static structure of the video frame. The spatial layout information can be a further abstraction of the spatial distribution features, focusing on describing the relative positions and overall structure of the elements in the video frame.

[0078] The temporal convolution can be a convolution operation for processing time series data. The temporal convolution extracts local features in the time series by sliding the convolution kernel in the time dimension. The sliding window attention mechanism can be a method combining sliding window and attention mechanism. The sliding window attention mechanism weights the frames in the window by sliding the window in the time dimension, to extract important time features.

[0079] The short-time dynamic information can be the motion change information in the local time range of the video frame sequence. The short-time dynamic information reflects the position change and motion trend of the objects in a short time.

[0080] In the embodiments of the present application, the global context modeling module extracts the spatial layout information of the video frame through the down-sampling residual network, to help the model understand the static background of the video frame. The local temporal consistency modeling module extracts the short-time dynamic information of the video frame sequence through the temporal convolution or the sliding window attention mechanism, to help the model capture the motion change in the video. Both modules inject the condition vector into each stage of the diffusion generation module, to improve the spatio-temporal consistency in the generation process, and lay a foundation for subsequent video rectification.

[0081] In some embodiments, the inter-frame smoothing processing on the first video frame to obtain the second conditional steering vector of the first video frame comprises:

[0082] obtaining optical flow information of the first video frame and edge features of the first video frame;

[0083] fusing and encoding the optical flow information and the edge features to obtain the second conditional steering vector of the first video frame.

[0084] By integrating a pre-trained network such as a RAFT optical flow estimation model in the unified motion structure embedding module, optical flow features (optical flow information) are obtained; by using a Sobel or Canny operator to extract image contours, a structural edge map (edge features) is obtained. By encoding the optical flow information and the edge features into low-dimensional vectors respectively, a unified conditional embedding representation (second conditional steering vector) is obtained after channel splicing and linear mapping. The unified conditional embedding representation has spatiotemporal perception ability and can dynamically adjust the generated path during the diffusion process.

[0085] For optical flow calculation: in some implementations, if the optical flow calculation cost is too high, a neural radiation field (NeRF) based motion estimation method can be used to improve temporal consistency using 3D view synthesis technology.

[0086] In other implementations, the attention mechanism based on Transformer is used to model the inter-frame relationship in an autoregressive manner, but this method has a large computational overhead and is suitable for offline VR video repair.

[0087] For fusion encoding: in some implementations, the optical flow information and the edge features are directly spliced together to form a high-dimensional feature vector.

[0088] In other implementations, the optical flow information and the edge features are weighted and summed, and the weights can be dynamically adjusted according to the importance of the features.

[0089] In other implementations, a neural network (such as a convolutional neural network) is used to extract and fuse the optical flow information and the edge features to generate a higher-level feature representation.

[0090] It should be noted that the optical flow information can be a two-dimensional vector field of pixel motion between consecutive frames in an image, and the optical flow information describes the motion direction and speed of pixels in the time dimension.

[0091] The edge features can be edge information of object contours, shapes and textures in an image, and edges are usually boundaries formed by significant changes in pixel intensity (such as from light to dark or from dark to light).

[0092] The fusion encoding can be a process of fusing multiple features (such as optical flow information and edge features) and converting them into a unified feature representation through an encoder.

[0093] In the embodiments of the present application, the motion trend of the object is understood through the optical flow information, the contour and shape of the object are retained through the edge features, the optical flow information and the edge features are fused together to generate a second condition guiding vector, which is used to guide the subsequent video frame generation or correction, so that the video is smoother and the structure is clearer. Through this inter-frame smoothing processing method, the shaking and blur in the video can be effectively reduced, and the overall quality of the video can be improved.

[0094] In some embodiments, before the inputting the video to be corrected into the diffusion generation model and performing a correction processing operation on the video to be corrected through the diffusion generation model to obtain a corrected video of the video to be corrected, the method further comprises:

[0095] obtaining an initial training sample set;

[0096] performing preprocessing and data enhancement on the initial training sample set to obtain a target training sample set; wherein the preprocessing comprises image decoding, size standardization, inter-frame registration, optical flow estimation and structural edge extraction; and the data enhancement comprises motion blur kernel, view rotation transformation and edge occlusion simulation;

[0097] training a preset diffusion generation model using the target training sample set until a training stop condition is met, to obtain a trained diffusion generation model.

[0098] In the embodiments, a data set (initial training sample set) for training the diffusion generation model is collected, and a video to be corrected and a corresponding corrected video thereof are obtained from an existing VR video data set. The video to be corrected and the corresponding corrected video thereof can come from a public video data set, a user-uploaded video or be generated in an artificial labeling manner. For each video to be corrected, a corrected video is provided. A confidence label is generated for each corrected video through artificial labeling or automatic evaluation (such as structural similarity index SSIM, peak signal-to-noise ratio PSNR, etc.). The confidence label is used to represent the similarity or quality of the corrected video to the real corrected video. The initial training sample set is processed (the preprocessing includes image decoding, size standardization, inter-frame registration, optical flow estimation and structural edge extraction; the data enhancement includes random superposition of motion blur kernel, view rotation transformation and edge occlusion simulation, and other disturbances; in addition, a multi-frame sampling strategy can be introduced to ensure that the model can learn the potential consistency rules between time-continuous frames.). The diffusion generation model is trained using the target training sample set after preprocessing and data enhancement until a training stop condition is met. Through the training, the diffusion generation model learns how to convert the video to be corrected into a high-quality corrected video.

[0099] It should be noted that the training stop condition can be a condition for judging whether the model has been trained, such as loss value convergence or reaching a preset training round.

[0100] Exemplarily, the following forms can be used for dataset selection, experimental environment setting, data enhancement and preprocessing.

[0101] I. Dataset selection: 1. Padmanaban 3D VR video dataset: contains 19 VR 3D videos, resolution 1920x720 to 2880x1080, used for VR video repair training. 2. SEPE8K8K video dataset: contains 8K video sequences, used to test the performance of the model in high-resolution super-resolution tasks. 3. VR YouTube sample video: select 120Hz high frame rate VR video, verify the real-time performance of U-MAD in high dynamic VR content.

[0102] II. Experimental environment setting: 1. Training device: 6xNVIDIA RTX 4090 GPU, 200 rounds of training, optimizer AdamW (momentum parameter 0.9, 0.999), weight decay 0.02. 2. Learning rate strategy: learning rate warm-up, gradually increase learning rate in the first 3% rounds, improve stability.

[0103] III. Data enhancement: 1. Use Gaussian noise, motion blur and random frame loss for data enhancement to improve the generalization ability of the model. 2. Add multi-scale super-resolution strategy in the training process to improve the adaptability of the model to different resolution VR videos.

[0104] IV. Preprocessing operation is performed on the VR video data, including image decoding, size standardization, frame registration, optical flow estimation and structure edge extraction. The optical flow feature is generated by integrating a pre-trained network such as RAFT to obtain a bidirectional flow field, and the structure edge is extracted by Sobel or Canny operator to obtain the image contour.

[0105] Through the above steps, the embodiments of the present application can effectively train a diffusion generation model to convert the video to be corrected into a high-quality corrected video. The preprocessing steps (image decoding, size standardization, frame registration, optical flow estimation and structure edge extraction) can help improve data quality, extract useful information or make data more suitable for subsequent processing. Data enhancement methods (motion blur kernel, view rotation transformation and edge occlusion simulation, etc.) increase the diversity and complexity of training data by simulating complex scenarios in the real world, helping the model better adapt to various situations and improve generalization ability. In addition, each step provides necessary support for model training to ensure that the model can learn high-quality correction effects.

[0106] In some embodiments, the preset diffusion generative model is trained using the target training sample set until a training stop condition is met, to obtain a trained diffusion generative model, specifically comprising:

[0107] For each training sample, the following steps are performed respectively:

[0108] The to-be-corrected video is input into the preset diffusion generative model to obtain a prediction confidence that the corrected video is the real corrected video of the to-be-corrected video;

[0109] According to the prediction confidence and the confidence label, a loss value of a loss function of the diffusion generative model is determined; wherein the loss function comprises a Charbonnier loss function and an optical flow loss function;

[0110] In the case that the loss value does not meet the training stop condition, the model parameters of the diffusion generative model are adjusted, and the diffusion generative model with the adjusted parameters is trained using the target training sample set until the training stop condition is met, to obtain a trained diffusion generative model.

[0111] In this embodiment, the to-be-corrected video is input into the diffusion generative model as input data. The diffusion generative model processes the input to-be-corrected video through its internal neural network structure (such as an encoder-decoder architecture) to generate a corrected video. An evaluation module (such as a pre-trained classifier or regressor) is used to evaluate the quality of the corrected video generated by the model. The evaluation module calculates the similarity or quality score between the generated corrected video and the real corrected video to obtain a prediction confidence. A loss function is used to compare the difference between the prediction confidence and the confidence label. The output of the loss function is the loss value, which reflects the gap between the corrected video generated by the model and the real corrected video. If the loss value does not meet the preset training stop condition (such as the loss value no longer significantly decreases or reaches the preset number of training rounds), the parameters of the model need to be adjusted. An optimization algorithm (such as gradient descent) is used to update the parameters of the model according to the loss value to reduce the loss value. The target training sample set is used to continue training the adjusted model, and the above steps are repeated until the training stop condition is met.

[0112] It should be noted that the prediction confidence can be a numerical value output by the evaluation module, indicating the similarity or quality of the generated corrected video to the real corrected video.

[0113] The loss function can be a function for measuring the difference between the model output and the target. Specifically, it includes pixel-level reconstruction loss, structure preservation loss (such as edge map L1 / L2 error), temporal consistency loss (adjacent frame difference), Charbonnier loss function, and optical flow loss function. The loss function is trained in cooperation with the diffusion generation module, and all loss function feedbacks act on the output results of the diffusion generation module to form a back propagation path and optimize the overall performance of the model.

[0114] In the embodiments of the present application, the input of the diffusion generation model is the video to be corrected, and the output is the corrected video. The quality score (prediction confidence) of the generated corrected video is calculated by the evaluation module, and the prediction confidence is compared with the confidence label using the loss function, so that the model can learn how to generate higher quality corrected videos. Secondly, the model training combines pixel-level reconstruction loss (to ensure that the model output is as close as possible to the target at the pixel level), structure preservation loss (to ensure that the model output is consistent with the target in structure), temporal consistency loss (to ensure that the changes between adjacent frames in a video sequence are smooth), Charbonnier loss (for denoising and super-resolution), and optical flow loss (for temporal consistency optimization), which helps to balance the recovery quality and improve the computational efficiency.

[0115] Reference Figure 2 , Figure 2 The overall implementation architecture of the diffusion generation model (U-MAD) is shown in FIG. 1. Figure 2 As shown in the figure, it specifically includes a Mamba block, a first conditional guidance vector generation part, and the internal implementation of the U-MAD.

[0116] The Mamba block is a basic operation unit introduced in the diffusion generation model of the present application, mainly used for modeling the interaction between temporal and spatial features. The shape of the input feature is (B, C, H, W, D), which respectively represents batch size (Batch), channel number (Channel), height (Height), width (Width), and depth / frame sequence length (Depth). The Mamba block is composed of the following 4 parts:

[0117] 1. Convolutional residual unit (left yellow and gray area): the input is first extracted by two three-dimensional convolutional layers (Conv) in series, and then connected with the input residual (Summarization⊕) after normalization operation (IN, i.e. Instance Normalization) and activation function (Leaky ReLU) to enhance the non-linear expression ability of the network and prevent gradient disappearance.

[0118] 2. Flatten+Layer Norm: After the initial feature extraction, the five-dimensional tensor is compressed and flattened into a two-dimensional representation for subsequent linear operations, while using Layer Norm to maintain numerical stability.

[0119] 3. Linear transformation and Mamba module (middle blue and purple area): The left blue module is two parallel linear mappings (Linear), which are used to generate the initial vectors of the gating path and the trunk path, respectively. In the upper path, the results of linear transformation are locally modeled by point-by-point convolution (1D Conv) and enhanced by the SiLU activation function (Sigmoid-weighted Linear Unit) to enhance the non-linear expression. Then, it is input into the structural state space module (SSM, State Space Model) to simulate long-distance dependencies in long sequences. The lower path directly participates in the gating mechanism through the SiLU activation and element-wise multiplication with the output of the upper path to achieve selective information retention.

[0120] 4. Output integration and shape recovery (right blue and yellow modules): The SSM output is linearly transformed and fused with the original branch features, and then reshaped to restore the original spatial structure (B, C, H, W, D) to continue transmission to the subsequent network.

[0121] wherein the first conditional guidance vector is generated by the output of the global context modeling module and the local temporal consistency modeling module. Specifically:

[0122] The original degraded frame (a video frame formed by the video to be corrected) and the degraded frame T time steps later are sampled by the down-sampler to obtain the first group and the second group: the first group: the current degraded frame, the degraded frame one time step ago, and the degraded frame two time steps ago, and the second group: the current degraded frame and the degraded frame T time steps later.

[0123] The first group is input into the global context modeling module, encoded by the encoder, processed by the Mamba block, and mapped by the mapping head to obtain the global feature map.

[0124] The first group and the degraded frame T time steps later are connected by the residual connection and input into the local temporal consistency modeling module, encoded by the encoder, and processed by the Mamba block to obtain the temporal feature map.

[0125] The global feature map and the temporal feature map are encoded into the first conditional guidance vector.

[0126] wherein the internal implementation of the U-MAD includes:

[0127] The to-be-corrected VR video is input through 3D mapping, and after being processed through downsampling, upsampling, conditional guide vector embedding and five Mamba blocks, the corrected video is obtained through 3D mapping output. Specifically, the to-be-corrected VR video is input through 3D mapping, and after being extracted by a preliminary convolutional encoder, the original feature map containing basic spatial texture and edge information is obtained. The original feature map is input into the first Mamba block, and the first Mamba block outputs the first feature map. After the first feature map is downsampled, it is input into the second Mamba block to further enhance the correlation in the time dimension and introduce the second conditional guide vector of the guide vector and motion structure embedding module (unified motion structure embedding module) formed by the global feature map. The second Mamba block outputs the second feature map. The second feature map, the guide vector formed by the global feature map and the second conditional guide vector are connected and merged, and input into the third Mamba block to strengthen the long-distance time relationship and global scene consistency. The third Mamba block outputs the third feature map. The third feature map, the guide vector formed by the time sequence feature map and the second conditional guide vector are connected and merged, and input into the fourth Mamba block to further refine the spatial structure and suppress motion artifacts and image blur areas. The fourth Mamba block outputs the fourth feature map. The fourth feature map, the guide vector formed by the time sequence feature map and the second conditional guide vector are connected and merged, and after being sampled by the upsampler, they are input into the fifth Mamba block. The fifth Mamba block generates the final feature map, which is input into the decoder of the diffusion network, and the corrected VR video is output through 3D mapping.

[0128] It should be noted that the five Mamba blocks are not simply repeated, but constitute a hierarchical feature enhancement network structure, each layer of which undertakes a specific semantic coding task. At the same time, the conditional guide vectors provided by the context path module and the unified motion structure embedding module are injected in the form of conditions in these Mamba blocks, guiding the feature update and information selection at each stage, so as to realize the generation of spatio-temporally consistent images.

[0129] The embodiment of the application can also be summarized in the following form: the real-time VR correction system based on diffusion generation model adopts an end-to-end modular structure design as a whole, mainly including an input processing module, a context path module, a unified motion structure embedding module, a diffusion generation module, a loss and optimization module, and an output interface module. First, after the original VR video frame is input into the system, image decoding, size standardization and frame registration are performed by the input processing module, and the inter-frame optical flow information and the structural edge map are extracted, providing dynamic and spatial priori for subsequent modeling. The context path module includes GCM and PTCM, which are respectively responsible for extracting the spatial layout information of the complete picture and the short-time dynamic information of the target frame subsequent frame, and generating the context feature representation, i.e., the first conditional guide vector. The unified motion structure embedding module (UMSE) fuses and encodes the optical flow and edge features to form a guide embedding vector with spatio-temporal perception ability, i.e., the second conditional guide vector. The above features are jointly input as conditions to guide the diffusion generation module, and the diffusion generation module performs iterative denoising and reconstruction of the image based on the multi-stage U-Net diffusion network. In each stage, the conditional embedding is introduced to improve the generation quality. In the training stage, the system introduces multiple loss functions, including reconstruction loss, structure consistency loss and temporal smoothing loss, to strengthen the performance of the output frame in terms of spatial structure and temporal coherence. Finally, the reconstruction result is packaged and sequenced by the output interface module, and can be directly used for real-time playback or post-processing on the VR rendering platform. The above modules work cooperatively through unified training and inference processes, realizing an efficient VR image restoration system with clear structure, explicit interface, good scalability and deployment capability.

[0130] In addition, for the method of the embodiment of the application, optimization schemes suitable for specific application scenarios can be: for VR live streaming: using progressive super-resolution (Progressive Super-Resolution) to gradually improve the picture quality during transmission and improve the bandwidth utilization. For VR games: combined with reinforcement learning (Reinforcement Learning), optimize motion prediction to make the repair result more consistent with the physical rules. For medical VR applications: increase specific optimization based on medical image data, such as improving the clarity of low-contrast areas to ensure the accuracy of medical analysis.

[0131] By providing multiple alternative schemes, the method of the embodiment of the application can flexibly adapt to different computing environments and application requirements, making the VR video correction technology more adaptable and scalable.

[0132] Reference Figure 3 , Figure 3The video correction method provided in the embodiments of the present application is compared with a repaired image driven by optical flow (related to optical flow information) and a repaired image driven by other manners (MotionCtrl driving, CameraCtrl driving) on the basis of an original image. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) reflect the quality of video repair. PSNR focuses on measuring the noise level of an image, while SSIM pays more attention to the structural information and visual perception quality of an image. Through PSNR and SSIM, the effect of video repair can be objectively evaluated, and it is ensured that the generated video has high visual quality and temporal consistency. The PSNR and SSIM of the repaired image obtained by the method provided in the embodiments of the present application are the best.

[0133] With reference to Figure 4 An embodiment of the present application provides a video correction device, the device comprising:

[0134] The correction module 401 is configured to input a to-be-corrected video into the diffusion generation model, perform a correction processing operation on the to-be-corrected video through the diffusion generation model, and obtain a corrected video of the to-be-corrected video.

[0135] The correction processing operation comprises:

[0136] For a first video frame of the to-be-corrected video, the context path module is configured to obtain spatial layout information of the first video frame and short-time dynamic information of a subsequent video frame of the first video frame, and generate a first conditional guidance vector of the first video frame based on the spatial layout information and the short-time dynamic information; the first conditional guidance vector is configured to represent context features of the first video frame.

[0137] The unified motion structure embedding module is configured to perform inter-frame smoothing processing on the first video frame, and obtain a second conditional guidance vector of the first video frame; the second conditional guidance vector is configured to represent a spatio-temporal perception capability of the first video frame.

[0138] The diffusion generation module is configured to generate a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector.

[0139] It should be noted that the device embodiment and the method embodiment described above are based on the same inventive concept, and therefore the content of the method embodiment described above is also applicable to the device embodiment, which will not be described here again.

[0140] Optionally, as Figure 5As shown, the embodiments of the present application also provide an electronic device 500, comprising a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions executable on the processor 501, and the programs or instructions are executed by the processor 501 to implement the steps of the above-mentioned video correction method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0141] It should be noted that the electronic device in the embodiments of the present application includes the above-mentioned mobile electronic device and non-mobile electronic device.

[0142] Figure 6 To realize the hardware structure of the electronic device in the embodiments of the present application.

[0143] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610, etc.

[0144] Those skilled in the art can understand that the electronic device 600 can also include a power supply (such as a battery) for powering various components, and the power supply can be logically connected to the processor 610 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0145] The processor 610 is configured to: input a to-be-corrected video into the diffusion generation model, perform a correction processing operation on the to-be-corrected video through the diffusion generation model, and obtain a corrected video of the to-be-corrected video.

[0146] The correction processing operation includes:

[0147] For a first video frame of the to-be-corrected video, the spatial layout information of the first video frame and the short-time dynamic information of a subsequent video frame of the first video frame are obtained through the context path module, and a first conditional guidance vector of the first video frame is generated based on the spatial layout information and the short-time dynamic information; the first conditional vector is used to represent the context features of the first video frame;

[0148] The first video frame is subjected to inter-frame smoothing processing through the unified motion structure embedding module to obtain a second conditional guidance vector of the first video frame; the second conditional guidance vector is used to represent the spatio-temporal perception ability of the first video frame;

[0149] generate a corrected video frame of the first video frame based on the first conditional steering vector and the second conditional steering vector by the diffusion generation module.

[0150] In some embodiments, the processor 610 is further configured to embed the first conditional steering vector and the second conditional steering vector into each stage of the diffusion generation module to obtain the corrected video frame of the first video frame.

[0151] In some embodiments, the processor 610 is further configured to extract spatial distribution features of the first video frame by using a down-sampling residual network to obtain spatial layout information of the first video frame by the global context modeling module.

[0152] process subsequent video frames of the first video frame by using a time convolution or a sliding window attention mechanism to obtain short-time dynamic information of the subsequent video frames of the first video frame by the local temporal consistency modeling module.

[0153] In some embodiments, the processor 610 is further configured to obtain optical flow information of the first video frame and edge features of the first video frame.

[0154] fuse encode the optical flow information and the edge features to obtain a second conditional steering vector of the first video frame.

[0155] In some embodiments, the processor 610 is further configured to obtain an initial training sample set.

[0156] preprocess and data augment the initial training sample set to obtain a target training sample set; wherein the preprocessing includes image decoding, size standardization, inter-frame registration, optical flow estimation, and structural edge extraction; and the data augmentation includes motion blur kernel, view rotation transformation, and edge occlusion simulation.

[0157] train a preset diffusion generation model using the target training sample set until a training stop condition is met to obtain a trained diffusion generation model.

[0158] In some embodiments, the processor 610 is further configured to input the video to be corrected into a preset diffusion generation model to obtain a prediction confidence of the corrected video being a real corrected video of the video to be corrected.

[0159] determine a loss value of a loss function of the diffusion generation model according to the prediction confidence and the confidence label; wherein the loss function includes a Charbonnier loss function and an optical flow loss function.

[0160] In a case where the loss value does not satisfy the training stop condition, a model parameter of the diffusion generation model is adjusted, and the diffusion generation model after the parameter adjustment is trained by using the target training sample set until the training stop condition is satisfied, to obtain a trained diffusion generation model.

[0161] It should be understood that in the embodiments of the present application, the input unit 604 can include a graphics processor (GPU) 6041 and a microphone 6042. The graphics processor 6041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 can include a display panel 6061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 can include two parts of a touch detection device and a touch controller. The other input devices 6072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, an operation lever, and the like, which will not be described here.

[0162] The memory 609 can be used to store software programs and various data. The memory 609 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 609 can include a volatile memory or a non-volatile memory, or the memory 609 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 609 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0163] The processor 610 can include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 610.

[0164] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize the processes of the above-mentioned video correction method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0165] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0166] In addition, the embodiments of the present application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the processes of the above video correction method embodiments, and achieve the same technical effects. To avoid repetition, details are not described here.

[0167] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles, or systems that include a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles, or systems. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article, or system including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of performing functions as shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from the described order, and various steps can also be added, omitted, or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0168] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0169] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by software and necessary general hardware platform, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a plurality of instructions to make a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the methods described in various embodiments of the present application.

[0170] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A video correction method, characterized in that, Applied to electronic devices, the electronic devices include a diffusion generation model, the diffusion generation model includes a context path module, a unified motion structure embedding module, and a diffusion generation module; The method includes: The video to be corrected is input into the diffusion generation model, and the diffusion generation model performs a correction process on the video to be corrected to obtain the corrected video. The corrective processing operation includes: For the first video frame of the video to be corrected, the spatial layout information of the first video frame and the short-time dynamic information of the subsequent video frames of the first video frame are obtained through the context path module, and a first conditional guidance vector of the first video frame is generated based on the spatial layout information and the short-time dynamic information; the first conditional vector is used to characterize the contextual features of the first video frame. The first video frame is smoothed by the unified motion structure embedding module to obtain a second conditional guidance vector for the first video frame; the second conditional guidance vector is used to characterize the spatiotemporal awareness capability of the first video frame. The diffusion generation module generates a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector.

2. The method as described in claim 1, characterized in that, The diffusion generation module adopts a multi-stage U-shaped structure and performs nested diffusion denoising based on Mamba blocks. The step of generating a corrected video frame of the first video frame based on the first conditional guiding vector and the second conditional guiding vector by the diffusion generation module includes: The first conditional guidance vector and the second conditional guidance vector are embedded into each stage of the diffusion generation module to obtain the corrected video frame of the first video frame.

3. The method as described in claim 1, characterized in that, The context path module includes a global context modeling module and a local temporal consistency modeling module. The step of obtaining the spatial layout information of the first video frame and the short-term dynamic information of subsequent video frames includes: The spatial distribution features of the first video frame are extracted by the global context modeling module using a downsampling residual network to obtain the spatial layout information of the first video frame. The local temporal consistency modeling module uses temporal convolution or sliding window attention mechanisms to process subsequent video frames of the first video frame, thereby obtaining short-term dynamic information of the subsequent video frames of the first video frame.

4. The method as described in claim 1, characterized in that, The step of performing inter-frame smoothing on the first video frame to obtain the second conditional guidance vector of the first video frame includes: Obtain the optical flow information of the first video frame and the edge features of the first video frame; The optical flow information and the edge features are fused and encoded to obtain the second conditional guidance vector of the first video frame.

5. The method as described in claim 1, characterized in that, Before inputting the video to be corrected into the diffusion generation model, and performing a correction process on the video to be corrected through the diffusion generation model to obtain the corrected video, the process includes: Obtain the initial training sample set; The initial training sample set is preprocessed and data augmented to obtain the target training sample set; wherein, the preprocessing includes image decoding, size normalization, inter-frame registration, optical flow estimation, and structural edge extraction; the data augmentation includes motion blur kernel, viewpoint rotation transformation, and edge occlusion simulation; The preset diffusion generation model is trained using the target training sample set until the training stopping condition is met, resulting in the trained diffusion generation model.

6. The method as described in claim 5, characterized in that, The step of training a preset diffusion generation model using the target training sample set until the training stopping condition is met, to obtain the trained diffusion generation model, specifically includes: For each training sample, perform the following steps: The video to be corrected is input into a preset diffusion generation model to obtain the prediction confidence that the corrected video is the true corrected video of the video to be corrected. Based on the prediction confidence and the confidence label, the loss value of the loss function of the diffusion generation model is determined; wherein, the loss function includes the Charbonnier loss function and the optical flow loss function; If the loss value does not meet the training stopping condition, the model parameters of the diffusion generation model are adjusted, and the adjusted diffusion generation model is trained using the target training sample set until the training stopping condition is met, thus obtaining the trained diffusion generation model.

7. A video correction device, characterized in that, The device includes: The correction module is used to input the video to be corrected into the diffusion generation model, and perform correction processing operations on the video to be corrected through the diffusion generation model to obtain the corrected video. The corrective processing operation includes: For the first video frame of the video to be corrected, the spatial layout information of the first video frame and the short-time dynamic information of the subsequent video frames of the first video frame are obtained through the context path module, and a first conditional guidance vector of the first video frame is generated based on the spatial layout information and the short-time dynamic information; the first conditional vector is used to characterize the contextual features of the first video frame. The first video frame is smoothed by inter-frame processing using a unified motion structure embedding module to obtain a second conditional guidance vector for the first video frame; the second conditional guidance vector is used to characterize the spatiotemporal awareness capability of the first video frame. The diffusion generation module generates a corrected video frame of the first video frame based on the first conditional guidance vector and the second conditional guidance vector.

8. A video correction device, characterized in that, The device includes: a memory, a processor, and a computer processing program stored in the memory and executable on the processor, the computer processing program being configured to implement the video correction method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer processing program, which, when executed by a processor, implements the video correction method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the video correction method as described in any one of claims 1 to 6.

Images