Panoramic video frame insertion method based on potential diffusion model

Abstract

The invention discloses a panoramic video frame insertion method based on a potential diffusion model. The method comprises the following steps: compressing an input image to a potential space by using a panoramic perception vector quantization variational auto-encoder to obtain potential features; constructing initial noise, and fusing the motion features extracted by the panoramic optical flow adapter to obtain condition information; performing iterative denoising operation on the potential features of the intermediate frame to obtain final potential features of the intermediate frame; and restoring the final potential features of the intermediate frame into a frame insertion image of a pixel space through a condition decoder. According to the panoramic video frame interpolation method provided by the invention, the potential diffusion model and the panoramic characteristic enhancement technology are combined, so that the perception quality of a frame interpolation result can be remarkably improved, details and complex textures of a pole region can be better reserved, and a high-quality time interpolation solution is provided for immersive panoramic video application.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, specifically the field of video frame interpolation technology, and relates to a panoramic video frame interpolation method based on a latent diffusion model. Background Art

[0002] Video frame interpolation technology is an important video processing method, aiming to generate additional intermediate frames based on two or more known frames to improve the smoothness and frame rate of videos, and is widely used in fields such as video coding and decoding, slow motion generation, and restoring lost frames in video streams. As a special form of video, panoramic video provides an all-round perspective, enabling viewers to immerse in a 360-degree visual experience. However, due to the special nature of panoramic video, traditional video frame interpolation technology faces unprecedented challenges.

[0003] Panoramic videos are usually stored and processed in the equirectangular projection (ERP) format. This projection method maps spherical information onto a plane, resulting in severe geometric distortions in the top and bottom regions of the image. These distortions make it difficult for traditional frame interpolation algorithms based on deterministic optical flow mapping to accurately estimate the motion in the pole regions, often leading to severe blurring and artifacts in the frame interpolation results in these regions. At the same time, traditional methods also often perform poorly when dealing with complex dynamic textures, producing over-smoothed results and affecting the visual perception quality. In addition, the motion characteristics in panoramic videos are also significantly different from those in planar videos. There is periodic continuity in the horizontal direction and a close relationship with the latitude position in the vertical direction, which further increases the difficulty of panoramic video frame interpolation.

[0004] With the development of deep learning technology, generation methods based on diffusion models have shown great potential in the field of image and video processing. Diffusion models can effectively handle complex probability distributions and uncertainties, so they are particularly suitable for dealing with complex geometric deformations and texture preservation problems in panoramic video frame interpolation. However, how to combine diffusion models with the geometric characteristics of panoramic videos to achieve high-quality panoramic frame interpolation remains a technical challenge to be solved urgently. Summary of the Invention

[0005] The purpose of the present invention is to propose a panoramic video frame interpolation method based on a latent diffusion model, which is specially designed to solve the problems encountered in the process of panoramic video frame interpolation, such as severe distortions in the pole regions caused by equirectangular projection, insufficient preservation of complex dynamic textures, and poor visual perception quality.

[0006] The present invention regards the panoramic video frame interpolation problem as a conditional generation task, and realizes high-quality intermediate frame generation through diffusion modeling in the latent space. The frame interpolation network consists of three core components: a panoramic perception vector quantization variational autoencoder, a panoramic optical flow adapter, and a noise prediction module with a latitude perception mechanism.

[0007] First, the panoramic perception vector quantization variational autoencoder compresses the high-dimensional ERP format panoramic image into a low-dimensional structured latent space and can reconstruct high-quality images from the latent representation. This encoder introduces a pole perception attention mechanism that dynamically adjusts the attention weights according to the latitude position, enabling the pole region to obtain a wider attention field of view and effectively retaining the key geometric information under spherical projection.

[0008] Second, the panoramic optical flow adapter module receives the optical flow field generated by the panoramic optical flow estimation network, extracts the corresponding motion features, and fuses them with the features of the front and back frames extracted by the encoder as the conditional information for the diffusion process. This module is designed to specifically capture the longitude periodicity and latitude dependence in the ERP format and provide a spherical motion prior for the network.

[0009] Finally, the noise prediction module performs an iterative denoising process in the latent space, gradually transforming the random noise into the latent representation of the intermediate frame. This module introduces a latitude-aware loss function, considers the importance differences of different latitude regions in spherical projection, and ensures that the generated content conforms to the spherical geometric characteristics by evaluating the generation quality in the spherical coordinate system.

[0010] The specific steps of the method of the present invention are as follows: Step (1): Use the panoramic perception vector quantization variational autoencoder to compress the input image into the latent space to obtain latent features; the panoramic perception vector quantization variational autoencoder consists of a panoramic perception encoder, a vector quantization layer, and a conditional decoder; the panoramic perception encoder compresses the input image into low-dimensional latent features, the vector quantization layer discretizes the continuous latent features into vector representations in a predefined codebook, and the conditional decoder reconstructs the quantized latent representation into an image in the pixel space; Step (2): Construct initial noise and fuse the motion features extracted by the panoramic optical flow adapter to obtain conditional information; Step (3): The latent features of the intermediate frame are obtained through iterative denoising operations to obtain the final latent features of the intermediate frame ; Step (4): Restore the final latent features of the intermediate frame to an interpolated image in the pixel space through the conditional decoder.

[0011] The present invention adopts the generation framework of the diffusion model to process the panoramic video frame interpolation task, effectively solving the limitations of traditional deterministic methods in dealing with highly distorted and complex dynamic textures in the pole region. By operating in the latent space, this method not only reduces the computational complexity but also retains important structured information. In addition, the integrated panoramic optical flow adapter introduces the spherical motion prior into the diffusion process, making the generated intermediate frames more coherent and natural in terms of motion performance. The latitude-aware loss function and the pole-aware attention mechanism are specifically optimized for the geometric characteristics of panoramic videos, ensuring high-quality generation results even in the pole region.

[0012] By combining the latent diffusion model and panoramic feature enhancement techniques, the proposed panoramic video frame interpolation method of the present invention can not only significantly improve the perceptual quality of the frame interpolation results but also better retain the details and complex textures in the pole region, providing a high-quality temporal interpolation solution for immersive panoramic video applications. Brief Description of the Drawings

[0013] Figure 1 It is a flowchart of the panoramic video frame interpolation method based on the latent diffusion model of the present invention. Detailed Embodiments

[0014] A panoramic video frame interpolation method based on the latent diffusion model generates high-quality intermediate frames through the collaborative work of a panoramic perception vector quantization variational autoencoder, a panoramic optical flow adapter, and a denoising U-Net network. This method compresses the image into a low-dimensional latent space for processing, which not only reduces the computational complexity but also effectively retains the structured information; the fused optical flow features provide accurate motion priors, while the latitude-aware mechanism and the pole region enhancement strategy are specifically optimized for the geometric characteristics of panoramic videos. The overall design is flexible and efficient, especially suitable for dealing with complex geometric deformations and dynamic textures in panoramic videos, and can significantly enhance the user's immersive experience.

[0015] The panoramic optical flow adapter receives the optical flow field generated by the panoramic optical flow estimation network, extracts the corresponding motion features, and fuses them with the features of the front and rear frames extracted by the encoder as the conditional information for the diffusion process. This module is designed to capture the longitude periodicity and latitude dependence in the ERP format, providing the spherical motion prior for the network, so that the generated intermediate frames are more coherent and natural in terms of motion performance.

[0016] The denoising U-Net module performs an iterative denoising process in the latent space, gradually transforming random noise into the latent representation of the intermediate frame. The module adopts an encoder-decoder architecture, combined with temporal and conditional embedding, and effectively captures the global and local features of the image through multi-scale processing. During the training process, a latitude-aware loss function is introduced to consider the importance differences of different latitude regions in the spherical projection, and the generation quality is evaluated in the spherical coordinate system to ensure that the generated content conforms to the spherical geometric characteristics.

[0017] The specific steps of this method are as follows: Step (1) uses a panoramic-aware vector quantization variational autoencoder to compress the input image into a latent space and obtain latent features. The panoramic-aware vector quantization variational autoencoder consists of a panoramic-aware encoder, a vector quantization layer, and a conditional decoder. The panoramic-aware encoder compresses the input image into low-dimensional latent features, the vector quantization layer discretizes the continuous latent features into vector representations in a predefined codebook, and the conditional decoder reconstructs the quantized latent representation into a pixel space image. The panoramic-aware encoder introduces a pole-aware attention module that dynamically adjusts the attention weight according to the latitude position, so that the pole area can obtain a wider attention field of view and effectively retain the key geometric information under the spherical projection. The details are as follows: (1-1) The panoramic perception encoder extracts potential features from the input image: three consecutive frames of images , and Input into the panoramic perception encoder to obtain the corresponding latent features , and .

[0018] The encoder adopts a hierarchical structure of multi-level downsampling, with the spatial resolution decreasing step by step and the number of channels increasing step by step. The feature size of the final output is 1 / 100 of the original image. ,in is the downsampling factor, in this embodiment .

[0019] First, the original video frame is passed through a The initial convolution layer with a stride of 1 increases the number of channels from 3 to 128, and the spatial dimension remains unchanged; then it passes through five levels of downsampling modules, and the scale of the features of each downsampling module is reduced to half of the previous one, and the number of channels is twice the original. The downsampling module consists of a residual block and a pole-aware attention module; the output of the lowest resolution feature obtains global context information through the intermediate block, which contains two residual blocks and an attention module; finally, through normalization, nonlinear activation layers and convolution layers, the potential features of the corresponding frame are output.

[0020] For each residual block, the input features are first normalized and then non-linearly activated to introduce non-linear transformation. After that, the features pass through two convolutional kernels with a kernel size of and a stride of 1, while keeping the spatial dimension unchanged. At the same time, the original input features are transformed through the convolutional skip connection. Finally, the feature map on the main path is added to the feature on the skip connection to obtain the output of the residual block.

[0021] (1-2) The pole-aware attention module of the downsampling module processes the geometric distortion of the pole region to obtain the final attention .

[0022] First, calculate the latitude-based weight mask of the image , represents the real number field, is the height of the image, is the width of the image; calculate the weight of each pixel: , is the coordinate of the pixel, represents the vertical position coordinate; Truncate the weight mask to obtain the pole mask matrix , and use the pole mask matrix to adjust the key value in the attention calculation process: , is the original key value, represents element-wise multiplication; The final attention , the superscript represents transpose, is the number of channels, is the query value, represents the value vector.

[0023] (1-3) The vector quantization layer discretizes the latent features corresponding to the intermediate frames and calculates the Euclidean distance between the latent features output by the panoramic perception encoder and the entries in the predefined codebook . represents the number of entries in the predefined codebook, and the latent feature and the th entry The Euclidean distance between , represents the number of channels of the latent feature, represents the entry in the th channel of the feature, Find the codebook entry with the smallest Euclidean distance to replace the original input feature as the quantized latent feature .

[0024] (1 - 4) The conditional decoder reconstructs the quantized latent feature into a pixel - space image. The conditional decoder adopts a multi - scale upsampling structure and gradually restores the spatial resolution of the feature map through three - layer upsampling operations. To enhance temporal consistency, the conditional decoder introduces a cross - frame attention mechanism to fuse the features of the previous and subsequent frames with the features of the middle frame.

[0025] The conditional decoder includes an initial convolutional layer, an intermediate processing module, multi - scale upsampling, and a feature fusion module. The quantized latent feature first passes through an initial convolutional layer with a stride of 1, keeping the spatial resolution unchanged and only changing the channel dimension. Then it is input into the intermediate processing module, which consists of two residual modules and an intermediate attention module. The residual block contains two convolutional layers with a convolutional kernel size, normalization operation, and rectified linear unit. In the multi - scale upsampling and feature fusion module, the spatial resolution of the latent feature is gradually restored through three - layer upsampling operations: the latent feature first extracts feature representations through two residual blocks, then obtains global spatial dependencies through a self - attention mechanism, and then fuses with the latent feature and extracted by the panoramic perception encoder; finally, the height and width of the latent feature are doubled through a bilinear interpolation upsampling operation while keeping the number of channels unchanged, and then the upsampled latent feature is processed by a convolutional kernel with a kernel size of and a stride of 1 to make the feature smoother. After three - layer upsampling operations, the final feature is obtained, and a pixel - space image is synthesized through two offset prediction heads and a mask prediction head.

[0026] Step (2) constructs initial noise, fuses the motion features extracted by the panoramic optical flow adapter to obtain conditional information. Specifically as follows: (2 - 1) Estimate the panoramic optical flow from the input frame, and use a pre - trained panoramic optical flow estimation network to calculate the bidirectional optical flow fields from the middle frame to the previous frame and from the middle frame to the subsequent frame and .

[0027] (2 - 2) Extract motion features through the panoramic optical flow adapter. The panoramic optical flow adapter contains five - layer feature extraction modules. After each layer of downsampling, the size of the feature map is halved, and the number of channels increases according to The multiplication factor increases. The bidirectional optical flow and are merged into a 4-channel input through channel splicing, and features matching the latent space dimension are output after processing.

[0028] (2-3) Construct the initial noise: Sample random noise from the standard normal distribution and add it to the latent features of the intermediate frame to construct the initial state with noise: , where is the intermediate state, is the cumulative noise attenuation factor, and it is calculated according to the predefined noise schedule : , represents the total number of time steps, and the time step .

[0029] Fuse the extracted motion features with the latent features of the previous and next frames , and the initial state with noise to construct the conditional information.

[0030] In step (3), the latent features of the intermediate frame are obtained through iterative denoising operations, and the final latent features of the intermediate frame are obtained. Specifically as follows: (3-1) Use the noise prediction module U-Net to process the latent features of the intermediate frame with noise. The noise prediction module U-Net contains three levels of downsampling and three levels of upsampling paths.

[0031] (3-2) The downsampling path extracts multi-level features. Through multiple residual blocks and convolutional layers with a stride of 2, the spatial resolution is gradually reduced and hierarchical features are extracted. The number of channels is sequentially expanded to 1 times, 2 times, and 4 times the basic channel; the middle block of the noise prediction module U-Net adopts a ResBlock-Attention-ResBlock structure to capture global dependencies at the lowest resolution.

[0032] (3-3) The upsampling path fuses multi-scale features, gradually restores the spatial size through transposed convolution, and at the same time fuses the skip connection features from the downsampling path; the cross-attention module of the upsampling path is deployed at a specific resolution level, enabling the feature map to selectively extract guidance signals from the conditional information of the previous and next frames.

[0033] (3-4) The time step is processed by the MLP network of the noise prediction module U-Net and injected into the residual block in the upsampling path to output the predicted noise .

[0034] (3-5) Execute the reverse diffusion process using the DDIM sampler of the noise prediction module U-Net to update the latent features and obtain the denoised latent features. .

[0035] (3-6) Repeat (3-2) to (3-5) until denoising for all time steps is completed to obtain the final intermediate frame latent features. .

[0036] Step (4) Restores the final intermediate frame latent features to an interpolated image in the pixel space through the conditional decoder. Specifically as follows: (4-1) The final intermediate frame latent features are restored to an interpolated image in the pixel space through the conditional decoder of the panoramic perception variational autoencoder. .

[0037] (4-2) The generated interpolated image is calculated with the real intermediate image through the loss function to obtain the interpolation error, and backpropagation is performed to optimize the network parameters. The loss function , is the mathematical expectation, is the real noise.

[0038] Figure 1 is the flowchart of the panoramic video interpolation method based on the latent diffusion model, showing the complete processing process from the input front and back frames to the generation of the interpolated image, including four steps: panoramic perception vector quantization variational autoencoder, panoramic optical flow adapter and noise construction, iterative denoising process, and decoding and post-processing.

[0039] The present invention combines the latent diffusion model with the characteristics of panoramic videos and proposes a novel panoramic video interpolation method. Compared with traditional deterministic methods, the present invention shows obvious advantages in the pole region and complex dynamic texture scenes, and can generate intermediate frames with higher visual quality, stronger temporal consistency and better compliance with spherical geometric characteristics. In particular, the key components designed for the geometric characteristics of panoramic videos, such as the pole-aware attention module, panoramic optical flow adapter and latitude-aware loss function, enable this method to effectively handle the special challenges in the panoramic video interpolation task and provide a high-quality temporal interpolation solution for immersive panoramic video applications.

Claims

1. A panoramic video frame insertion method based on a potential diffusion model, characterized by: Step (1) uses a panoramic-aware vector quantization variational autoencoder to compress the input image into a latent space to obtain latent features; the panoramic-aware vector quantization variational autoencoder is composed of a panoramic-aware encoder, a vector quantization layer and a conditional decoder; the panoramic-aware encoder compresses the input image into low-dimensional latent features, the vector quantization layer discretizes the continuous latent features into vector representations in a predefined codebook, and the conditional decoder reconstructs the quantized latent representation into a pixel space image; Step (2) constructing initial noise and fusing motion features extracted by the panoramic optical flow adapter to obtain conditional information; Step (3) The latent features of the intermediate frame are subjected to an iterative denoising operation to obtain the final latent features of the intermediate frame. ; Step (4) converts the final intermediate frame potential features The interpolated image is restored to pixel space through the conditional decoder.

2. The panoramic video frame insertion method based on the potential diffusion model according to claim 1, characterized in that: Step (1) is as follows: (1-1) The panoramic perception encoder extracts potential features from the input image: three consecutive frames of images , and Input into the panoramic perception encoder to obtain the corresponding latent features , and ; The encoder adopts a hierarchical structure of multi-level downsampling, with the spatial resolution decreasing step by step and the number of channels increasing step by step. The feature size of the final output is ,in is the downsampling factor; (1-2) The pole-aware attention module of the downsampling module processes the geometric distortion of the pole area and obtains the final attention ; First, calculate the latitude-based weight mask of the image , represents the field of real numbers, The height of the image, is the width of the image; Calculate the weight of each pixel: , For coordinates The weight of the pixel, Indicates the vertical position coordinate; Mask the weights Truncate to get the pole mask matrix , the pole mask matrix Key values ​​used to adjust the attention calculation process: , is the original key value, represents point-wise multiplication; Final Attention , superscript represents transpose, is the number of channels, is the query value, represents a value vector; (1-3) The vector quantization layer converts the potential features corresponding to the intermediate frames into Discretization, computing the latent features of the panoramic perception encoder output With predefined codebook The Euclidean distance between the entries in , represents the number of predefined codebook entries, potential features With Items The Euclidean distance between , represents the number of channels of latent features, Indicates the entry In the The characteristics of the channels, ; Find the codebook entry with the smallest Euclidean distance to replace the original input feature as the quantized potential feature ; (1-4) The conditional decoder converts the quantized latent features Reconstructed into pixel space image; the conditional decoder adopts a multi-scale upsampling structure and gradually restores the spatial resolution of the feature map through three layers of upsampling operations.

3. The panoramic video frame insertion method based on the potential diffusion model according to claim 2, characterized in that: Step (1-1) is as follows: first, the original video frame is passed through a The initial convolution layer with a stride of 1 increases the number of channels from 3 to 128, and the spatial dimension remains unchanged; then through five levels of downsampling modules, the scale of each downsampling module is reduced to half of the previous one, and the number of channels is doubled. The downsampling module consists of a residual block and a pole-aware attention module; the lowest resolution feature is output to obtain global context information through the intermediate block, which contains two residual blocks and an attention module; finally, through normalization, nonlinear activation layers and convolution layers, the potential features of the corresponding frame are output; For each residual block, the input features are first normalized and then nonlinearly activated to introduce nonlinear transformations; Afterwards, the features are passed through two kernels with sizes The convolution kernel has a step size of 1 and keeps the spatial dimension unchanged; at the same time, the original input features are The convolutional skip connection performs feature transformation; finally, the feature map on the main path is added to the features on the skip connection to obtain the output of the residual block.

4. The panoramic video frame insertion method based on the potential diffusion model according to claim 3, characterized in that: The conditional decoder in step (1-4) includes an initial convolution layer, an intermediate processing module, a multi-scale upsampling and feature fusion module; the quantized potential features First, through a The initial convolutional layer with a step size of 1 keeps the spatial resolution unchanged and only changes the channel dimension; then it is input to the intermediate processing module, which consists of two residual modules and an intermediate attention module; the residual block contains two The convolutional layers, normalization operations, and linear rectification functions of the convolution kernel size are used; in the multi-scale upsampling and feature fusion module, the spatial resolution of the latent features is gradually restored through three layers of upsampling operations: the latent features are first extracted through two residual blocks, and then the global spatial dependencies are obtained through the self-attention mechanism, and then combined with the latent features extracted by the panoramic perception encoder. and to fuse; finally, the height and width of the potential features are doubled by upsampling with bilinear interpolation, while keeping the number of channels unchanged, and then a kernel size of , the convolution kernel with a step size of 1 processes the potential features after upsampling to make the features smoother; after three layers of upsampling operations, the final features are obtained, and the pixel space image is synthesized through two offset prediction heads and one mask prediction head.

5. The panoramic video frame insertion method based on the potential diffusion model according to claim 4, characterized in that: Step (2) is as follows: (2-1) Estimate panoramic optical flow from the input frame and calculate the intermediate frame using the pre-trained panoramic optical flow estimation network To previous frame and intermediate frames To next frame Bidirectional optical flow field and ; (2-2) Extracting motion features through panoramic optical flow adapter ; The panoramic optical flow adapter contains five layers of feature extraction modules. After each layer of downsampling, the feature map size is halved and the number of channels is calculated according to The multiplication factor of the bidirectional optical flow is increased; and The input is merged into 4 channels through channel stitching, and after processing, the output features match the dimensions of the latent space; ( 2-3) Construct initial noise: Sample random noise from standard normal distribution Add to the potential features of the intermediate frame to construct the initial state with noise: ,in is the intermediate state, is the cumulative noise attenuation factor, according to the predefined noise schedule calculate: , represents the total time step, time step ; The extracted motion features Latent features of previous and next frames , and the noisy initial state Perform fusion and construct conditional information.

6. The panoramic video frame insertion method based on the potential diffusion model according to claim 5, characterized in that: Step (3) is as follows: (3-1) Using the noise prediction module U-Net to process the potential features of the noisy intermediate frame, the noise prediction module U-Net includes three-level downsampling and three-level upsampling paths; (3-2) Downsampling path extracts multi-level features. The spatial resolution is gradually reduced and hierarchical features are extracted through multiple residual blocks and convolutional layers with a stride of 2. The number of channels is expanded to 1, 2, and 4 times that of the basic channel respectively. The middle block of the noise prediction module U-Net adopts the ResBlock-Attention-ResBlock structure to capture global dependencies at the lowest resolution. (3-3) The upsampling path fuses multi-scale features, gradually restores the spatial size through transposed convolution, and fuses the skip connection features from the downsampling path; The criss-cross attention modules of the upsampling path are deployed at specific resolution levels, enabling the feature maps to selectively extract guidance signals from the previous and next frame conditional information; (3-4) Time step After being processed by the MLP network of the noise prediction module U-Net, the residual block in the upsampling path is injected to output the predicted noise ; (3-5) Use the DDIM sampler of the noise prediction module U-Net to perform the reverse diffusion process, update the latent features, and obtain the denoised latent features ; (3-6) Repeat (3-2) to (3-5) until all time steps are denoised and the final intermediate frame potential features are obtained. .

7. The panoramic video frame insertion method based on the potential diffusion model according to claim 6, characterized in that: Step (4) is as follows: (4-1) Final intermediate frame potential features The interpolated image is restored to pixel space through the conditional decoder of the panorama-aware variational autoencoder ; (4-2) Generated interpolated image The loss function is used to calculate the interpolation error with the real intermediate image, and then back propagation is performed to optimize the network parameters. , is the mathematical expectation, is the real noise.

Images