Real-time intermediate stream estimation algorithm integrated with self-attention mechanism
Through the real-time intermediate flow estimation algorithm enhanced by the self-attention mechanism, the accuracy and robustness of optical flow estimation in complex scenarios is solved, and efficient video frame rate improvement is achieved.
Patent Information
- Application Number
- CN202510609376.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-13
- Publication Date
- 2025-08-08
AI Technical Summary
The traditional video frame interpolation method has limited performance when dealing with problems such as large motion, occlusion and non-rigid deformation in complex scenarios, and traditional convolutional neural networks are difficult to accurately capture global information.
The real-time intermediate flow estimation algorithm using the self-attention mechanism is used to build an image pyramid and introduce IFNet and FusionNet. The self-attention mechanism is used to enhance the model's ability to capture global information, and combine the "coarse to thin" optical flow estimation strategy to generate high-quality intermediate frames.
Improves the accuracy and robustness of optical flow estimation, especially in handling large motions and complex scenarios, and maintains real-time processing capabilities.
Smart Images
Figure CN120451875A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time intermediate flow estimation algorithm incorporating a self-attention mechanism. Background Art
[0002] Video frame interpolation technology aims to generate intermediate frames from existing frames, thereby increasing the video frame rate and enhancing video smoothness. Traditional frame interpolation methods are mainly based on optical flow estimation. However, the performance of traditional methods is often limited when dealing with problems such as large motion, occlusion, and non-rigid deformation in complex scenes.
[0003] In recent years, deep learning-based optical flow estimation algorithms have made significant progress. Among them, the "coarse-to-fine" optical flow estimation strategy has been widely used. This strategy first calculates coarse optical flow at a low resolution and then gradually refines it to obtain high-resolution optical flow. However, traditional convolutional neural networks have limitations in capturing global information and struggle to accurately process motion relationships in complex scenes. Summary of the Invention
[0004] The purpose of the present invention is to provide a real-time intermediate flow estimation algorithm that incorporates a self-attention mechanism to solve the problems raised in the above background technology.
[0005] To achieve the above objectives, the present invention provides the following technical solutions: a real-time intermediate flow estimation algorithm incorporating a self-attention mechanism, comprising the following steps: obtaining two adjacent frames of video images and ; Build an image pyramid and downsample the input frame; Use the improved IFNet to estimate the intermediate flow and IFNet adopts a "coarse-to-fine" optical flow estimation strategy and introduces a self-attention mechanism in the IFBlock module; the estimated intermediate flow is used to perform reverse optical flow distortion on the input frame to generate a preliminary intermediate frame; FusionNet is used to fuse multi-source information to generate the final intermediate frame, in which FusionNet introduces a self-attention mechanism in the encoder.
[0006] Furthermore, each IFBlock module includes:
[0007] Convolutional layer, used to extract local information of input features;
[0008] Activation function layer, used to introduce nonlinearity;
[0009] Self-attention module, used to calculate the global dependencies between different locations in the feature map;
[0010] Feature fusion layer, which is used to add the output of the self-attention module to the original features.
[0011] Furthermore, the calculation steps of the self-attention module include:
[0012] The input feature map is passed through three convolutional layers to generate the query matrix Q, key matrix K and value matrix V respectively;
[0013] Calculate the matrix multiplication of the transpose of Q and K and scale it by dividing by √d to get the attention score matrix S;
[0014] Use the softmax function to normalize the attention score matrix S to obtain the attention weight matrix A;
[0015] Multiply the attention weight matrix A with the value matrix V to get the weighted sum result;
[0016] The weighted sum result is mapped back to the original feature dimension through a convolutional layer and added to the input feature.
[0017] Furthermore, FusionNet includes an encoder and a decoder, where the encoder concatenates the input frame, intermediate stream and preliminary intermediate frame, and extracts features through convolutional layers and self-attention modules, and the decoder generates the final intermediate frame through deconvolution layers and upsampling operations.
[0018] Furthermore, the encoder introduces self-attention modules in multiple layers to enhance the model's ability to capture global information.
[0019] Furthermore, the reverse optical flow warping step includes:
[0020] Using an intermediate stream Input frame Twist to get ;
[0021] Using an intermediate stream Input frame Twist to get ;
[0022] Perform weighted fusion on the distorted frames according to the time ratio t to obtain the preliminary intermediate frame .
[0023] Further, the "coarse to fine" optical flow estimation strategy includes:
[0024] Start processing from the lowest resolution image and estimate the rough optical flow;
[0025] Upsample the low-resolution optical flow to a higher resolution and use it as the initial value, and combine it with higher-resolution features to further refine the optical flow estimation;
[0026] Repeat the above steps until the highest resolution is reached and the final optical flow estimation result is obtained.
[0027] Furthermore, the algorithm also includes model training steps, including:
[0028] Prepare a video dataset containing adjacent frame pairs and annotate the middle frame;
[0029] Design a combined loss function that includes optical flow consistency loss, pixel reconstruction loss, and perceptual loss;
[0030] Use an optimization algorithm to train the model and minimize the loss function.
[0031] This paper provides a real-time intermediate flow estimation algorithm incorporating a self-attention mechanism, which offers the following benefits: By incorporating the self-attention mechanism, the model can capture global motion information in images, improving the accuracy of optical flow estimation, particularly when dealing with large motions and complex scenes. The self-attention module also better handles occlusions and non-rigid deformations, enhancing the robustness of the algorithm. While maintaining algorithmic performance, the algorithm maintains real-time processing capabilities by optimizing the network structure and computational process, making it suitable for practical applications. DETAILED DESCRIPTION
[0032] The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0033] A real-time intermediate flow estimation algorithm incorporating a self-attention mechanism includes the following steps: Obtain two adjacent video frames and ; Build an image pyramid and downsample the input frame; Use the improved IFNet to estimate the intermediate flow and IFNet adopts a "coarse-to-fine" optical flow estimation strategy and introduces a self-attention mechanism in the IFBlock module; the estimated intermediate flow is used to perform reverse optical flow distortion on the input frame to generate a preliminary intermediate frame; FusionNet is used to fuse multi-source information to generate the final intermediate frame, in which FusionNet introduces a self-attention mechanism in the encoder.
[0034] Each IFBlock module includes:
[0035] Convolutional layer, used to extract local information of input features;
[0036] Activation function layer, used to introduce nonlinearity;
[0037] Self-attention module, used to calculate the global dependencies between different locations in the feature map;
[0038] Feature fusion layer, which is used to add the output of the self-attention module to the original features.
[0039] The calculation steps of the self-attention module include:
[0040] The input feature map is passed through three convolutional layers to generate the query matrix Q, key matrix K and value matrix V respectively;
[0041] Calculate the matrix multiplication of the transpose of Q and K and scale it by dividing by √d to get the attention score matrix S;
[0042] Use the softmax function to normalize the attention score matrix S to obtain the attention weight matrix A;
[0043] Multiply the attention weight matrix A with the value matrix V to get the weighted sum result;
[0044] The weighted sum result is mapped back to the original feature dimension through a convolutional layer and added to the input feature.
[0045] FusionNet consists of an encoder and a decoder, where the encoder concatenates the input frame, intermediate stream, and preliminary intermediate frame, and extracts features through convolutional layers and self-attention modules, and the decoder generates the final intermediate frame through deconvolution layers and upsampling operations.
[0046] The encoder introduces self-attention modules in multiple layers to enhance the model's ability to capture global information.
[0047] The reverse optical flow warping steps include:
[0048] Using an intermediate stream Input frame Twist to get ;
[0049] Using an intermediate stream Input frame Twist to get ;
[0050] Perform weighted fusion on the distorted frames according to the time ratio t to obtain the preliminary intermediate frame .
[0051] The "coarse to fine" optical flow estimation strategy includes:
[0052] Start processing from the lowest resolution image and estimate the rough optical flow;
[0053] Upsample the low-resolution optical flow to a higher resolution and use it as the initial value, and combine it with higher-resolution features to further refine the optical flow estimation;
[0054] Repeat the above steps until the highest resolution is reached and the final optical flow estimation result is obtained.
[0055] The algorithm also includes model training steps, including:
[0056] Prepare a video dataset containing adjacent frame pairs and annotate the middle frame;
[0057] Design a combined loss function that includes optical flow consistency loss, pixel reconstruction loss, and perceptual loss;
[0058] Use an optimization algorithm to train the model and minimize the loss function.
[0059] The core strategy of IFNet is a "coarse-to-fine" optical flow estimation approach. This strategy first reduces the resolution of the input frame to quickly calculate coarse optical flow information on the low-resolution image. Low-resolution images require less computation and can quickly determine the general motion trend. Subsequently, using the low-resolution optical flow results as the initial value, fine-tuning is performed on the high-resolution image, gradually refining the optical flow estimate to achieve more accurate results. This process is accomplished collaboratively by IFBlock, the downsampling module, and the upsampling module. The downsampling module reduces computational complexity by reducing the image size, while the upsampling module restores the image resolution. Combined with IFBlock's feature processing, this achieves a coarse-to-fine optical flow estimation.
[0060] The IFBlock module is a key component for processing features in IFNet. Inside the module, the input features are first processed by the convolution layer. The convolution layer extracts various local features in the image, such as edges and textures, through different convolution kernels. The convolved features are activated by the Leaky ReLU function to introduce nonlinearity, which enhances the expressiveness of the model and enables the model to learn more complex feature relationships. After integrating the self-attention mechanism, the self-attention module is added after the convolution and activation operations. The self-attention module breaks the limitation of traditional convolution operations that only focus on local areas. It can analyze the relationship between different positions in the feature map from a global perspective, helping the model capture the global information of the image at different resolutions, thereby more accurately extracting inter-frame motion features and optimizing the accuracy of optical flow estimation.
[0061] The core of the self-attention module is to calculate the attention relationship between different positions in the input feature map. , through three convolutional layers to generate query, key, and value. The query is used to find information related to itself, the key is used to represent the feature information of each position, and the value is the actual feature content of the corresponding position. After the generated Q, K, and V are transformed in dimension, the inner product of the transposed query and key is calculated and divided by The weighted sum of the attention values is weighted by the attention weights to obtain a new representation of each position based on global information. Finally, the weighted sum is transformed back to the original dimension and added to the original input feature map to obtain the output of the self-attention module, thereby enhancing the global information of the input features.
[0062] Linear motion assumption: In order to obtain the intermediate flow Then, the linear motion assumption is used to approximate the The linear motion hypothesis is based on the principle that the motion of an object is approximately linear in a short period of time. , according to the known The optical flow in the other direction is inferred based on the time ratio t of the intermediate frames. This assumption simplifies the calculation to a certain extent and can effectively estimate a reasonable optical flow in most common scenarios, providing a basis for subsequent frame distortion and fusion.
[0063] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.
Claims
1. A real-time intermediate flow estimation algorithm incorporating a self-attention mechanism, characterized by: The following steps are involved: Get two adjacent frames of video images and ; Build an image pyramid and downsample the input frame; Use the improved IFNet to estimate the intermediate flow and IFNet adopts a "coarse-to-fine" optical flow estimation strategy and introduces a self-attention mechanism in the IFBlock module; the estimated intermediate flow is used to perform reverse optical flow distortion on the input frame to generate a preliminary intermediate frame; FusionNet is used to fuse multi-source information to generate the final intermediate frame, in which FusionNet introduces a self-attention mechanism in the encoder.
2. A real-time intermediate flow estimation algorithm incorporating a self-attention mechanism according to claim 1, characterized in that: Each IFBlock module includes: Convolutional layer, used to extract local information of input features; Activation function layer, used to introduce nonlinearity; Self-attention module, used to calculate the global dependencies between different locations in the feature map; Feature fusion layer, which is used to add the output of the self-attention module to the original features.
3. The real-time intermediate flow estimation algorithm incorporating the self-attention mechanism according to claim 1 is characterized in that: The calculation steps of the self-attention module include: The input feature map is passed through three convolutional layers to generate the query matrix Q, key matrix K and value matrix V respectively; Calculate the matrix multiplication of the transpose of Q and K and scale it by dividing by √d to get the attention score matrix S; Use the softmax function to normalize the attention score matrix S to obtain the attention weight matrix A; Multiply the attention weight matrix A with the value matrix V to get the weighted sum result; The weighted sum result is mapped back to the original feature dimension through a convolutional layer and added to the input feature.
4. The real-time intermediate flow estimation algorithm incorporating the self-attention mechanism according to claim 1 is characterized in that: FusionNet consists of an encoder and a decoder, where the encoder concatenates the input frame, the intermediate stream, and the preliminary intermediate frame, and extracts features through convolutional layers and self-attention modules. The decoder generates the final intermediate frame through deconvolution layers and upsampling operations.
5. The real-time intermediate flow estimation algorithm incorporating the self-attention mechanism according to claim 1 is characterized in that: The encoder introduces self-attention modules in multiple layers to enhance the model's ability to capture global information.
6. The real-time intermediate flow estimation algorithm incorporating a self-attention mechanism according to claim 1, characterized in that: The reverse optical flow warping steps include: Using an intermediate stream The input frame I1 is distorted to obtain ; Using an intermediate stream The input frame I2 is distorted to obtain ; Perform weighted fusion on the distorted frames according to the time ratio t to obtain the preliminary intermediate frame .
7. The real-time intermediate flow estimation algorithm incorporating a self-attention mechanism according to claim 1, characterized in that: The "coarse to fine" optical flow estimation strategy includes: Start processing from the lowest resolution image and estimate the rough optical flow; Upsample the low-resolution optical flow to a higher resolution and use it as the initial value, and combine it with higher-resolution features to further refine the optical flow estimation; Repeat the above steps until the highest resolution is reached and the final optical flow estimation result is obtained.
8. The real-time intermediate flow estimation algorithm incorporating a self-attention mechanism according to claim 1, characterized in that: The algorithm also includes model training steps, including: Prepare a video dataset containing adjacent frame pairs and annotate the middle frame; Design a combined loss function that includes optical flow consistency loss, pixel reconstruction loss, and perceptual loss; Use an optimization algorithm to train the model and minimize the loss function.