A panoramic video frame interpolation method based on bidirectional separation correlation

Abstract

This invention discloses a panoramic video frame interpolation method based on bidirectional separation of correlation quantities. The method first extracts appearance features and correlation quantity features from the input image. Then, it uses the correlation quantity features to perform cost aggregation to obtain the initial optical flow. The initial optical flow is iteratively updated to obtain refined optical flow, final feature values, mask, and residual information. Finally, the information is fused to obtain the interpolated frame image. This invention employs a cost aggregation method to separate the generated correlation quantities, more effectively handling the special problems of motion anisotropy and uneven motion distribution in panoramic videos. It effectively solves the problems of artifacts and inaccurate estimation caused by differences in sampling rates of projections in different dimensions. This invention better adapts to motion query errors caused by differences in motion amplitude at different latitudes, thereby improving the accuracy of optical flow prediction and ultimately obtaining higher-quality intermediate frames.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, specifically the field of video frame interpolation technology, and relates to a panoramic video frame interpolation method that uses flexible querying of bidirectional separation correlation quantities. Background Technology

[0002] Video frame interpolation is an important video processing method that aims to generate additional intermediate frames based on two or more known frames to improve video smoothness and frame rate. It is widely used in video encoding and decoding, slow-motion generation, and recovering lost frames from video streams. Panoramic video, as a special video format, provides a full-range perspective, immersing viewers in a 360-degree visual experience. It typically involves multiple cameras simultaneously capturing scenes from different directions and then stitching them together to create a complete spherical view. However, due to the unique nature of panoramic video, traditional video frame interpolation techniques face unprecedented challenges. For example, after being projected into a rectangular image using equidistant cylindrical projection, panoramic video exhibits severe image distortion, especially at the top and bottom of the rectangular image. This distortion complicates traditional motion estimation based on planar images. Furthermore, the shooting equipment for panoramic video may be limited by technology, resulting in a lower frame rate and causing a jerky playback. This is particularly problematic in fast-moving scenes, not only degrading the user experience but also potentially causing discomfort such as motion sickness. Meanwhile, in virtual reality environments, users can freely change their viewpoint, meaning that panoramic videos must maintain consistent motion continuity across different perspectives. This places higher demands on video frame interpolation technology, requiring accurate motion estimation and generation of appropriate intermediate frames from various viewpoints. Due to image distortion, traditional feature detection and matching methods often assume uniform geometric properties, making them difficult to apply effectively to panoramic videos. Furthermore, motion in panoramic videos may not be simple translation or rotation, but rather complex three-dimensional spatial motion, further increasing the difficulty of implementing panoramic video frame interpolation technology. Summary of the Invention

[0003] The purpose of this invention is to propose a panoramic video frame interpolation method based on bidirectional separation correlation. This method is specifically designed to solve the problems encountered in the frame interpolation process of panoramic video, such as image distortion, motion anisotropy, and non-uniformity of motion distribution caused by equidistant cylindrical projection.

[0004] The frame interpolation network mainly consists of three parts: a dual-channel feature extraction subnetwork, an optical flow refinement subnetwork based on correlation quantity separation and flexible query strategies, and a frame synthesis module. These three parts work closely together to ensure the successful completion of the panoramic video frame interpolation task.

[0005] First, the dual-channel feature extraction subnetwork consists of a correlation feature encoder and an appearance feature encoder, which extract two types of features from the input image—correlation features and appearance features, respectively. The correlation features obtained by the correlation feature encoder are used in subsequent correlation generation and cost aggregation processes to further refine the prediction of the optical flow field and intermediate frames. The appearance feature encoder focuses on extracting the appearance features of the image, which are used for optical flow calculation and iterative generation of intermediate frames.

[0006] Secondly, optical flow refinement based on the separation correlation quantity and flexible query strategy utilizes the bidirectional separation correlation quantity after cost aggregation to guide the iterative process of optical flow and intermediate features. This sub-network provides the necessary data foundation for frame synthesis by generating information such as optical flow, intermediate values, residuals, and masks.

[0007] Finally, the frame synthesis module fuses the refined optical flow, intermediate value, residual, and mask to obtain the final intermediate frame.

[0008] The specific steps of the method of this invention are as follows:

[0009] Step (1) Extract the appearance features and related quantitative features of the input image;

[0010] Step (2) Use relevant quantity features to perform cost aggregation to obtain the initial optical flow;

[0011] Step (3) iteratively updates the initial optical flow to obtain the refined optical flow, final eigenvalues, mask, and residual;

[0012] Step (4) fuses the final feature value, refined optical flow, mask, and residual to obtain the interpolated image.

[0013] This invention employs a cost aggregation method to separate generated correlation quantities, more effectively addressing the unique challenges of motion anisotropy and uneven motion distribution in panoramic videos. The method integrates image matching costs in both the horizontal and vertical directions, effectively resolving artifacts and inaccurate estimations caused by differences in sampling rates across different projected dimensions. Furthermore, the flexible query strategy within the optical flow thinning network allows for the use of latitude-based offsets to obtain correlation quantity information when querying for corresponding correlation quantities. This better adapts to motion query errors caused by differences in motion amplitude across different latitudes, thereby improving the accuracy of optical flow prediction and ultimately obtaining higher-quality intermediate frames.

[0014] This invention proposes a panoramic video frame interpolation method that combines relevant quantity separation with flexible query strategies. This method not only significantly improves the frame interpolation effect but also enhances the user's immersive experience, making it particularly suitable for virtual reality and other application scenarios that require high-quality panoramic video. Detailed Implementation

[0015] A panoramic video frame interpolation method is proposed. This method utilizes a feature extraction network, a bidirectional optical flow iterative network based on correlation and flexible querying, and a frame synthesis network to work collaboratively, improving video smoothness while maintaining visual quality. The feature extraction network employs dual-channel feature extraction, utilizing parallel paths to capture different features of the image, providing rich information for subsequent processing. The optical flow iterative network uses a cost aggregation module to estimate object motion and enhances accuracy through optical flow refinement. The frame synthesis sub-network integrates feature and optical flow information to generate intermediate frames. Furthermore, an encoder-decoder architecture ensures effective extraction and reconstruction of image features. The overall design is flexible and efficient, adaptable to complex scene changes, and applicable to virtual reality, augmented reality, and video surveillance fields, significantly enhancing the user's immersive experience.

[0016] 1. The feature extraction network uses two dedicated channels to extract different types of features: the correlation encoder module focuses on extracting correlation information between input images, which is used to form the cost volume; the appearance encoder module is responsible for extracting appearance features such as color, texture and edges from the input image, which are used for the next step of optical flow generation and intermediate feature value generation.

[0017] 2. The optical flow iterative network comprises a correlation quantity generation and separation module and a cost aggregation module. The cost aggregation module estimates object motion and enhances accuracy through optical flow refinement. The optical flow iterative network receives information from the appearance feature encoder and decodes it to obtain intermediate features. Simultaneously, based on the correlation quantity features obtained from the correlation quantity feature encoder, it calculates the matching degree between two frames. First, it generates a bidirectional cost volume through element-wise multiplication. Then, the cost volume is aggregated in the horizontal and vertical directions by the cost aggregation module to reduce noise and enhance the response of reliable matching points. Finally, it iterative updates are performed, querying the cost volume using a query strategy and using the query results to optimize the optical flow and intermediate feature values. This process is repeated until the generated optical flow and intermediate feature values ​​reach the target accuracy of the frame interpolation task. The iterative refinement process helps improve the accuracy and robustness of optical flow estimation.

[0018] The correlation generation module generates a 4D tensor that records the matching degree between all pixels in two consecutive frames. Optical flow is used to query this tensor to obtain more accurate optical flow data to guide the generation of high-quality intermediate frames. Then, a refinement iterative network for optical flow and intermediate feature values ​​iteratively optimizes the optical flow field, gradually improving its accuracy to ensure that the generated intermediate frames are as close as possible to the real scene. The accuracy of optical flow estimation directly affects the quality of the final intermediate frames. Through the combined action of these two modules, the effective execution of the video frame interpolation task is ensured.

[0019] Workflow: First, information is received from the appearance feature encoder module and decoded to obtain intermediate features. Simultaneously, in the correlation path, the correlation features obtained from the correlation feature encoder are used to calculate the matching degree between two frames, specifically by generating bidirectional cost volumes through element-wise multiplication. These cost volumes are then further aggregated horizontally and vertically by the cost aggregation module to reduce noise and enhance the response of reliable matching points. Finally, through an iterative update method, the optical flow refinement module uses a flexible query strategy to query the cost volumes, continuously optimizing the optical flow and related intermediate feature values ​​with the query results. This process is repeated until the generated optical flow and intermediate feature values ​​reach the target accuracy of the frame interpolation task. The iterative refinement process helps improve the accuracy and robustness of optical flow estimation.

[0020] 3. The frame synthesis network fuses intermediate feature values, optical flow, mask, and residual information. To generate high-quality interpolated images, it fuses various information sources, including final feature values, optical flow, mask, and residuals. Intermediate feature values ​​capture the structural and texture details of the original frames; the optical flow provides the direction and velocity information of object motion between adjacent frames, which is crucial for predicting the motion of intermediate frames; the mask identifies areas requiring special attention, such as occluded parts or motion boundaries; and the residual represents the difference between the predicted image and the real image, used to further improve image quality. Through the effective integration of this information, the frame fusion network can generate interpolated images that are both accurate and rich in detail.

[0021] The specific steps of this method are as follows:

[0022] Step (1) Extract the appearance features and related quantitative features of the input image; specifically as follows:

[0023] (1-1) The appearance feature encoder module extracts appearance features from the input image, including color features, texture features, and edge features. The appearance feature encoder uses a pyramid-structured convolutional neural network (CNN) to extract appearance feature values ​​at four different scales. superscript Indicates the number of floors. 1,2,3,4, subscript 1 and 2 represent the previous frame, respectively. and the next frame Each layer's feature values ​​are used for iterative updates; as appearance feature values ​​pass through each layer of the convolutional neural network, the feature scale becomes the original scale. The number of channels is doubled. After the two consecutive input images are processed by the appearance feature encoder, each image obtains four appearance feature values ​​at different scales.

[0024] (1-2) The correlation encoder module extracts correlation features from the input image. The correlation encoder uses a pyramid-structured convolutional neural network (CNN). When two consecutive frames pass through the pyramid network, each frame outputs a correlation feature value. and Its tensor size is , and Indicates the height and width of the image. This indicates the layer number of the features, with subscripts 0 and 1 representing the input layers, respectively. and .

[0025] Step (2) uses relevant quantity features to perform cost aggregation to obtain the initial optical flow; specifically as follows:

[0026] (2-1) The correlation quantity is obtained by calculating the dot product of the characteristic values ​​of the two correlation quantities. , , The number of channels for the correlation quantity eigenvalues. The total number of channels for the relevant characteristic values. , and Coordinates representing the feature; It is a 4D tensor with coordinates as , This represents the real number field. The higher the value of the correlation quantity, the greater the correlation between the two corresponding coordinates.

[0027] (2-2) Separate the relevant quantities and calculate the maximum and average values ​​of the correlation matrix in the horizontal and vertical directions respectively:

[0028] The average value of the relevant quantity in the horizontal direction ;

[0029] The maximum value of the correlation quantity in the horizontal direction ;

[0030] The vertical average value of the relevant quantity ;

[0031] The maximum value of the correlation quantity in the vertical direction ; , .

[0032] The average and maximum values ​​of the obtained correlation quantities in the horizontal direction are upsampled and concatenated to obtain the horizontal tensor. The average and maximum values ​​of the obtained correlation quantities in the vertical direction are upsampled and concatenated to obtain the vertical tensor. .

[0033] (2-3) Tensor for the horizontal direction Cost aggregation is performed sequentially in four directions: top, bottom, left, and right, resulting in aggregated horizontal correlation values ​​in all four directions. Aggregation direction (up, down, left, right), where r represents the unit vector in the aggregation direction. Represents the horizontal distance between corresponding pixels in two consecutive frames, a variable. , This represents the summation operation. , , and These represent the four weights obtained from the input image through a convolutional neural network of an appearance feature encoder.

[0034] Tensor in the vertical direction Cost aggregation is performed sequentially in four directions: top, bottom, left, and right, resulting in the vertical correlation values ​​after aggregation in all four directions. ; It represents the vertical distance between corresponding pixels in two consecutive frames.

[0035] (2-4) Horizontal correlation quantity The initialized horizontal optical flow component is obtained through the codec network. Vertical correlation The initialized vertical optical flow component is obtained through the codec network. The codec network is a multi-layer convolutional neural network. It first performs convolutional operations on the input layer by layer, then performs deconvolutional operations layer by layer, and finally outputs the result.

[0036] The forward initial optical flow is obtained by splicing the optical flow components from the two directions. , cat means concatenation.

[0037] (2-5) Regarding the relevant quantities Transpose to obtain the relevant quantity after transposition Repeat steps (2-2) to (2-4) to obtain the reverse initialization optical flow. .

[0038] Step (3) iteratively updates the initial optical flow to obtain the refined optical flow, final eigenvalues, mask, and residual; specifically as follows:

[0039] (3-1) Initialize optical flow with forward direction and reverse initialization optical flow For relevant quantities as well as and The query strategy involves using the positional encoding of pixels in the panoramic image to obtain an offset for the query. Latitude-related positional information is input into the position encoder to obtain an offset associated with that positional information. This offset is then added to the original query radius to obtain the query result. Specifically:

[0040] First, based on the location information, at each pixel coordinate... Generate query box above Query range , where a and b are the offsets of the horizontal and vertical coordinates, respectively, and L is the side length of the query box.

[0041] Then, the forward initialization optical flow will be performed. and reverse initialization optical flow Input position encoder Generate a positive offset based on the position information p. and reverse offset : , The location information p is obtained by calculating the vertical distance between a pixel and the height midline of the panoramic image (i.e., the equator), and then normalizing that distance.

[0042] Query box Add the obtained positive offset Get the forward query position Query box Add the obtained reverse offset Get the reverse lookup location , and These are the x and y coordinates of the offset, respectively.

[0043] Using forward lookup location Query related volume positive query volume ; Use reverse lookup location Query the relevant quantities after transposition Get the reverse query volume .

[0044] (3-2) Forward query volume The input is fed into the update module U to obtain the forward optical flow increment. and positive intermediate eigenvalue increment ; Reverse query volume The input is fed into the update module U to obtain the reverse optical flow increment. and the inverse intermediate eigenvalue increment The update module U is a multi-layer convolutional neural network that performs convolutional operations on the input layer by layer and then outputs the result.

[0045] (3-3) Increment the positive intermediate eigenvalue and the inverse intermediate eigenvalue increment By concatenating the values, the intermediate feature value increments are obtained. Appearance feature values Increment of intermediate eigenvalues Adding them together yields the first-order intermediate eigenvalues. .

[0046] Forward optical flow increment With forward initialization optical flow Adding them together yields the first-order forward optical flow. Reverse optical flow increment With reverse initialization optical flow Adding them together yields the first-order reverse optical flow. .

[0047] (3-4) Using first-order forward optical flow and first-order reverse optical flow For relevant quantities as well as and Perform the query and repeat steps (3-1) to (3-3) to obtain the secondary intermediate feature value, the secondary forward optical flow, and the secondary reverse optical flow;

[0048] (3-5) Repeat step (3-4) to iterate until the set number of iterations is reached, and then stop iterating to obtain intermediate feature values, forward optical flow and reverse optical flow.

[0049] (3-6) Input the intermediate feature value, forward optical flow, and reverse optical flow into the decoder to obtain the final feature value. Forward refinement of optical flow and reverse refinement of optical flow Simultaneously, the mask Y and residual S are obtained. The decoder is a multi-layer convolutional neural network that performs convolutional operations on the input layer by layer and then outputs the result.

[0050] Step (4) fuses the final feature values, refined optical flow, mask, and residual to obtain the interpolated image; specifically as follows:

[0051] (4-1) Interpolated Image ; This indicates a backward warp operation, which uses optical flow to perform a spatial transformation on the image, mapping the pixels of the current frame to the corresponding positions at the time of the interpolation frame; This indicates pixel-by-pixel multiplication, and the resulting value is... arrive The mask Y and the back-distorted image are multiplied pixel by pixel; the difference term S is used to compensate for the errors in the interpolation process in the synthesized image.

[0052] (4-2) The interpolated image and the real intermediate image are calculated using a loss function to obtain the interpolation error. Backpropagation is then performed to optimize the parameters of each convolutional neural network and improve the accuracy of the interpolated image.

[0053] loss function ; I represents the actual value of the intermediate frame, It is a very small constant to ensure that the denominator is not zero during network backpropagation. ERP weight The design principle of this weighting function lies in the fact that in equidistant cylindrical projection, the top and bottom of the image are significantly distorted due to projection deformation. Therefore, pixels in these areas are less important for reconstructing or processing panoramic images. By assigning smaller weights to these areas, their impact on the overall image processing result can be reduced.

Claims

1. A panoramic video frame interpolation method based on bidirectional separation correlation, characterized in that: Step (1) Extract the appearance features and related quantitative features of the input image; Step (2) uses relevant quantity features to perform cost aggregation to obtain the initial optical flow; specifically as follows: (2-1) The correlation quantity is obtained by calculating the dot product of the characteristic values ​​of the two correlation quantities. , , and These are two related feature values ​​output by the pyramid network for each of the two consecutive frames. The number of channels for the correlation quantity eigenvalues. The total number of channels for the relevant characteristic values. and Coordinates representing the feature; , Represents the real number field. It is a 4D tensor with coordinates as ; (2-2) Separate the relevant quantities and calculate the maximum and average values ​​of the correlation matrix in the horizontal and vertical directions respectively: The average value of the relevant quantity in the horizontal direction ; The maximum value of the correlation quantity in the horizontal direction ; The vertical average value of the relevant quantity ; The maximum value of the correlation quantity in the vertical direction ; , ; The average and maximum values ​​of the obtained correlation quantities in the horizontal direction are upsampled and concatenated to obtain the horizontal tensor. The average and maximum values ​​of the obtained correlation quantities in the vertical direction are upsampled and concatenated to obtain the vertical tensor. ; (2-3) Tensor for the horizontal direction Cost aggregation is performed sequentially in four directions: top, bottom, left, and right, resulting in aggregated horizontal correlation values ​​in all four directions. Aggregation direction (up, down, left, right), where r represents the unit vector in the aggregation direction. Represents the horizontal distance between corresponding pixels in two consecutive frames, a variable. , This represents the summation operation. , , and These represent the four weights obtained from the input image through a convolutional neural network of an appearance feature encoder. Tensor in the vertical direction Cost aggregation is performed sequentially in four directions: top, bottom, left, and right, resulting in the vertical correlation values ​​after aggregation in all four directions. ; It represents the vertical distance between corresponding pixels in two consecutive frames; (2-4) Horizontal correlation quantity The initialized horizontal optical flow component is obtained through the codec network. Vertical correlation The initialized vertical optical flow component is obtained through the codec network. The codec network is a multi-layer convolutional neural network. It first performs convolutional operations on the input layer by layer, then performs deconvolutional operations layer by layer, and finally outputs the result. The forward initial optical flow is obtained by splicing the optical flow components from the two directions. cat indicates concatenation; (2-5) Regarding the relevant quantities Transpose to obtain the relevant quantity after transposition Repeat steps (2-2) to (2-4) to obtain the reverse initialization optical flow. ; Step (3) iteratively updates the initial optical flow to obtain the refined optical flow, final eigenvalues, mask, and residual; specifically as follows: (3-1) Initialize optical flow with forward direction and reverse initialization optical flow For relevant quantities as well as and Perform a query to obtain the query volume; details are as follows: First, based on the location information, at each pixel coordinate... Generate query box above Query range a and b are the offsets of the horizontal and vertical coordinates, respectively, and L is the side length of the query box; Then, the forward initialization optical flow will be performed. and reverse initialization optical flow Input position encoder Generate a positive offset based on the position information p. and reverse offset : , The location information p is obtained by calculating the vertical distance between a pixel and the height center line of the panoramic image, and then normalizing that distance. Query box Add the obtained positive offset Get the forward query position Query box Add the obtained reverse offset Get the reverse lookup location , and These are the x and y coordinates of the offset, respectively; Using forward lookup location Query related volume positive query volume ; Use reverse lookup location Query the relevant quantities after transposition Get the reverse query volume ; (3-2) Forward query volume The input is fed into the update module U to obtain the forward optical flow increment. and positive intermediate eigenvalue increment ; Reverse query volume The input is fed into the update module U to obtain the reverse optical flow increment. and the inverse intermediate eigenvalue increment The update module U is a multi-layer convolutional neural network that performs convolutional operations on the input layer by layer and then outputs the result. (3-3) Increment the positive intermediate eigenvalue and the inverse intermediate eigenvalue increment By concatenating the values, the intermediate feature value increments are obtained. Appearance feature values Increment of intermediate eigenvalues Adding them together yields the first-order intermediate eigenvalues. ; Forward optical flow increment With forward initialization optical flow Adding them together yields the first-order forward optical flow. Reverse optical flow increment With reverse initialization optical flow Adding them together yields the first-order reverse optical flow. ; (3-4) Using first-order forward optical flow and first-order reverse optical flow For relevant quantities as well as and Perform the query and repeat steps (3-1) to (3-3) to obtain the secondary intermediate feature value, the secondary forward optical flow, and the secondary reverse optical flow; (3-5) Repeat step (3-4) to iterate until the set number of iterations is reached, and stop iterating to obtain intermediate feature values, forward optical flow and reverse optical flow; (3-6) Input the intermediate feature value, forward optical flow, and reverse optical flow into the decoder to obtain the final feature value. Forward refinement of optical flow and reverse refinement of optical flow Simultaneously, the mask Y and the residual S are obtained; The decoder is a multi-layer convolutional neural network that performs convolutional operations on the input layer by layer and then outputs the result. Step (4) fuses the refined optical flow, mask, and residual to obtain the interpolated image.

2. The panoramic video frame interpolation method based on bidirectional separation correlation as described in claim 1, characterized in that, Step (1) is as follows: (1-1) The appearance feature encoder module extracts appearance features from the input image; The appearance feature encoder uses a pyramid-structured convolutional neural network to extract appearance feature values ​​at four different scales. superscript Indicates the number of floors. 1,2,3,4, subscript 1 and 2 represent the previous frame, respectively. and the next frame ; When appearance features pass through each layer of a convolutional neural network, the scale of the features changes from their original value. The number of channels is doubled; after the two consecutive frames of input images are processed by the appearance feature encoder, each image obtains four appearance feature values ​​at different scales. (1-2) The correlation encoder module extracts correlation features from the input image. The correlation encoder uses a pyramid-structured convolutional neural network. Each frame of the image outputs a correlation feature value when it passes through the pyramid network. and Its tensor size is , and Indicates the height and width of the image. This indicates the layer number of the features, with subscripts 0 and 1 representing the input layers, respectively. and .

3. The panoramic video frame interpolation method based on bidirectional separation correlation as described in claim 2, characterized in that, Step (4) is as follows: (4-1) Interpolated Image ; This indicates a backward warp operation, which uses optical flow to perform a spatial transformation on the image, mapping the pixels of the current frame to the corresponding positions at the time of the interpolation frame; This indicates pixel-by-pixel multiplication, and the resulting value is... arrive The mask Y and the back-distorted image are multiplied pixel by pixel; the residual S is used to compensate for the errors in the interpolation process in the synthesized image; (4-2) The interpolated image and the real intermediate image are calculated using a loss function to obtain the interpolation error. Backpropagation is then performed to optimize the parameters of each convolutional neural network and improve the accuracy of the interpolated image. loss function ; I represents the actual value of the intermediate frame. ERP weight is a constant. .

Images