Panoramic video frame insertion method based on bidirectional separation correlation quantity

Abstract

The invention discloses a panoramic video frame insertion method based on bidirectional separation correlation quantity. The method comprises the following steps: firstly, extracting appearance characteristics and correlation quantity characteristics of an input image, then, carrying out cost aggregation by utilizing the correlation quantity characteristics to obtain an initial optical flow, carrying out iterative updating on the initial optical flow to obtain refined optical flow, final characteristic values, masks and residual error information, and finally, fusing the information to obtain a frame insertion image. According to the method, a cost aggregation method is adopted to separate the generated correlation quantity, so that the special problems of motion anisotropy and non-uniform motion distribution existing in the panoramic video are more effectively solved, and the problems of artifacts and inaccurate estimation caused by the difference of sampling rates of projections in different dimensions are effectively solved. According to the method, motion query errors caused by motion amplitude differences on different latitudes can be better adapted, so that the precision of optical flow prediction is improved, and finally, a middle frame with higher quality is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of video processing technology, in particular to the field of video frame insertion technology, and relates to a panoramic video frame insertion method using flexible query bidirectional separation related quantities. Background Art

[0002] Video interpolation technology is an important video processing method that aims to generate additional intermediate frames based on two or more known frames to improve the smoothness and frame rate of the video. It is widely used in video encoding and decoding, slow motion generation, and recovery of lost frames in video streams. As a special form of video, panoramic video provides a full range of perspectives, allowing viewers to immerse themselves in a 360-degree visual experience. It is usually composed of multiple cameras simultaneously capturing scenes in different directions and synthesizing a complete spherical view through stitching software. However, due to the special nature of panoramic video, traditional video interpolation technology faces unprecedented challenges. For example, after being projected into a rectangular image by equirectangular projection, the panoramic video will have severe image distortion, especially in the top and bottom areas of the rectangular image. This distortion complicates traditional motion estimation based on planar images. In addition, the shooting equipment of panoramic video may be limited by technology and produce a low frame rate, resulting in a sense of jumpiness when the video is played. Especially in fast-motion scenes, this not only reduces the user experience, but also may cause discomfort to the viewer, such as motion sickness. At the same time, in a virtual reality environment, users can change their perspectives freely, which means that panoramic videos must be able to maintain consistent motion continuity at different perspectives, which places higher requirements on video interpolation technology, that is, accurately estimating motion and generating appropriate intermediate frames at different perspectives. Due to the existence of image distortion, traditional feature detection and matching methods often assume that images have uniform geometric properties, which makes it difficult to effectively apply them to panoramic videos. In addition, the motion in panoramic videos may not be simple translation or rotation, but complex three-dimensional spatial motion, which further increases the difficulty of implementing panoramic video interpolation technology. Summary of the invention

[0003] The purpose of the present invention is to propose a panoramic video interpolation method based on bidirectional separation correlation quantity, which is specially designed to solve the problems encountered in the interpolation process of panoramic video, such as image distortion, motion anisotropy and uneven motion distribution caused by equirectangular projection.

[0004] The frame insertion network is mainly composed of three parts: a dual-channel feature extraction sub-network, an optical flow refinement sub-network based on correlation separation and flexible query strategy, and a frame synthesis module. These three parts work closely together to ensure the successful completion of the panoramic video frame insertion task.

[0005] First, the dual-channel feature extraction subnetwork consists of a correlation feature encoder and an appearance feature encoder, which extract two types of features from the input image, namely correlation features and appearance features. The correlation features obtained by the correlation feature encoder will be used in the subsequent correlation generation and cost aggregation process to further refine the prediction of the optical flow field and the intermediate frame. The appearance feature encoder focuses on extracting the appearance features of the image, which will be used for optical flow calculation and iterative generation of intermediate frames.

[0006] Secondly, the optical flow refinement based on separation correlation and flexible query strategy uses the bidirectional separation correlation after cost aggregation to guide the iterative process of optical flow and intermediate features. This sub-network provides the necessary data basis for frame synthesis by generating information such as optical flow, intermediate values, residuals and masks.

[0007] Finally, the frame synthesis module fuses the above iteratively refined optical flow, intermediate value, residual and mask to obtain the final intermediate frame.

[0008] The specific steps of the method of the present invention are: Step (1) extracting appearance features and related quantity features of the input image; Step (2) using the relevant quantity features to perform cost aggregation to obtain the initial optical flow; Step (3) iteratively updates the initial optical flow to obtain refined optical flow, final eigenvalues, masks and residuals; Step (4) fuses the final eigenvalue, refined optical flow, mask, and residual to obtain an interpolated image.

[0009] The present invention uses a cost aggregation method to separate the generated related quantities, and more effectively handles the special problems of motion anisotropy and uneven motion distribution in panoramic videos. The method of the present invention can integrate the matching costs of the image in the horizontal and vertical directions, thereby effectively solving the problems of artifacts and inaccurate estimates caused by differences in sampling rates of projections in different dimensions. In addition, the flexible query strategy in the optical flow refinement network allows the use of offsets generated based on latitude information to obtain related quantity information when the optical flow query corresponds to the related quantity, which can better adapt to the motion query errors caused by differences in motion amplitudes at different latitudes, thereby improving the accuracy of optical flow prediction and ultimately obtaining higher quality intermediate frames.

[0010] By combining the separation of related quantities with a flexible query strategy, the panoramic video interpolation method proposed in the present invention can not only significantly improve the interpolation effect, but also enhance the user's immersive experience. It is particularly suitable for virtual reality and other application scenarios that require high-quality panoramic videos. DETAILED DESCRIPTION

[0011] A panoramic video interpolation method, through the feature extraction network, optical flow iteration network based on bidirectional separation correlation and flexible query, and frame synthesis network work together to improve the smoothness of the video while maintaining visual quality. The feature extraction network adopts dual-channel feature extraction and uses parallel paths to capture different features of the image, providing rich information for subsequent processing; the optical flow iteration network adopts a cost aggregation module to estimate object motion and enhances accuracy through optical flow refinement; the frame synthesis sub-network integrates features and optical flow information to generate intermediate frames. In addition, the encoder-decoder architecture is used to ensure the effective extraction and reconstruction of image features. The overall design is flexible and efficient, can adapt to complex scene changes, and can be applied to virtual reality, augmented reality and video surveillance fields, significantly improving the user's immersive experience.

[0012] 1. The feature extraction network uses two dedicated channels to extract different types of features: the correlation encoder module focuses on extracting the correlation information between the input images, which is used to form the cost volume; the appearance encoder module is responsible for extracting appearance features such as color, texture, and edges from the input image for the next step of optical flow generation and intermediate feature value generation.

[0013] 2. The optical flow iteration network includes a correlation quantity generation and separation module and a cost aggregation module. The cost aggregation module estimates the object motion and enhances the accuracy through optical flow refinement. The optical flow iteration network receives information from the appearance feature encoder and decodes it to obtain intermediate features. At the same time, the correlation quantity features obtained from the correlation quantity feature encoder calculate the matching degree between the two frames of images. First, a bidirectional cost volume is generated through an element-by-element multiplication operation, and then the cost volume is aggregated in the horizontal and vertical directions through the cost aggregation module to reduce noise and enhance the response of reliable matching points. Finally, it is iteratively updated, and the cost volume is queried through the query strategy, and the query results are used to optimize the optical flow and intermediate feature values. This process is repeated until the generated optical flow and intermediate feature values ​​reach the target accuracy of the interpolation task. The iterative refinement process helps to improve the accuracy and robustness of optical flow estimation.

[0014] The correlation generation module generates a 4D tensor that records the degree of matching between all pixels in the previous and next frames. The optical flow is used to query this tensor to obtain more accurate optical flow to guide the generation of high-quality intermediate frames. Then comes the refinement iterative network of optical flow and intermediate feature values. It iteratively optimizes the optical flow field and gradually improves its accuracy, thereby ensuring that the generated intermediate frames are as close to the real scene as possible. The accuracy of optical flow estimation directly affects the quality of the final intermediate frame. The joint action of the above two modules ensures the effective execution of the video interpolation task.

[0015] Workflow: First, the information from the appearance feature encoder module is received and decoded to obtain intermediate features. At the same time, in the correlation path, the correlation features obtained from the correlation feature encoder are used to calculate the match between the two frames of images by generating a bidirectional cost volume through element-by-element multiplication operations. These cost volumes are then further aggregated in the horizontal and vertical directions through the cost aggregation module to reduce noise and enhance the response of reliable matching points. Finally, through an iterative update method, the optical flow refinement module uses a flexible query strategy to query the cost volume, and uses the query results to continuously optimize the optical flow and related intermediate feature values. This process is repeated until the generated optical flow and intermediate feature values ​​reach the target accuracy of the interpolation task. The iterative refinement process helps to improve the accuracy and robustness of optical flow estimation.

[0016] 3. The frame synthesis network fuses the intermediate eigenvalues, optical flow field, mask, and residual information. In order to generate high-quality interpolated images, the final eigenvalues, optical flow, mask, residual and other information are fused. Among them, the intermediate eigenvalues ​​capture the structure and texture details in the original frame; the optical flow field provides the direction and speed information of the object movement between adjacent frames, which is crucial for predicting the movement of the intermediate frame; the mask is used to identify areas that need special attention, such as occluded parts or motion boundaries; and the residual represents the difference between the predicted image and the real image, which is used to further improve the image quality. Through the effective integration of this information, the frame fusion network can generate interpolated images that are both accurate and rich in details.

[0017] The specific steps of this method are as follows: Step (1) extracts the appearance features and related quantity features of the input image; specifically, as follows: (1-1) The appearance feature encoder module extracts appearance features from the input image. Appearance features include color features, texture features, and edge features. The appearance feature encoder uses a pyramid-structured convolutional neural network (CNN) to extract four layers of appearance feature values ​​at different scales. , superscript Indicates the number of layers, 1,2,3,4, subscript 1,2 represent the previous frame respectively and the next frame Each layer of feature values ​​is used for iterative update. When the appearance feature value passes through each layer of the convolutional neural network, the scale of the feature becomes the original , the number of channels is increased to twice the original. After the two input frames are passed through the appearance feature encoder, each image obtains four appearance feature values ​​of different scales.

[0018] (1-2) The correlation encoder module extracts correlation features from the input image. The correlation encoder uses a pyramid-structured convolutional neural network (CNN). When the two frames of images pass through the pyramid network, each outputs a correlation feature value. and , whose tensor size is , and Represents the height and width of the image, Indicates the number of layers of features, subscripts 0 and 1 represent the input and .

[0019] Step (2) uses the relevant quantity features to perform cost aggregation to obtain the initial optical flow; the details are as follows: (2-1) The correlation quantity is obtained by calculating the dot product of the two correlation quantity eigenvalues. , , is the number of eigenvalue channels of the relevant quantity, is the total number of channels of the eigenvalues ​​of the relevant quantity, , and represents the coordinates of the feature; , is a 4D tensor with coordinates , Represents the field of real numbers. The higher the value of the correlation quantity, the greater the degree of association between the corresponding two coordinates.

[0020] (2-2) Separate the correlation quantities and calculate the maximum and average values ​​of the correlation matrix in the horizontal and vertical directions respectively: The horizontal average value of the correlation quantity ; The maximum value of the horizontal direction of the correlation quantity ; The vertical average value of the correlation quantity ; The maximum vertical value of the correlation quantity ; , .

[0021] The average and maximum values ​​of the horizontal direction of the obtained correlation quantity are upsampled and concatenated to obtain the horizontal tensor ; Upsample the average and maximum values ​​of the vertical direction of the obtained correlation quantity, and concatenate them to obtain the vertical tensor .

[0022] (2-3) For the horizontal tensor Cost aggregation is performed in order from the top, bottom, left, and right directions to obtain the aggregated horizontal correlation quantities in the four directions: ; Aggregation direction (up, down, left, right), r represents the unit vector in the aggregation direction, Indicates the horizontal distance between the corresponding pixels of the previous and next frames, the variable , represents the sum operation, , , and Represents the four weights obtained by the input image through the appearance feature encoder convolutional neural network.

[0023] For the vertical tensor Cost aggregation is performed in order from the top, bottom, left, and right directions to obtain the aggregated vertical correlation quantities in the four directions: ; Indicates the vertical distance between corresponding pixels in the previous and next frames.

[0024] (2-4) Horizontal correlation Obtain the initialized horizontal optical flow component through the encoder-decoder network , vertical correlation The initialized vertical optical flow component is obtained through the encoder-decoder network ; The codec network is a multi-layer convolutional neural network, which first performs layer-by-layer convolution operations on the input, then performs layer-by-layer deconvolution operations, and then outputs.

[0025] The optical flow components in two directions are spliced ​​to obtain the forward initialized optical flow , cat means splicing.

[0026] (2-5) For related quantities Transpose to obtain the relevant amount after transposition , repeat (2-2) to (2-4) to obtain the reverse initialization optical flow .

[0027] Step (3) iteratively updates the initial optical flow to obtain refined optical flow, final eigenvalues, masks and residuals; the details are as follows: (3-1) Use forward initialization optical flow And reverse initialize optical flow For related quantities as well as and To query, the query strategy is to perform position encoding based on the position of the pixel latitude in the panoramic image to obtain the offset for query, input a position information related to the latitude into the position encoder to obtain the offset related to the position information, and then add the offset to the original query radius to obtain the query amount. The details are as follows:

[0028] First, based on the position information at each pixel coordinate Generate query box , query range , a and b are the offsets of the horizontal and vertical coordinates respectively, and L is the side length of the query box.

[0029] Then initialize the optical flow forward And reverse initialize optical flow Input position encoder , generate a positive offset based on the position information p and reverse offset : , ; The position information p is obtained by calculating the vertical distance between a pixel and the height midline of the panoramic image (i.e., the equatorial position) and then normalizing the distance.

[0030] Query Box Add the positive offset obtained Get forward query location , query box Add the reverse offset obtained Get reverse query location , and are the horizontal and vertical coordinates of the offset respectively.

[0031] Using forward query location Query related quantity , get the forward query volume ; Use reverse query location Query the relevant quantity after transposition , get the reverse query volume .

[0032] (3-2) Forward query volume Input into the update module U to obtain the forward optical flow increment and the positive intermediate eigenvalue increment ; Reverse query volume Input into the update module U to obtain the reverse optical flow increment and reverse intermediate eigenvalue increment The update module U is a multi-layer convolutional neural network, which performs layer-by-layer convolution operations on the input and then outputs it.

[0033] (3-3) Increment the positive intermediate eigenvalue and reverse intermediate eigenvalue increment Splice to get the intermediate eigenvalue increment ; Appearance feature value With the intermediate eigenvalue increment Add together to get the first-level intermediate eigenvalue .

[0034] Forward optical flow increment With forward initialization optical flow Add together to get the first-level forward optical flow ; Reverse optical flow increment Initialize the optical flow with reverse Add together to get the first-level reverse optical flow .

[0035] (3-4) Using the first-level forward optical flow and first-level reverse optical flow For related quantities as well as and Perform a query and repeat steps (3-1) to (3-3) to obtain the secondary intermediate eigenvalue, the secondary forward optical flow, and the secondary reverse optical flow; (3-5) Repeat step (3-4) to iterate, and stop the iteration when the set number of iterations is reached to obtain the intermediate eigenvalues, forward optical flow, and reverse optical flow.

[0036] (3-6) Input the intermediate eigenvalue, forward optical flow and reverse optical flow into the decoder to obtain the final eigenvalue , forward refinement optical flow and reverse refinement optical flow , and the mask Y and residual S are obtained at the same time. The decoder is a multi-layer convolutional neural network, which performs convolution operations on the input layer by layer and then outputs it.

[0037] Step (4) fuses the final eigenvalue, refined optical flow, mask, and residual to obtain an interpolated image; the details are as follows: (4-1) Insertion Image ; Represents the backward warping operation, which uses optical flow to spatially transform the image and map the pixels of the current frame to the corresponding positions at the time of the interpolated frame; Indicates pixel-by-pixel multiplication, and the obtained value is arrive The mask Y between the two images is multiplied pixel by pixel with the image after backward warping; the difference term S is used to compensate the error in the interpolation process in the synthesized image.

[0038] (4-2) The generated interpolated image and the real intermediate image are calculated through the loss function to obtain the interpolated error, and then back-propagation is performed to optimize the parameters of each convolutional neural network to improve the accuracy of the interpolated image.

[0039] Loss Function ; I represents the true value of the intermediate frame, is a very small constant to ensure that the denominator is not zero in the network backward pass. The design principle of this weight function is that in equirectangular projection, the top and bottom of the image will be greatly distorted due to projection deformation, so the pixels in these areas are less important for reconstructing or processing panoramic images. By giving these areas smaller weights, the impact of these areas on the overall image processing results can be reduced.

Claims

1. A panoramic video frame insertion method based on bidirectional separation correlation quantity, characterized in that: Step (1) extracting appearance features and related quantity features of the input image; Step (2) using the relevant quantity features to perform cost aggregation to obtain the initial optical flow; Step (3) iteratively updates the initial optical flow to obtain refined optical flow, final eigenvalues, masks and residuals; Step (4) fuses the final eigenvalue, refined optical flow, mask, and residual to obtain an interpolated image.

2. The panoramic video frame insertion method based on bidirectional separation correlation according to claim 1, characterized in that: Step (1) is as follows: (1-1) The appearance feature encoder module extracts appearance features from the input image; The appearance feature encoder uses a pyramid-structured convolutional neural network to extract four layers of appearance feature values ​​at different scales. , superscript Indicates the number of layers, 1,2,3,4, subscript 1,2 represent the previous frame respectively and the next frame ; When the appearance feature value passes through each layer of the convolutional neural network, the scale of the feature becomes the original , the number of channels is increased to twice the original; after the two input frames pass through the appearance feature encoder, each image obtains four appearance feature values ​​of different scales; (1-2) The correlation encoder module extracts correlation features from the input image. The correlation encoder uses a pyramid-structured convolutional neural network. When the two frames of images pass through the pyramid network, each outputs a correlation feature value. and , whose tensor size is , and Represents the height and width of the image, Indicates the number of layers of features, subscripts 0 and 1 represent the input and .

3. The panoramic video frame insertion method based on bidirectional separation correlation according to claim 2, characterized in that: Step (2) is as follows: (2-1) The correlation quantity is obtained by calculating the dot product of the two correlation quantity eigenvalues. , , is the number of eigenvalue channels of the relevant quantity, is the total number of channels of the eigenvalues ​​of the relevant quantity, and represents the coordinates of the feature; , represents the field of real numbers, is a 4D tensor with coordinates ; (2-2) Separate the correlation quantities and calculate the maximum and average values ​​of the correlation matrix in the horizontal and vertical directions respectively: The horizontal average value of the correlation quantity ; The maximum value of the horizontal direction of the correlation quantity ; The vertical average value of the correlation quantity ; The maximum vertical value of the correlation quantity ; , ; The average and maximum values ​​of the horizontal direction of the obtained correlation quantity are upsampled and concatenated to obtain the horizontal tensor ; Upsample the average and maximum values ​​of the vertical direction of the obtained correlation quantity, and concatenate them to obtain the vertical tensor ; (2-3) For the horizontal tensor Cost aggregation is performed in order from the top, bottom, left, and right directions to obtain the aggregated horizontal correlation quantities in the four directions: ; Aggregation direction (up, down, left, right), r represents the unit vector in the aggregation direction, Indicates the horizontal distance between the corresponding pixels of the previous and next frames, the variable , represents the sum operation, , , and Represents the four weights obtained by the input image through the appearance feature encoder convolutional neural network; For the vertical tensor Cost aggregation is performed in order from the top, bottom, left, and right directions to obtain the aggregated vertical correlation quantities in the four directions: ; Indicates the vertical distance between corresponding pixels of the previous and next frames; (2-4) Horizontal correlation Obtain the initialized horizontal optical flow component through the encoder-decoder network , vertical correlation The initialized vertical optical flow component is obtained through the encoder-decoder network ; The codec network is a multi-layer convolutional neural network, which first performs layer-by-layer convolution operations on the input, then performs layer-by-layer deconvolution operations, and then outputs; The optical flow components in two directions are spliced ​​to obtain the forward initialized optical flow , cat means splicing; (2-5) For related quantities Transpose to obtain the relevant amount after transposition , repeat (2-2) to (2-4) to obtain the reverse initialization optical flow .

4. The panoramic video frame insertion method based on bidirectional separation correlation according to claim 3, characterized in that: Step (3) is as follows: (3-1) Use forward initialization optical flow And reverse initialize optical flow For related quantities as well as and Perform a query to obtain the query volume; the details are as follows: First, based on the position information at each pixel coordinate Generate query box , query range , a and b are the offsets of the horizontal and vertical coordinates respectively, and L is the side length of the query box; Then initialize the optical flow forward And reverse initialize optical flow Input position encoder , generate a positive offset based on the position information p and reverse offset : , ; The position information p is obtained by calculating the vertical distance between a pixel and the height midline of the panoramic image (i.e., the equatorial position), and then normalizing the distance; Query Box Add the positive offset obtained Get forward query location , query box Add the reverse offset obtained Get reverse query location , and are the horizontal and vertical coordinates of the offset respectively; Using forward query location Query related quantity , get the forward query volume ; Use reverse query location Query the relevant quantity after transposition , get the reverse query volume ; (3-2) Forward query volume Input into the update module U to obtain the forward optical flow increment and the positive intermediate eigenvalue increment ; Reverse query volume Input into the update module U to obtain the reverse optical flow increment and reverse intermediate eigenvalue increment ; The update module U is a multi-layer convolutional neural network, which performs convolution operations on the input layer by layer and then outputs; (3-3) Increment the positive intermediate eigenvalue and reverse intermediate eigenvalue increment Splice to get the intermediate eigenvalue increment ; Appearance feature value With the intermediate eigenvalue increment Add together to get the first-level intermediate eigenvalue ; Forward optical flow increment With forward initialization optical flow Add together to get the first-level forward optical flow ; Reverse optical flow increment Initialize the optical flow with reverse Add together to get the first-level reverse optical flow ; (3-4) Using the first-level forward optical flow and first-level reverse optical flow For related quantities as well as and Perform a query and repeat steps (3-1) to (3-3) to obtain the secondary intermediate eigenvalue, the secondary forward optical flow, and the secondary reverse optical flow; (3-5) Repeat step (3-4) to iterate, and stop the iteration when the set number of iterations is reached to obtain the intermediate eigenvalue, forward optical flow, and reverse optical flow; (3-6) Input the intermediate eigenvalue, forward optical flow and reverse optical flow into the decoder to obtain the final eigenvalue , forward refinement optical flow and reverse refinement optical flow , and obtain the mask Y and residual S at the same time; The decoder is a multi-layer convolutional neural network that performs convolution operations on the input layer by layer and then outputs it.

5. The panoramic video frame insertion method based on bidirectional separation correlation according to claim 4, characterized in that: Step (4) is as follows: (4-1) Insertion Image ; Represents the backward warping operation, which uses optical flow to spatially transform the image and map the pixels of the current frame to the corresponding positions at the time of the interpolated frame; Indicates pixel-by-pixel multiplication, and the obtained value is arrive The mask Y between the two images is multiplied pixel by pixel with the image after backward warping. The difference term S is used to compensate the error in the interpolation process in the synthesized image. (4-2) The generated interpolation image and the real intermediate image are calculated through the loss function to obtain the interpolation error, and then back propagation is performed to optimize the parameters of each convolutional neural network to improve the accuracy of the interpolation image; Loss Function ; I represents the true value of the intermediate frame, is a constant, ERP weight .

Images