A method, apparatus, device, and storage medium for determining intermediate frames in a video.

Abstract

This invention discloses a method, apparatus, device, and storage medium for determining intermediate video frames. The method includes: acquiring adjacent video frames of the frame to be interpolated; performing branch feature extraction on the adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction; performing optical flow estimation on the video depth features to obtain optical flow motion information; and constructing a target intermediate frame based on the optical flow motion information and the adjacent video frames. This invention's technical solution can better capture motion trajectories in fast-moving small object scenes, ensuring the smoothness of the video intermediate frame interpolation result, solving the error problem when small objects are moving, and significantly improving the interpolation effect.

Description

Technical Field

[0001] This invention relates to the field of video frame processing technology, and in particular to a method, apparatus, device, and storage medium for determining intermediate video frames. Background Technology

[0002] With the rapid development of multimedia technology, video applications have permeated all aspects of daily life. To improve the video viewing experience, increasing the video frame rate has become a key task. Video frame interpolation technology increases the frame rate by inserting extra frames between adjacent frames in low frame rate videos, and is widely used in scenarios such as video playback optimization, slow-motion video generation, virtual reality (VR), and augmented reality (AR).

[0003] Traditional video frame interpolation methods, such as inserting black frames, frame averaging, or frame repetition, are simple to implement but often lead to severe motion blur and jitter problems, especially when dealing with fast-moving small objects. They also have room for improvement in terms of computational complexity. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for determining intermediate frames in video. In scenes with small objects moving rapidly, it can better capture motion trajectories, ensure the smoothness of the interpolation results of intermediate frames, solve the error problem when small objects are moving, and significantly improve the interpolation effect.

[0005] According to one aspect of the present invention, a method for determining intermediate frames in a video is provided, the method comprising:

[0006] Obtain the adjacent video frames of the frame to be interpolated;

[0007] Branch feature extraction is performed on the adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction;

[0008] Optical flow estimation is performed on the video depth features to obtain optical flow motion information;

[0009] Based on the optical flow motion information and the adjacent video frames, a target intermediate frame is constructed.

[0010] According to another aspect of the present invention, a video intermediate frame determination apparatus is provided. The apparatus includes:

[0011] The adjacent video frame acquisition module is used to acquire the adjacent video frames of the frame to be interpolated;

[0012] The depth feature determination module is used to extract branch features from the adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction.

[0013] The optical flow information determination module is used to perform optical flow estimation on the video depth features to obtain optical flow motion information;

[0014] The target intermediate frame determination module is used to construct a target intermediate frame based on the optical flow motion information and the adjacent video frames.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video intermediate frame determination method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video intermediate frame determination method according to any embodiment of the present invention.

[0020] The technical solution of this invention involves acquiring adjacent video frames of the frame to be interpolated. Branch feature extraction is performed on these adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction. Optical flow estimation is performed on the video depth features to obtain optical flow motion information. Based on the optical flow motion information and the adjacent video frames, a target intermediate frame is constructed. In scenes with fast-moving small objects, this method can better capture motion trajectories, thereby improving the accuracy of determining the target intermediate frame, thus improving the interpolation effect, enhancing the smoothness of the interpolation result, and solving the error problem when small objects are moving.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a video intermediate frame determination method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a video intermediate frame determination method according to Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the construction of a target intermediate frame according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of spatial feature mining processing provided according to Embodiment 2 of the present invention;

[0027] Figure 5 This is a schematic diagram of temporal feature mining processing provided in Embodiment 2 of the present invention;

[0028] Figure 6 This is a structural diagram of a video intermediate frame determination device according to Embodiment 3 of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the video intermediate frame determination method of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1This is a flowchart of a video intermediate frame determination method provided in Embodiment 1 of the present invention. This embodiment is applicable to real-time video processing, video frame rate enhancement, and slow-motion video generation, etc. The method can be executed by a video intermediate frame determination device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S101. Obtain the adjacent video frames of the frame to be interpolated.

[0035] In this context, adjacent video frames can refer to two adjacent frames of the video frame to be inserted. For example, any two consecutive frames in an original video.

[0036] Specifically, the original video is split into segments, and the specific position where frames are to be inserted is selected according to actual needs. The video frames adjacent to that position are then identified as adjacent video frames.

[0037] S102. Extract branch features from the adjacent video frames to obtain video depth features.

[0038] Here, video depth features can refer to the depth features between adjacent video frames. It should be noted that branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction.

[0039] Specifically, coarse-grained feature extraction and fine-grained feature extraction are performed on adjacent video frames respectively, and the extracted results are fused to obtain video depth features.

[0040] S103. Perform optical flow estimation on the video depth features to obtain optical flow motion information.

[0041] Among them, optical flow motion information can refer to the displacement information of each pixel in the video depth features between consecutive frames.

[0042] Specifically, optical flow motion information can be directly predicted by training neural networks using deep learning methods, or it can be directly calculated using common optical flow estimation algorithms, such as the Lucas-Kanade method and the Farneback method.

[0043] For example, the step of performing optical flow estimation on the video depth features to obtain optical flow motion information includes: separating the video depth features according to the adjacent video frames to obtain a first feature image and a second feature image corresponding to the adjacent video frames; determining image-related features between the first feature image and the second feature image based on pixel-wise inner product; and aggregating the image-related features to determine the optical flow motion information between the adjacent video frames.

[0044] Here, the first feature image can refer to the feature image corresponding to any one of the adjacent video frames, and the second feature image can refer to the feature image corresponding to another adjacent video frame. The inner product is a method for measuring the similarity between two vectors; in this invention, the inner product can be used to determine the feature similarity between the first feature image and the second feature image at the same spatial location.

[0045] Specifically, the extracted depth features are separated to obtain a first feature image corresponding to any one of the adjacent video frames and a second feature image corresponding to another adjacent video frame. A pixel-by-pixel inner product is calculated on the first and second feature images to obtain an image-related feature of the same size as both the first and second feature images. Each value in the image-related feature represents the feature similarity between the first and second feature images at a corresponding location. The image-related features are then aggregated to extract useful optical flow motion information. This can be achieved by calculating the maximum, minimum, and average values ​​of the image-related features or by applying a more complex aggregation strategy. Based on the results of the aggregation process, the optical flow motion information between adjacent video frames can be determined.

[0046] Specifically, the video depth features f are first separated, and feature maps of two adjacent frames are extracted, i.e., the first feature image. Second feature image Next, a pixel-wise inner product operation is used to calculate the image correlation features between the first feature image f1 and the second feature image f2. The 4D correlation is defined as... To capture a wide range of pixel motion, multi-level downsampling is performed on the third and fourth dimensions of the 4D correlation C, generating multiple correlation layers {C}. 0 C 1 C 2 C 3}, This processing operation not only effectively captures motion information with large amplitudes but also preserves high-resolution feature dimensions to recover detailed texture information. To improve the accuracy of optical flow estimation, this invention also defines a lookup table operation to optimize the precision of the optical flow. A local neighborhood search mechanism is defined in the lookup table operation, assuming the current optical flow is (F... 1 ,F 2 Mapping each pixel coordinate x = (u, v) in f1 to the corresponding position x′ in f2 can be done as follows:

[0047] x′=(u+F 1 (u),v+F 2 (v));

[0048] And through the local neighborhood N(x′) of x′ r To find the most relevant features. Local neighborhood N(x′) r As shown in formula (4-14):

[0049] N(x′) r ={x′+dx};

[0050] Where, N(x′) r Let x' represent the set of points with center x' and radius r, where ||dx|| 1 ≤ r. This can be achieved using N(x'). r The most relevant feature information can be queried and aggregated from various related features, thereby optimizing optical flow and effectively capturing motion information between adjacent video frames. 0→1 and F 1→0 .

[0051] S104. Construct the target intermediate frame based on the optical flow motion information and the adjacent video frames.

[0052] The target intermediate frame can refer to the insertion frame of the adjacent video frame to be inserted.

[0053] Specifically, optical flow motion information is used to map the pixels of the previous frame in adjacent video frames to the pixel coordinate system of the next frame, thus simulating the temporal motion trajectory of the pixels. For each pixel position in the target intermediate frame, interpolation is performed using the pixel values ​​of the corresponding positions in adjacent frames and the optical flow motion information. All interpolated pixel values ​​are combined to form the image of the target intermediate frame. The architecture of the fusion network used to construct the target intermediate frame in this invention is based on a simplified three-layer U-Net network, containing three downsampling layers and three upsampling layers.

[0054] It should be noted that during pixel interpolation, occlusion areas or motion ambiguity may be encountered. These areas may not have direct corresponding pixels in adjacent frames. Various strategies can be used to handle this, such as using pixel values ​​from the previous or next frame, performing smoothing, or applying deep learning models to predict missing pixel values.

[0055] For example, constructing a target intermediate frame based on the optical flow motion information and the adjacent video frames includes: determining a video intermediate anchor frame based on the intermediate anchor frame algorithm, according to the optical flow motion information and the adjacent video frames; constructing an anchor frame mask and a residual image corresponding to the video intermediate anchor frame based on the video intermediate anchor frame; and constructing the target intermediate frame based on the intermediate frame algorithm, according to the anchor frame mask, the residual image, and the optical flow motion information.

[0056] In this context, the intermediate anchor frame can be a hypothetical frame located temporally between adjacent video frames. The anchor frame mask can be a weight matrix of the same size as the adjacent video frames. The anchor frame mask is a binary image of the same size as the intermediate anchor frame, where the value of each pixel represents the importance of the original frame and the anchor frame at that location. A pixel with a value of 1 indicates that the location is important or significantly changed within the anchor frame, while a pixel with a value of 0 indicates that the location has no significant change. The residual image refers to the difference between adjacent video frames and the intermediate anchor frame.

[0057] Specifically, the intermediate anchor frame algorithm is as follows:

[0058]

[0059] in, Let t represent any time interval [0,1], W represent the backward mapping, and F represent the intermediate anchor frame. 0→t With F 1→t The optical flow motion information from adjacent video frames to time t is represented by I0 and I1, which represent adjacent video frames.

[0060] By substituting optical flow motion information and adjacent video frames into the intermediate anchor frame algorithm, intermediate anchor frames that represent the motion changes between adjacent video frames can be obtained. By comparing the content of the intermediate anchor frame with that of adjacent video frames, anchor frame masks and residual images can be generated, which helps to preserve key information and details in the subsequent construction of target intermediate frames. Substituting the anchor frame mask and residual image into the intermediate anchor frame algorithm, intermediate anchor frames that represent the motion changes between adjacent video frames can be obtained.

[0061] For example, the target intermediate frame algorithm is as follows:

[0062] I t =M⊙W(I0,F 0→t )+(1-M)⊙W(I1,F 1→t )+ΔI t ;

[0063] Among them, I t Indicates the target intermediate frame, M represents the anchor frame mask, ⊙ represents element-wise multiplication, W represents back mapping, and F represents the target intermediate frame. 0→t With F 1→t This represents the optical flow motion information from adjacent video frames to time t, where I0 and I1 represent adjacent video frames, and ΔI t This represents the residual image. Through the above fusion operations, high-quality intermediate frames are finally generated. The back-mapping method used enhances the coherence between frames and ensures visual consistency. The fusion process aims to utilize the feature information of multiple anchor frames to compensate for each other, thereby achieving high-quality video frame interpolation.

[0064] The technical solution of this invention involves acquiring adjacent video frames of the frame to be interpolated. Branch feature extraction is performed on these adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction. Optical flow estimation is performed on the video depth features to obtain optical flow motion information. Based on the optical flow motion information and the adjacent video frames, a target intermediate frame is constructed. In scenes with fast-moving small objects, this method can better capture motion trajectories, thereby improving the accuracy of determining the target intermediate frame, thus improving the interpolation effect, enhancing the smoothness of the interpolation result, and solving the error problem when small objects are moving.

[0065] Example 2

[0066] Figure 2 This is a flowchart of a video intermediate frame determination method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment further refines the process of extracting branch features from adjacent video frames to obtain video depth features. For example... Figure 2 As shown, the method includes:

[0067] S201. Obtain the adjacent video frames of the frame to be interpolated.

[0068] S202. Extract coarse-grained features from the adjacent video frames to obtain coarse-grained video features.

[0069] It should be noted that coarse-grained feature extraction refers to the operation of extracting features from adjacent video frames using a pre-selected coarse-grained convolution kernel, which can be 5*5.

[0070] In one embodiment of the present invention, coarse-grained convolution kernels can be used to extract coarse-grained features from adjacent video frames to obtain coarse-grained video features.

[0071] S203. Fine-grained feature extraction is performed on the adjacent video frames to obtain fine-grained video features.

[0072] Similarly, coarse-grained feature extraction refers to the operation of extracting features from adjacent video frames using a pre-selected coarse-grained convolutional kernel, while fine-grained convolutional kernels can be 3x3. Fine-grained features are obtained by using fine-grained convolutional kernels to extract fine-grained features from adjacent video frames.

[0073] S204. The coarse-grained features and fine-grained features of the video are fused to obtain the video depth features corresponding to the adjacent video frames.

[0074] The feature extraction process includes at least convolution processing, wherein the convolution kernel for coarse-grained feature extraction is larger than the convolution kernel for fine-grained feature extraction.

[0075] Specifically, in the coarse-grained branch, convolutional layers are used to extract coarse-grained video features f. c In the fine-grained branch, convolutional layers are used to extract coarse-grained video features f. s By fusing elements through addition, we obtain video depth features f with multiple scales.

[0076]

[0077] in, f represents the element-by-element addition operation. s This refers to coarse-grained features of the video, f s "f" refers to fine-grained video features, while "f" refers to video depth features.

[0078] Figure 3 This is a schematic diagram illustrating the construction of a target intermediate frame as provided in an embodiment of the present invention. Figure 3 As shown, the process of determining video depth features includes a coarse-grained feature extraction stage and a fine-grained feature extraction stage. In the coarse-grained feature extraction stage, the convolutional layer kernel example is 5*5, while in the fine-grained feature extraction stage, the convolutional layer kernel example is 3*3. Fusing the coarse-grained and fine-grained feature extraction results yields the video depth features. This structure not only enhances the perception of feature information at different scales but also provides a richer and more accurate information foundation for optical flow estimation.

[0079] S205. Perform optical flow estimation on the video depth features to obtain optical flow motion information.

[0080] S206. Construct a target intermediate frame based on the optical flow motion information and the adjacent video frames.

[0081] For example, the step of extracting coarse-grained features from the adjacent video frames to obtain coarse-grained video features includes:

[0082] Based on a preset coarse-grained convolution kernel, the adjacent video frames are subjected to coarse-grained convolution processing to obtain coarse-grained shallow features; the coarse-grained shallow features are then subjected to spatiotemporal feature mining in the temporal and spatial domains to obtain video coarse-grained features.

[0083] Among them, coarse-grained shallow features can refer to video frame features obtained through coarse-grained convolution kernels.

[0084] Specifically, the process of mining coarse-grained video features can also involve performing coarse-grained convolution on adjacent video frames using coarse-grained convolution kernels to obtain coarse-grained shallow features. To improve the accuracy of feature extraction, this invention further mines the coarse-grained shallow features in both the temporal and spatial domains, thereby obtaining coarse-grained video features.

[0085] For example, the step of performing spatiotemporal feature mining on the coarse-grained shallow features in the temporal and spatial domains to obtain coarse-grained video features includes:

[0086] The coarse-grained shallow features are subjected to spatial feature mining processing to obtain coarse-grained spatial features;

[0087] The coarse-grained shallow features are subjected to temporal feature mining to obtain coarse-grained temporal features;

[0088] Based on the coarse-grained spatial features and the coarse-grained temporal features, video coarse-grained features are constructed.

[0089] Specifically, spatial feature mining is performed on the coarse-grained shallow features to obtain coarse-grained spatial features O. s The coarse-grained shallow features are then subjected to temporal feature mining to obtain coarse-grained temporal features O. t O s and O t The coarse-grained video feature O1 is obtained by summing the features, as expressed by the following formula:

[0090]

[0091] Among them, O1 video coarse-grained features, O s Represents coarse-grained spatial features, O t Represents coarse-grained time-domain features. This represents the summation operation of eigenvectors.

[0092] For example, the step of performing spatial feature mining processing on the coarse-grained shallow features to obtain coarse-grained spatial features includes: performing channel dimension division processing on the coarse-grained shallow features to obtain spatial horizontal shallow features and spatial vertical shallow features; performing multi-head attention extraction on the spatial horizontal shallow features to obtain spatial horizontal correlation features; performing multi-head attention extraction on the spatial vertical shallow features to obtain spatial vertical correlation features; and constructing coarse-grained spatial features based on the spatial horizontal correlation features and the spatial vertical correlation features.

[0093] Among them, the spatial horizontal shallow features and spatial vertical shallow features are the processing results obtained by dividing the coarse-grained shallow features in the channel dimension. The spatial horizontal correlation features and spatial vertical correlation features are the results of multi-head attention extraction.

[0094] Figure 4 This is a schematic diagram of spatial feature mining processing provided in an embodiment of the present invention. Figure 4As shown, spatial feature mining is divided into two parallel branches: horizontal spatial feature mining and vertical spatial feature mining, in order to extract long-range spatial feature correlations. Layer normalization (LN) is used to process coarse-grained shallow features. The spatial horizontal shallow features are divided into channels. Spatial vertical shallow features It can be expressed by the following formula:

[0095]

[0096] Here, Split(·) represents the segmentation operation, and LN(·) represents the normalization operation, which transforms coarse-grained shallow features. Segmented into shallow lateral features Spatial vertical shallow features

[0097] Multi-head attention is used to extract the acquired shallow spatial horizontal and vertical features to obtain spatially correlated features. These features are then summed to construct coarse-grained spatial features.

[0098] For example, the step of extracting spatial horizontal shallow features through multi-head attention to obtain spatial horizontal related features includes: determining the query vector, key vector, and value vector corresponding to the spatial horizontal shallow features based on the multi-head attention mechanism;

[0099] For each dimension, based on the query vector, the key vector, and the value vector, determine the horizontal branch-related features corresponding to the shallow horizontal features of the space;

[0100] All the aforementioned lateral branch-related features are spliced ​​together along the channel dimension to obtain spatial lateral correlation features.

[0101] Specifically, in the spatial lateral feature mining branch, the query vector is estimated by utilizing a multi-head self-attention mechanism. key vector Sum value vector It can be expressed by the following formula:

[0102]

[0103] in, and The transformation matrix is ​​represented by m, where m∈{1,2,...,M} represents the number of heads in the multi-head attention mechanism. In this invention, M is preferably set to 8.

[0104] query vector key vector Sum value vector Convert to a two-dimensional vector to facilitate the subsequent merging of features related to the horizontal branches. The dimensional information after transformation is as follows: The transformed 2D vector's H is then used as the horizontal space dividing line. The horizontal branch-related features are extracted using multi-head attention in the horizontal space. It can be expressed by the following formula:

[0105]

[0106] Where W represents the feature width, C′ represents the number of channels, and M represents the number of multi-head attention heads. This refers to query vectors, This refers to the key vector and This refers to the value vector. Next, the relevant features of the lateral branches will be... Dimensional information transformation This facilitates the merging of different branches in a multi-head attention mechanism. Subsequently, the lateral branch correlation features of different branches are concatenated along the channel dimension to obtain a spatial lateral correlation feature that integrates multi-dimensional information. It can be expressed by the following formula:

[0107]

[0108] Wherein, Concat(·) represents the concatenation operation. This indicates the characteristics related to horizontal branches.

[0109] For example, constructing coarse-grained spatial features based on the spatial lateral correlation features and the spatial vertical correlation features includes: performing convolution processing on the spatial lateral correlation features and the spatial vertical correlation features to obtain spatial convolutional correlation features; summing the spatial convolutional correlation features with the coarse-grained shallow features to obtain first spatial correlation features; obtaining second spatial correlation features based on the first spatial correlation features according to a multilayer perception mechanism; and summing the first spatial correlation features and the second spatial correlation features to obtain coarse-grained spatial features.

[0110] Specifically, the spatial horizontal correlation features and spatial vertical correlation features are summed, and the summation result is convolved to obtain spatial convolutional correlation features. These spatial convolutional correlation features are then summed with the coarse-grained shallow features to obtain the first spatial correlation feature. The first spatial correlation feature is shown below:

[0111]

[0112] Among them, O s,h This refers to the spatial lateral correlation characteristics, O s,v This refers to spatial longitudinal correlation characteristics. The expression represents the summation of eigenvectors, and Conv(·) represents a 1×1 convolution operation. This refers to the characteristics related to the first space.

[0113] Based on the multi-layer sensing mechanism, a second spatial correlation feature is obtained from the first spatial correlation feature. The first and second spatial correlation features are then summed to obtain coarse-grained spatial features, as shown in the following formula:

[0114]

[0115] in, This refers to the first spatial correlation features, where MLP(·) represents a multilayer perceptron, LN(·) represents a normalization operation, and O s This refers to coarse-grained spatial features. The above method enables the extraction of deeper spatial features from coarse-grained shallow features, providing a rich information foundation for subsequent operations.

[0116] For example, the step of performing temporal feature mining processing on the coarse-grained shallow features to obtain coarse-grained temporal features includes: performing channel dimension partitioning processing on the coarse-grained shallow features to obtain temporal horizontal shallow features and temporal vertical shallow features; performing multi-head attention extraction on the temporal horizontal shallow features to obtain temporal horizontal correlation features; performing multi-head attention extraction on the temporal vertical shallow features to obtain temporal vertical correlation features; and constructing coarse-grained temporal features based on the temporal horizontal correlation features and the temporal vertical correlation features.

[0117] Among them, the temporal horizontal shallow features and temporal vertical shallow features are the processing results obtained by dividing the coarse-grained shallow features in the channel dimension. The temporal horizontal correlation features and temporal vertical correlation features are the results of multi-head attention extraction.

[0118] Figure 5 This is a schematic diagram of temporal feature mining processing provided in an embodiment of the present invention. Figure 5 As shown, the temporal feature mining process is divided into two parallel branches: temporal horizontal feature mining and temporal vertical feature mining, in order to extract long-range temporal feature correlations. Multi-head attention is applied to the acquired temporal horizontal and vertical shallow features to extract temporal horizontal and vertical correlation features, respectively. Then, these temporal horizontal and vertical correlation features are summed to construct coarse-grained temporal features.

[0119] For example, the branching process of temporal feature mining is also divided into two branches: horizontal temporal strip attention and vertical strip attention. In terms of channel dimension, it is divided into temporal horizontal shallow features O. hand temporal domain vertical shallow features O v As shown in the following formula:

[0120] (O h O v ) = Split(LN(O s ));

[0121] Where LN(·) represents the layer normalization operation, and Split(·) represents the splitting operation, which divides the coarse-grained shallow features O s Segmented into temporal lateral shallow features

[0122] O h With temporal domain vertical shallow features O v Lateral features This represents horizontal bars that do not overlap and are located on the same horizontal coordinate. i∈{1,2,...,H}. Next, the query vector is obtained using the following formula. key vector Sum value vector

[0123]

[0124] in, and This represents the corresponding transformation matrix. and Convert to a two-dimensional vector and obtain temporal correlation features As shown in the following formula:

[0125]

[0126] Where W represents the feature width, C′ represents the number of channels, and M represents the number of multi-head attention heads. The relevant features obtained from multi-head attention are then concatenated along the channel dimension to obtain... The different horizontal bars are then spliced ​​together to obtain the temporal lateral feature O. t,h .

[0127] For vertical feature O t,v The acquisition of these features follows a similar process to that used for lateral features. The final attention-perceived feature information O is obtained based on the following formula. t .

[0128]

[0129] in, The summation operation of the feature vector is represented by Conv(·), the 1×1 convolution operation is represented by MLP(·), the multilayer perceptron is represented by LN(·), and the normalization operation is represented by LN(·).

[0130] It should be noted that the process of mining fine-grained features in videos is the same as that of mining coarse-grained features in videos, and will not be described in detail here.

[0131] The technical features of this invention involve processing coarse-grained shallow features using a multi-head attention mechanism in both the temporal and spatial domains, with attention mechanisms applied to the horizontal and vertical branches respectively. This effectively captures coarse-grained video features across multiple spatiotemporal directions, combining features at both coarse and fine-grained scales. This further enhances the model's understanding of features in adjacent video frames, thereby efficiently completing the video frame interpolation task. This invention significantly improves frame interpolation performance, especially in scenes with fast-moving small objects, better capturing motion trajectories, reducing computational complexity, significantly accelerating processing speed, and solving the error problem associated with small object movement.

[0132] Example 3

[0133] Figure 6 This is a schematic diagram of a video intermediate frame determination device provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes:

[0134] The adjacent video frame acquisition module 301 is used to acquire the adjacent video frames of the frame to be interpolated;

[0135] The depth feature determination module 302 is used to extract branch features from the adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction.

[0136] The optical flow information determination module 303 is used to perform optical flow estimation on the video depth features to obtain optical flow motion information;

[0137] The target intermediate frame determination module 304 is used to construct a target intermediate frame based on the optical flow motion information and the adjacent video frames.

[0138] The technical solution of this invention involves acquiring adjacent video frames of the frame to be interpolated. Branch feature extraction is performed on these adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction. Optical flow estimation is performed on the video depth features to obtain optical flow motion information. Based on the optical flow motion information and the adjacent video frames, a target intermediate frame is constructed. In scenes with fast-moving small objects, this method can better capture motion trajectories, thereby improving the accuracy of determining the target intermediate frame, thus improving the interpolation effect, enhancing the smoothness of the interpolation result, and solving the error problem when small objects are moving.

[0139] Optionally, the depth feature determination module 302 includes:

[0140] The coarse-grained feature determination submodule is used to extract coarse-grained features from the adjacent video frames to obtain video coarse-grained features;

[0141] The fine-grained feature determination submodule is used to extract fine-grained features from the adjacent video frames to obtain fine-grained video features;

[0142] The depth feature determination submodule is used to fuse the coarse-grained video features and the fine-grained video features to obtain the video depth features corresponding to the adjacent video frames. The feature extraction includes at least convolution processing, and the convolution kernel of the coarse-grained feature extraction is larger than the convolution kernel of the fine-grained feature extraction.

[0143] Optionally, the coarse-grained feature determination submodule includes:

[0144] The shallow feature determination unit is used to perform coarse-grained convolution processing on the adjacent video frames based on a preset coarse-grained convolution kernel to obtain coarse-grained shallow features.

[0145] The coarse-grained feature determination unit is used to perform spatiotemporal feature mining on the coarse-grained shallow features in the time and spatial domains to obtain coarse-grained features of the video.

[0146] Optionally, the coarse-grained feature determination unit includes:

[0147] The spatial feature determination subunit is used to perform spatial feature mining processing on the coarse-grained shallow features to obtain coarse-grained spatial features.

[0148] The temporal feature determination subunit is used to perform temporal feature mining processing on the coarse-grained shallow features to obtain coarse-grained temporal features;

[0149] Based on the coarse-grained spatial features and the coarse-grained temporal features, video coarse-grained features are constructed.

[0150] Optionally, spatial features determine sub-units, including:

[0151] Spatial feature division micro-units are used to divide the coarse-grained shallow features into channel dimensions to obtain spatial horizontal shallow features and spatial vertical shallow features.

[0152] Lateral feature determination micro-units are used to extract the shallow spatial lateral features through multi-head attention to obtain spatial lateral related features;

[0153] The longitudinal feature determination micro-unit is used to extract the spatial longitudinal shallow features through multi-head attention to obtain spatial longitudinal related features;

[0154] The spatial feature determination micro-unit is used to construct coarse-grained spatial features based on the spatial lateral correlation features and the spatial longitudinal correlation features.

[0155] Optionally, lateral features determine the micro-units, specifically for:

[0156] Based on the multi-head attention mechanism, the query vector, key vector, and value vector corresponding to the shallow horizontal features of the space are determined;

[0157] For each dimension, based on the query vector, the key vector, and the value vector, determine the horizontal branch-related features corresponding to the shallow horizontal features of the space;

[0158] All the aforementioned lateral branch-related features are spliced ​​together along the channel dimension to obtain spatial lateral correlation features.

[0159] Optionally, spatial features determine the micro-units, specifically for:

[0160] The spatial horizontal correlation features and the spatial vertical correlation features are convolved to obtain spatial convolutional correlation features;

[0161] The spatial convolutional correlation features are summed with the coarse-grained shallow features to obtain the first spatial correlation features;

[0162] Based on the multi-layer perception mechanism, the second spatial related features are obtained based on the first spatial related features;

[0163] The first spatial correlation feature and the second spatial correlation feature are summed to obtain coarse-grained spatial features.

[0164] Optionally, time-domain features determine sub-units, specifically for:

[0165] The coarse-grained shallow features are divided into channel dimensions to obtain temporal horizontal shallow features and temporal vertical shallow features.

[0166] Multi-head attention is used to extract the temporal lateral shallow features to obtain temporal lateral correlation features;

[0167] Multi-head attention is used to extract the temporal longitudinal shallow features to obtain temporal longitudinal correlation features;

[0168] Based on the temporal lateral correlation features and the temporal longitudinal correlation features, coarse-grained temporal features are constructed.

[0169] Optionally, the optical flow information determination module 303 is specifically used for:

[0170] Based on the adjacent video frames, the video depth features are separated to obtain a first feature image and a second feature image corresponding to the adjacent video frames;

[0171] Based on pixel-wise inner product, the image-related features between the first feature image and the second feature image are determined;

[0172] The image-related features are aggregated to determine the optical flow motion information between adjacent video frames.

[0173] Optionally, the target intermediate frame determination module 304 is specifically used for:

[0174] Based on the intermediate anchor frame algorithm, the intermediate anchor frame of the video is determined according to the optical flow motion information and the adjacent video frames;

[0175] Based on the intermediate anchor frames of the video, construct the anchor frame mask and residual image corresponding to the intermediate anchor frames of the video;

[0176] Based on the intermediate frame algorithm, a target intermediate frame is constructed according to the anchor frame mask, the residual image, and the optical flow motion information.

[0177] The video intermediate frame determination device provided in the embodiments of the present invention can execute the video intermediate frame determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0178] Example 4

[0179] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0180] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0181] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0182] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video intermediate frame determination methods.

[0183] In some embodiments, the video intermediate frame determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video intermediate frame determination method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video intermediate frame determination method by any other suitable means (e.g., by means of firmware).

[0184] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0185] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0186] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0188] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0189] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0190] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0191] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining intermediate frames in a video, characterized in that, include: Obtain the adjacent video frames of the frame to be interpolated; Branch feature extraction is performed on the adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction; Optical flow estimation is performed on the video depth features to obtain optical flow motion information; Based on the optical flow motion information and the adjacent video frames, a target intermediate frame is constructed; The step of extracting branch features from the adjacent video frames to obtain video depth features includes: Coarse-grained features are extracted from the adjacent video frames to obtain coarse-grained video features. Fine-grained features are extracted from the adjacent video frames to obtain fine-grained video features; The coarse-grained features and fine-grained features of the video are fused to obtain the video depth features corresponding to the adjacent video frames. The feature extraction includes at least convolution processing, and the convolution kernel of the coarse-grained feature extraction is larger than the convolution kernel of the fine-grained feature extraction. The coarse and fine granular feature extraction is performed by using coarse and fine granular convolution kernels to extract features from adjacent video frames; The step of extracting coarse-grained features from the adjacent video frames to obtain coarse-grained video features includes: Based on a preset coarse-grained convolution kernel, the adjacent video frames are subjected to coarse-grained convolution processing to obtain coarse-grained shallow features. The coarse-grained shallow features are then subjected to spatiotemporal feature mining in the temporal and spatial domains to obtain coarse-grained features of the video.

2. The method according to claim 1, characterized in that, The step of performing spatiotemporal feature mining on the coarse-grained shallow features in the temporal and spatial domains to obtain coarse-grained video features includes: The coarse-grained shallow features are subjected to spatial feature mining processing to obtain coarse-grained spatial features; The coarse-grained shallow features are subjected to temporal feature mining to obtain coarse-grained temporal features; Based on the coarse-grained spatial features and the coarse-grained temporal features, video coarse-grained features are constructed.

3. The method according to claim 2, characterized in that, The step of performing spatial feature mining processing on the coarse-grained shallow features to obtain coarse-grained spatial features includes: The coarse-grained shallow features are divided into channel dimensions to obtain spatial horizontal shallow features and spatial vertical shallow features. Multi-head attention is used to extract the shallow spatial lateral features to obtain spatial lateral correlation features. Multi-head attention is used to extract the spatial vertical shallow features to obtain spatial vertical correlation features; Based on the spatial lateral correlation features and the spatial longitudinal correlation features, coarse-grained spatial features are constructed.

4. The method according to claim 3, characterized in that, The step of extracting spatial lateral shallow features through multi-head attention to obtain spatial lateral correlation features includes: Based on the multi-head attention mechanism, the query vector, key vector, and value vector corresponding to the shallow horizontal features of the space are determined; For each dimension, based on the query vector, the key vector, and the value vector, determine the horizontal branch-related features corresponding to the shallow horizontal features of the space; All the aforementioned lateral branch-related features are spliced ​​together along the channel dimension to obtain spatial lateral correlation features.

5. The method according to claim 3, characterized in that, The step of constructing coarse-grained spatial features based on the spatial lateral correlation features and the spatial longitudinal correlation features includes: The spatial horizontal correlation features and the spatial vertical correlation features are convolved to obtain spatial convolutional correlation features; The spatial convolutional correlation features are summed with the coarse-grained shallow features to obtain the first spatial correlation features; Based on the multi-layer perception mechanism, the second spatial related features are obtained based on the first spatial related features; The first spatial correlation feature and the second spatial correlation feature are summed to obtain coarse-grained spatial features.

6. The method according to claim 2, characterized in that, The step of performing temporal feature mining processing on the coarse-grained shallow features to obtain coarse-grained temporal features includes: The coarse-grained shallow features are divided into channel dimensions to obtain temporal horizontal shallow features and temporal vertical shallow features. Multi-head attention is used to extract the temporal lateral shallow features to obtain temporal lateral correlation features; Multi-head attention is used to extract the temporal longitudinal shallow features to obtain temporal longitudinal correlation features; Based on the temporal horizontal correlation features and the temporal vertical correlation features, coarse-grained temporal features are constructed.

7. The method according to claim 1, characterized in that, The step of estimating optical flow from the video depth features to obtain optical flow motion information includes: Based on the adjacent video frames, the video depth features are separated to obtain a first feature image and a second feature image corresponding to the adjacent video frames; Based on pixel-wise inner product, the image-related features between the first feature image and the second feature image are determined; The image-related features are aggregated to determine the optical flow motion information between adjacent video frames.

8. The method according to claim 1, characterized in that, The step of constructing the target intermediate frame based on the optical flow motion information and the adjacent video frames includes: Based on the intermediate anchor frame algorithm, the intermediate anchor frame of the video is determined according to the optical flow motion information and the adjacent video frames; Based on the intermediate anchor frames of the video, construct the anchor frame mask and residual image corresponding to the intermediate anchor frames of the video; Based on the intermediate frame algorithm, a target intermediate frame is constructed according to the anchor frame mask, the residual image, and the optical flow motion information.

9. A device for determining intermediate frames in a video, characterized in that, include: The adjacent video frame acquisition module is used to acquire the adjacent video frames of the frame to be interpolated; The depth feature determination module is used to extract branch features from the adjacent video frames to obtain video depth features, wherein the branch feature extraction includes at least coarse-grained feature extraction and fine-grained feature extraction. The optical flow information determination module is used to perform optical flow estimation on the video depth features to obtain optical flow motion information; The target intermediate frame determination module is used to construct a target intermediate frame based on the optical flow motion information and the adjacent video frames; The depth feature determination module includes: The coarse-grained feature determination submodule is used to extract coarse-grained features from the adjacent video frames to obtain video coarse-grained features; The fine-grained feature determination submodule is used to extract fine-grained features from the adjacent video frames to obtain fine-grained video features; The depth feature determination submodule is used to fuse the coarse-grained video features and the fine-grained video features to obtain the video depth features corresponding to the adjacent video frames. The feature extraction includes at least convolution processing, and the convolution kernel for coarse-grained feature extraction is larger than the convolution kernel for fine-grained feature extraction. The coarse and fine-grained feature extraction is performed on the adjacent video frames using coarse and fine-grained convolution kernels. The coarse-grained feature determination submodule includes: The shallow feature determination unit is used to perform coarse-grained convolution processing on the adjacent video frames based on a preset coarse-grained convolution kernel to obtain coarse-grained shallow features. The coarse-grained feature determination unit is used to perform spatiotemporal feature mining on the coarse-grained shallow features in the time and spatial domains to obtain coarse-grained features of the video.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video intermediate frame determination method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video intermediate frame determination method according to any one of claims 1-8.

Images