Video frame insertion method, device, equipment, medium and product

Abstract

The invention relates to a video frame insertion method and device, equipment, a medium and a product. The method comprises the following steps: acquiring an initial video, wherein the initial video comprises a plurality of video frames; performing explicit motion estimation processing and implicit motion estimation processing on the plurality of video frames by using a preset frame insertion generation model to generate a plurality of target insertion frames; and performing frame insertion processing on the initial video according to the plurality of target insertion frames to obtain a target video. The method can improve the quality of the generated video.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video frame interpolation method, apparatus, device, medium, and product. Background Technology

[0002] As users increasingly demand more realistic and clear visual experiences, videos are constantly evolving towards higher resolutions and higher frame rates. However, compared to regular videos, high frame rate videos require significantly more storage and bandwidth, necessitating video frame interpolation technology.

[0003] Traditional video frame interpolation methods use optical flow networks or motion field estimation networks to obtain pixel-level motion offsets between adjacent reference frames, and generate interpolated frames by resampling adjacent reference frames at the pixel level.

[0004] However, current video frame interpolation methods suffer from insufficient video quality. Summary of the Invention

[0005] Therefore, it is necessary to provide a video frame interpolation method, apparatus, device, medium, and product that can improve the quality of generated video in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a video frame interpolation method, including:

[0007] Obtain the initial video, which includes multiple video frames;

[0008] The preset frame interpolation generation model is used to perform explicit motion estimation processing and implicit motion estimation processing on multiple video frames to generate multiple target interpolated frames.

[0009] The initial video is processed by interpolating multiple target frames to obtain the target video.

[0010] In one embodiment, the frame interpolation generation model includes an explicit estimation sub-model and an implicit estimation sub-model. The preset frame interpolation generation model is used to perform explicit motion estimation processing and implicit motion estimation processing on multiple video frames to generate multiple target interpolated frames, including:

[0011] For every two adjacent video frames, the explicit estimation sub-model is used to perform explicit estimation processing on the two adjacent video frames to generate an explicit prediction frame, and the implicit estimation sub-model is used to perform implicit estimation processing on the two adjacent video frames to generate an implicit prediction frame.

[0012] The explicit prediction frames and implicit prediction frames are fused to obtain the target interpolated frames corresponding to two adjacent video frames.

[0013] In one embodiment, an explicit estimation sub-model is used to explicitly estimate two adjacent video frames to generate an explicit prediction frame, including:

[0014] Optical flow estimation is performed on two adjacent video frames to obtain the forward optical flow and the backward optical flow;

[0015] Motion compensation processing is performed on the forward optical flow to obtain the forward offset frame, and motion compensation processing is performed on the backward optical flow to obtain the backward offset frame.

[0016] Dynamic weights are assigned to the forward and backward offset frames, and a weighted summation is performed on the forward and backward offset frames based on the dynamic weights to obtain the explicit prediction frame.

[0017] In one embodiment, an implicit estimation sub-model is used to perform implicit estimation processing on two adjacent video frames to generate an implicit prediction frame, including:

[0018] The implicit estimation sub-model is used to downsample two adjacent video frames to obtain downsampled information;

[0019] The downsampled information is processed by channel encoding to obtain multi-channel features;

[0020] The multi-channel features are subjected to channel-by-channel attention weighting to obtain motion features;

[0021] The motion features are upsampled to obtain the implicit prediction frame.

[0022] In one embodiment, the frame interpolation generation model further includes a global context transform block; fusion processing is performed on explicit predicted frames and implicit predicted frames to obtain target interpolated frames corresponding to two adjacent video frames, including:

[0023] Explicit prediction features are obtained by extracting features from explicit prediction frames, and implicit prediction features are obtained by extracting features from implicit prediction frames.

[0024] The explicit and implicit predicted features are fused using a global context transformation block to obtain fused features.

[0025] The fused features are decoded and reconstructed to obtain the target interpolated frames corresponding to two adjacent video frames.

[0026] In one embodiment, the method further includes:

[0027] Obtain sample videos;

[0028] The initial explicit sub-model and the initial implicit sub-model were trained using sample videos, respectively.

[0029] When both the initial explicit sub-model and the initial implicit sub-model have reached convergence, the initial frame interpolation model is trained using sample videos to obtain the frame interpolation generation model. The model parameters of the initial frame interpolation model are the same as those of the initial explicit sub-model and the initial implicit sub-model when they reached convergence.

[0030] Secondly, this application also provides a video frame interpolation apparatus, comprising:

[0031] The video receiving module is used to acquire the initial video, which includes multiple video frames.

[0032] The motion estimation module is used to perform explicit motion estimation and implicit motion estimation on multiple video frames using a preset frame interpolation generation model to generate multiple target interpolated frames.

[0033] The video frame interpolation module is used to perform frame interpolation on the initial video based on multiple target frames to obtain the target video.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the video frame interpolation method as described in the first aspect.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video frame interpolation method as described in the first aspect.

[0036] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the video frame interpolation method as described in the first aspect.

[0037] The aforementioned video frame interpolation methods, apparatuses, devices, media, and products utilize an interpolation generation model to perform explicit and implicit motion estimation processing on multiple video frames, generating multiple target interpolated frames. Explicit motion estimation processing enables precise pixel-level alignment of video frames in the initial video, allowing the network to reliably recover fast motion, regular displacements, and clear edge details. Implicit motion estimation processing supplements the long-range dependencies that explicit motion estimation processing struggles to capture, thereby improving the reconstruction quality of large deformation regions, nonlinear motion regions, and the motion of flexible objects. Thus, the video frame interpolation method in this application can simultaneously fully utilize the pixel-level precise motion information obtained from explicit motion offset and the semantic-level spatiotemporal variation features captured by implicit motion modeling to generate high-quality target interpolated frames, thereby improving the quality of the generated video. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating the application environment of a video frame interpolation method in one embodiment.

[0040] Figure 2 This is a flowchart illustrating a video frame interpolation method in one embodiment;

[0041] Figure 3 This is a flowchart illustrating the steps for generating multiple target interpolated frames in one embodiment.

[0042] Figure 4 This is a flowchart illustrating a video frame interpolation method in another embodiment;

[0043] Figure 5 This is a structural block diagram of a video frame interpolation device in one embodiment;

[0044] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0047] The video frame interpolation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0048] In one exemplary embodiment, such as Figure 2 As shown, a video frame interpolation method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 202 to 206. Wherein:

[0049] Step 202: Obtain the initial video, which includes multiple video frames.

[0050] Step 204: Use a preset frame interpolation generation model to perform explicit motion estimation processing and implicit motion estimation processing on multiple video frames to generate multiple target interpolated frames.

[0051] Explicit motion estimation processing refers to learning pixel-level motion information in a video frame used as a reference frame through optical flow estimation, and generating a predicted frame based on explicit motion estimation through motion compensation. Optical flow estimation can employ any existing optical flow estimation network. Motion compensation offsets the reference frame based on the optical flow estimation results and performs a weighted summation operation on the offset reference frames to obtain the predicted frame based on explicit motion estimation. Implicit motion estimation processing refers to improving the interpolation quality of complex motion scenes by learning semantic-level motion information between video frames. For example, implicit motion estimation processing can implicitly model the motion of objects in the video and generate a predicted frame based on implicit motion estimation by combining pixel recombination operations with channel attention. The target interpolated frame is the predicted frame obtained from the predicted frame based on explicit motion estimation and the predicted frame based on implicit motion estimation.

[0052] Step 206: Perform frame interpolation on the initial video based on multiple target frames to obtain the target video.

[0053] Frame interpolation refers to inserting a target frame into the vicinity of a video frame that serves as a reference frame in the initial video. For example, a target frame generated by using two adjacent video frames in the initial video as reference frames is inserted between the two adjacent video frames, ultimately resulting in a high frame rate target video with a significantly increased number of frames.

[0054] In this embodiment of the application, the number of interpolated frames to be generated between each video frame of the initial video can be calculated based on the frame rate of the initial video and the frame rate of the target video.

[0055] In the aforementioned video interpolation method, multiple video frames are subjected to explicit and implicit motion estimation processing using an interpolation generation model to generate multiple target interpolated frames. Explicit motion estimation processing enables precise pixel-level alignment of video frames in the initial video, allowing the network to reliably recover fast motion, regular displacement, and clear edge details. Implicit motion estimation processing supplements the long-range dependencies that are difficult to capture by explicit motion estimation processing, thereby improving the reconstruction quality of large deformation regions, nonlinear motion regions, and the motion of flexible objects. Thus, the video interpolation method in this embodiment can simultaneously fully utilize the pixel-level precise motion information brought by explicit motion offset and the semantic-level spatiotemporal variation features captured by implicit motion modeling to generate high-quality target interpolated frames, thereby improving the quality of the generated video.

[0056] In one exemplary embodiment, based on Figure 2 The illustrated embodiments, such as Figure 3 As shown, the frame interpolation generation model includes an explicit estimation sub-model and an implicit estimation sub-model. Using the preset frame interpolation generation model, multiple video frames undergo explicit and implicit motion estimation processing to generate multiple target interpolated frames, including:

[0057] Step 302: For every two adjacent video frames, use the explicit estimation sub-model to perform explicit estimation processing on the two adjacent video frames to generate an explicit prediction frame, and use the implicit estimation sub-model to perform implicit estimation processing on the two adjacent video frames to generate an implicit prediction frame.

[0058] In this embodiment of the application, two adjacent video frames are the reference frames for the explicit prediction frame and the implicit prediction frame.

[0059] In one possible implementation, an explicit estimation sub-model is used to perform explicit estimation processing on two adjacent video frames to generate an explicit prediction frame. This includes: performing optical flow estimation processing on the two adjacent video frames to obtain forward optical flow and backward optical flow; performing motion compensation processing on the forward optical flow to obtain a forward offset frame, and performing motion compensation processing on the backward optical flow to obtain a backward offset frame; assigning dynamic weights to the forward offset frame and the backward offset frame, and performing weighted summation processing on the forward offset frame and the backward offset frame based on the dynamic weights to obtain the explicit prediction frame.

[0060] The optical flow estimation process extracts image features from video frames used as reference frames through a multi-scale pyramid structure. At each scale level, local feature alignment is performed on the reference frames, and feature correlation is used to measure the matching degree of different regions, thereby inferring the motion direction and amplitude of that region. The optical flow estimation process employs a top-down refinement process: after predicting the overall motion trend at a coarse scale, local details are corrected layer by layer at a fine scale, resulting in forward and backward optical flows with high density and accuracy. Forward optical flow describes the pixel offset from the earlier frame to the explicitly predicted frame, while backward optical flow describes the pixel offset from the later frame to the explicitly predicted frame.

[0061] Motion compensation processing refers to a resampling process based on optical flow guidance. This involves reverse mapping of pixel positions in the reference frame according to the offsets represented by forward and backward optical flow, repositioning the pixels in the reference frame to their predicted positions. For example, bilinear interpolation can be used to ensure sufficient continuity and visual consistency between the reference frame and the explicit prediction frame. Through resampling, a forward-offset frame compensated by forward optical flow and a backward-offset frame compensated by backward optical flow are obtained. These two offset frames estimate the pixel states of the explicit prediction frame at corresponding times from different temporal directions, providing multi-view information for subsequent fusion.

[0062] In this embodiment, dynamic weights can be assigned to forward and backward offset frames by analyzing factors such as optical flow amplitude, local uncertainty, or image gradient changes in the offset region, so as to weaken the influence of regions with low optical flow estimation reliability. Then, weighted summation is performed to obtain explicit prediction frames with accurate pixel position offset and spatial continuity.

[0063] In one possible implementation, implicit estimation processing is performed on two adjacent video frames using an implicit estimation sub-model to generate an implicit prediction frame, including: downsampling the two adjacent video frames using the implicit estimation sub-model to obtain downsampled information; performing channel encoding processing on the downsampled information to obtain multi-channel features; performing channel-by-channel attention weighting processing on the multi-channel features to obtain motion features; and performing upsampling processing on the motion features to obtain the implicit prediction frame.

[0064] The implicit estimation process includes stitching two video frames used as reference frames along the channel dimension and downsampling them to proportionally reduce the spatial resolution while expanding the number of feature channels to obtain downsampled information; and using a first convolutional layer to perform channel encoding on the downsampled information to extract preliminary local texture changes.

[0065] Channel-wise attention weighting can be adaptively weighted through a semantic modeling module, progressively strengthening key features in multi-channel features while weakening irrelevant or noisy ones. For example, the semantic modeling module may include five residual channel attention groups, each consisting of 12 stacked residual attention modules. This multi-layered residual stacking structure forms a powerful implicit motion modeling structure.

[0066] In some embodiments, the implicit estimation sub-model further includes a second convolutional layer for aggregating motion features and mapping them to the target channel dimension, followed by an upsampling operation to restore the high-dimensional semantic features to a spatial resolution consistent with the video frames of the initial video by rearranging the channels, thereby obtaining the implicit prediction frame.

[0067] Step 304: The explicit prediction frame and the implicit prediction frame are fused to obtain the target interpolated frame corresponding to two adjacent video frames.

[0068] In one possible implementation, the frame interpolation generation model further includes a global context transform block. Step 304 may further include: extracting features from explicit prediction frames to obtain explicit prediction features, and extracting features from implicit prediction frames to obtain implicit prediction features; using the global context transform block to perform feature fusion processing on the explicit and implicit prediction features to obtain fused features; and performing decoding and reconstruction processing on the fused features to obtain the target interpolated frames corresponding to two adjacent video frames.

[0069] The Global Context Transformation (GCT) block is used to model the long-range dependencies between explicit and implicit predicted features. Each GCT block can include three sub-modules: a self-attention-based global interaction sub-module, used to establish long-range dependencies on a single feature map in the spatial dimension; a cross-attention-based explicit-implicit interaction sub-module, used to introduce supplementary information from implicit predicted features with explicit predicted features as the primary driver, or to fuse explicit predicted features with implicit predicted features as the primary driver; and a channel and spatial gating sub-module, used to generate adaptive fusion gates or weighted masks to control the proportion and location of information injection.

[0070] For example, predictive features can be extracted from explicit and implicit predictive frames using three stacked convolutional layers. Three cascaded global context transformation blocks are used to model the long-range dependencies between explicit and implicit predictive features. The first block primarily aligns low-level semantics and performs coarse-grained information interaction; the second further corrects channel weights and spatial correlation; and the third enhances high-level semantic consistency and compensates for details, progressively aggregating implicit predictive features into explicit predictive features to obtain fused features that integrate explicit and implicit predictive information. Then, three deconvolutional layers and four convolutional layers are used to restore the fused features to their original size and map them to the image domain to obtain the target interpolated frame. This approach balances preserving sufficient receptive field to capture local structural information with avoiding detail loss due to excessive depth.

[0071] In this embodiment, an explicit estimation sub-model is used to precisely align two adjacent video frames at the pixel level, resulting in an explicit prediction frame with clear edge details. An implicit estimation sub-model is used to compensate for the limitations of explicit motion estimation in non-rigid motion and uncertain regions, resulting in an implicit prediction frame with better texture consistency and more natural structural transitions. By fusing the explicit and implicit prediction frames, a fusion method is formed that can retain both explicit pixel-level geometric accuracy and implicit semantic details. The advantages of explicit and implicit motion estimation processing are complementary and integrated at the feature level, thereby generating high-quality target interpolated frames with strong visual consistency, effectively improving the quality of the target video.

[0072] In an exemplary embodiment, the method further includes: acquiring sample videos; training an initial explicit sub-model and an initial implicit sub-model using the sample videos respectively; and, when both the initial explicit sub-model and the initial implicit sub-model have reached convergence, training an initial frame interpolation model using the sample videos to obtain a frame interpolation generation model, wherein the model parameters of the initial frame interpolation model are the model parameters of the initial explicit sub-model and the initial implicit sub-model when they reach convergence.

[0073] The model training process can be divided into two stages. The first stage trains the initial explicit sub-model and the initial implicit sub-model separately, allowing each sub-module to independently learn its corresponding motion features. The learning rate of the initial explicit sub-model can be set to 0.0005 to quickly converge pixel-level motion information, while the learning rate of the initial implicit sub-model can be set to 0.0001 to robustly capture complex semantic-level motion and avoid training instability caused by gradient fluctuations in deep networks. Once both the initial explicit and implicit sub-models have reached convergence, the second stage begins. The parameters of the converged initial explicit and implicit sub-models are loaded into the initial frame interpolation model, and a unified learning rate of 0.0005 is used for overall training.

[0074] In this embodiment, the frame interpolation generation model is obtained through phased training. While retaining the original feature expression capabilities of each module, explicit and implicit motion information are further integrated through global optimization to ensure that the obtained frame interpolation generation model can generate high-quality, smooth and detailed target interpolated frames.

[0075] In one exemplary embodiment, such as Figure 4 As shown, a video frame interpolation method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 401 to 411. Wherein:

[0076] Step 401: Obtain the initial video, which includes multiple video frames.

[0077] Step 402: For every two adjacent video frames, perform optical flow estimation processing on the two adjacent video frames to obtain forward optical flow and backward optical flow.

[0078] Step 403: Perform motion compensation processing on the forward optical flow to obtain a forward offset frame, and perform motion compensation processing on the backward optical flow to obtain a backward offset frame.

[0079] Step 404: Assign dynamic weights to the forward offset frame and the backward offset frame, and perform weighted summation on the forward offset frame and the backward offset frame based on the dynamic weights to obtain the explicit prediction frame.

[0080] Step 405: Use the implicit estimation sub-model to downsample two adjacent video frames to obtain downsampled information.

[0081] Step 406: Perform channel encoding on the downsampled information to obtain multi-channel features, and perform channel-by-channel attention weighting on the multi-channel features to obtain motion features.

[0082] Step 407: Upsample the motion features to obtain the implicit prediction frame.

[0083] Step 408: Extract features from explicit prediction frames to obtain explicit prediction features, and extract features from implicit prediction frames to obtain implicit prediction features.

[0084] Step 409: Use the global context transformation block to perform feature fusion processing on the explicit and implicit predicted features to obtain fused features.

[0085] Step 410: Decode and reconstruct the fused features to obtain the target interpolated frames corresponding to two adjacent video frames.

[0086] Step 411: Perform frame interpolation on the initial video based on multiple target frames to obtain the target video.

[0087] In one embodiment, the method further includes: acquiring sample videos; training an initial explicit sub-model and an initial implicit sub-model using the sample videos respectively; and, when both the initial explicit sub-model and the initial implicit sub-model have reached convergence, training an initial frame interpolation model using the sample videos to obtain a frame interpolation generation model, wherein the model parameters of the initial frame interpolation model are the model parameters of the initial explicit sub-model and the initial implicit sub-model when they reach convergence.

[0088] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0089] Based on the same inventive concept, this application also provides a video frame interpolation apparatus for implementing the video frame interpolation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video frame interpolation apparatus embodiments provided below can be found in the limitations of the video frame interpolation method described above, and will not be repeated here.

[0090] In one exemplary embodiment, such as Figure 5 As shown, a video frame interpolation device is provided, including: a video receiving module 502, a motion estimation module 504, and a video frame interpolation module 506, wherein:

[0091] The video receiving module 502 is used to acquire an initial video, which includes multiple video frames;

[0092] The motion estimation module 504 is used to perform explicit motion estimation processing and implicit motion estimation processing on multiple video frames using a preset frame interpolation generation model to generate multiple target interpolated frames.

[0093] The video frame interpolation module 506 is used to perform frame interpolation processing on the initial video based on multiple target frames to obtain the target video.

[0094] In one embodiment, the frame interpolation generation model includes an explicit estimation sub-model and an implicit estimation sub-model. The motion estimation module 504 is further configured to, for every two adjacent video frames, use the explicit estimation sub-model to perform explicit estimation processing on the two adjacent video frames to generate an explicit prediction frame, and use the implicit estimation sub-model to perform implicit estimation processing on the two adjacent video frames to generate an implicit prediction frame; and perform fusion processing on the explicit prediction frame and the implicit prediction frame to obtain the target interpolated frame corresponding to the two adjacent video frames.

[0095] In one embodiment, the motion estimation module 504 is further configured to perform optical flow estimation processing on two adjacent video frames to obtain forward optical flow and backward optical flow; perform motion compensation processing on the forward optical flow to obtain a forward offset frame, and perform motion compensation processing on the backward optical flow to obtain a backward offset frame; assign dynamic weights to the forward offset frame and the backward offset frame, and perform weighted summation processing on the forward offset frame and the backward offset frame based on the dynamic weights to obtain an explicit prediction frame.

[0096] In one embodiment, the motion estimation module 504 is further configured to downsample two adjacent video frames using an implicit estimation sub-model to obtain downsampled information; perform channel encoding on the downsampled information to obtain multi-channel features; perform channel-by-channel attention weighting on the multi-channel features to obtain motion features; and perform upsampling on the motion features to obtain implicit prediction frames.

[0097] In one embodiment, the frame interpolation generation model further includes a global context transform block; the motion estimation module 504 is also used to extract features from the explicit prediction frame to obtain explicit prediction features, and to extract features from the implicit prediction frame to obtain implicit prediction features; the global context transform block is used to perform feature fusion processing on the explicit prediction features and the implicit prediction features to obtain fused features; the fused features are then decoded and reconstructed to obtain the target interpolated frames corresponding to two adjacent video frames.

[0098] In one embodiment, the apparatus further includes: a model training module for acquiring sample videos; training an initial explicit sub-model and an initial implicit sub-model using the sample videos respectively; and, when both the initial explicit sub-model and the initial implicit sub-model have reached convergence, training an initial frame interpolation model using the sample videos to obtain a frame interpolation generation model, wherein the model parameters of the initial frame interpolation model are the model parameters of the initial explicit sub-model and the initial implicit sub-model when they reach convergence.

[0099] Each module in the aforementioned video frame interpolation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0100] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores initial video and frame interpolation generation models. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a video frame interpolation method.

[0101] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0103] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0104] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0108] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video frame interpolation method, characterized in that, The method includes: Obtain an initial video, which includes multiple video frames; Multiple target interpolated frames are generated by performing explicit motion estimation and implicit motion estimation on multiple video frames using a preset frame interpolation generation model. The initial video is processed by interpolating frames based on the multiple target interpolation frames to obtain the target video.

2. The method according to claim 1, characterized in that, The frame interpolation generation model includes an explicit estimation sub-model and an implicit estimation sub-model. The step of using the preset frame interpolation generation model to perform explicit motion estimation processing and implicit motion estimation processing on multiple video frames to generate multiple target interpolated frames includes: For every two adjacent video frames, the explicit estimation sub-model is used to perform explicit estimation processing on the two adjacent video frames to generate an explicit prediction frame, and the implicit estimation sub-model is used to perform implicit estimation processing on the two adjacent video frames to generate an implicit prediction frame. The explicit prediction frame and the implicit prediction frame are fused to obtain the target interpolated frame corresponding to the two adjacent video frames.

3. The method according to claim 2, characterized in that, The step of using the explicit estimation sub-model to perform explicit estimation processing on the two adjacent video frames to generate explicit prediction frames includes: Optical flow estimation is performed on the two adjacent video frames to obtain the forward optical flow and the backward optical flow; Motion compensation processing is performed on the forward optical flow to obtain a forward offset frame, and motion compensation processing is performed on the backward optical flow to obtain a backward offset frame. Dynamic weights are assigned to the forward offset frame and the backward offset frame, and a weighted summation is performed on the forward offset frame and the backward offset frame based on the dynamic weights to obtain the explicit prediction frame.

4. The method according to claim 2, characterized in that, The step of implicitly estimating the two adjacent video frames using the implicit estimation sub-model to generate implicitly predicted frames includes: The implicit estimation sub-model is used to downsample the two adjacent video frames to obtain downsampled information; The downsampled information is processed by channel encoding to obtain multi-channel features; The multi-channel features are subjected to channel-by-channel attention weighting to obtain motion features; The motion features are upsampled to obtain the implicit prediction frame.

5. The method according to claim 2, characterized in that, The frame interpolation generation model further includes a global context transform block; the fusion processing of the explicit predicted frame and the implicit predicted frame to obtain the target interpolated frame corresponding to the two adjacent video frames includes: Explicit prediction features are obtained by extracting features from the explicit prediction frames, and implicit prediction features are obtained by extracting features from the implicit prediction frames. The explicit and implicit prediction features are fused using the global context transformation block to obtain fused features; The fusion features are decoded and reconstructed to obtain the target interpolated frames corresponding to the two adjacent video frames.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain sample videos; The initial explicit sub-model and the initial implicit sub-model were trained using the sample videos, respectively. When both the initial explicit sub-model and the initial implicit sub-model have reached the convergence condition, the initial frame interpolation model is trained using the sample video to obtain the frame interpolation generation model. The model parameters of the initial frame interpolation model are the model parameters of the initial explicit sub-model and the initial implicit sub-model when they reach the convergence condition.

7. A video frame interpolation device, characterized in that, The device includes: A video receiving module is used to acquire an initial video, which includes multiple video frames; The motion estimation module is used to perform explicit motion estimation processing and implicit motion estimation processing on multiple video frames using a preset frame interpolation generation model to generate multiple target interpolated frames. The video frame interpolation module is used to perform frame interpolation processing on the initial video based on the multiple target frames to obtain the target video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images