Video Frame Interpolation Method and Device

Abstract

The present invention discloses a video frame interpolation method and apparatus. Among them, the method includes: obtaining a first image and a second image in the video to be frame interpolated, where the first image and the second image are adjacent frames in the video to be frame interpolated; respectively inputting the first image and the second image into two symmetric feature extraction units in a feature extraction network, and predicting the feature information of a target image through the feature extraction network, where the target image is an image used for frame interpolation of the first image and the second image at a target time; predicting the target image based on the feature information. The present invention solves the technical problem in the prior art that when frame interpolating a video, the frame interpolation effect is poor due to the influence of the input image order.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular, to a video frame interpolation method and apparatus. Background Art

[0002] Currently, ultra-high definition videos have been widely used due to their smooth video picture movements and good user viewing experiences. However, there are still a large number of low-definition videos on the market, which have poor effects in terms of resolution, frame rate, color gamut, etc. The frame rate of low-definition videos is only 25-30, while the frame rate of 4K videos is 50-60. Therefore, video frame interpolation technology is required to increase the frame rate of low-definition videos. In related technologies, for the video frame interpolation algorithm adopted by the frame interpolation network based on deep learning, if the input order of the two frames of images to be interpolated is different, it will interfere with the output result of the video frame interpolation algorithm. For example, when the input order is from the first frame to the second frame, the middle frame interpolated by the algorithm is not exactly the same as the middle frame interpolated by the algorithm with the input order from the second frame to the first frame, resulting in poor frame interpolation effects.

[0003] In view of the above technical problem that the frame interpolation effect is poor due to the influence of the input image order when performing frame interpolation on a video in the prior art, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present invention provide a video frame interpolation method and apparatus to at least solve the technical problem that the frame interpolation effect is poor due to the influence of the input image order when performing frame interpolation on a video in the prior art.

[0005] According to one aspect of an embodiment of the present invention, there is provided a video frame interpolation method, including: obtaining a first image and a second image in a video to be interpolated, where the first image and the second image are adjacent frames in the video to be interpolated; respectively inputting the first image and the second image into two symmetric feature extraction units in a feature extraction network; predicting, by the feature extraction network, feature information of a target image, where the target image is an image used to perform frame interpolation on the first image and the second image at a target time; and predicting the target image based on the feature information.

[0006] According to another aspect of the embodiments of the present invention, there is also provided a video frame interpolation method, including: obtaining a first image and a second image in a video to be frame-interpolated, where the first image and the second image are adjacent frames in the video to be frame-interpolated; obtaining optical flow information through an optical flow prediction network based on the first image, the second image, and time information, where the time information is used to represent the target time at which frame interpolation needs to be performed between the first image and the second image; respectively inputting the first image and the second image into a first feature extraction unit and a second feature extraction unit to obtain a first encoding result of the first image and a second encoding result of the second image, where the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures; inputting the second encoding result and the optical flow information into the decoder of the first feature extraction unit to output first feature information, and inputting the first encoding result and the optical flow information into the decoder of the second feature extraction unit to output second feature information; predicting a target image to be inserted based on the first feature information and the second feature information.

[0007] According to another aspect of the embodiments of the present invention, there is also provided a video frame interpolation device, including: an acquisition module, configured to obtain a first image and a second image in a video to be frame-interpolated, where the first image and the second image are adjacent frames in the video to be frame-interpolated; an input module, configured to respectively input the first image and the second image into two symmetric feature extraction units in a feature extraction network; an extraction module, configured to predict feature information of a target image through the feature extraction network, where the target image is an image used for frame interpolation between the first image and the second image at a target time; a prediction module, configured to predict the target image based on the feature information.

[0008] According to another aspect of the embodiments of the present invention, there is also provided a video frame interpolation device, including: an image acquisition module, configured to obtain a first image and a second image in a video to be frame-interpolated, where the first image and the second image are adjacent frames in the video to be frame-interpolated; an optical flow acquisition module, configured to obtain optical flow information through an optical flow prediction network based on the first image, the second image, and time information, where the time information is used to represent the target time at which frame interpolation needs to be performed between the first image and the second image; an encoding module, configured to respectively input the first image and the second image into a first feature extraction unit and a second feature extraction unit to obtain a first encoding result of the first image and a second encoding result of the second image, where the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures; a decoding module, configured to input the second encoding result and the optical flow information into the decoder of the first feature extraction unit to output first feature information, and input the first encoding result and the optical flow information into the decoder of the second feature extraction unit to output second feature information; an insertion module, configured to predict a target image to be inserted based on the first feature information and the second feature information.

[0009] According to another aspect of the embodiments of the present invention, there is also provided a storage medium including a stored program, wherein when the program runs, it controls the device where the storage medium is located to execute the above video frame interpolation method.

[0010] According to another aspect of the embodiments of the present invention, there is also provided a processor for running a program, wherein when the program runs, it executes the above video frame interpolation method.

[0011] According to another aspect of the embodiments of the present invention, there is also provided a video frame interpolation method, including: receiving a video to be frame interpolated and a frame interpolation instruction; predicting a target image between any two adjacent frames in the video to be frame interpolated according to the frame interpolation instruction, wherein the two adjacent frames are respectively input into two symmetric feature extraction units in a feature extraction network, the feature information of the target image is predicted through the feature extraction network, and the target image is predicted based on the feature information; performing frame interpolation processing on the video to be frame interpolated according to the target image.

[0012] In the embodiments of the present invention, the first image and the second image are respectively input into two symmetric feature extraction units in the feature extraction network, the feature information of the target image is predicted through the feature extraction network, and the target image is predicted based on the feature information. By using completely symmetric feature extraction units, the feature extraction network is not interfered by the input order of the first image and the second image, realizing that the feature extraction network can output exactly the same feature information of the target image when the input order of the first image and the second image is different, and solving the technical problem in the related art that the frame interpolation effect is poor due to the influence of the input image order when performing frame interpolation on a video. Description of the Drawings

[0013] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation to the present invention. In the drawings:

[0014] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method according to an embodiment of the present application;

[0015] Figure 2 is a flowchart of a video frame interpolation method according to an embodiment of the present invention;

[0016] Figure 3 is a network framework diagram of an optional video frame interpolation method according to an embodiment of the present invention;

[0017] Figure 4 is a flowchart of a video frame interpolation method according to an embodiment of the present invention;

[0018] Figure 5Schematic diagram of a video frame interpolation device according to an embodiment of the present invention;

[0019] Figure 6 Schematic diagram of a video frame interpolation device according to an embodiment of the present invention;

[0020] Figure 7 Block diagram of a computer terminal according to an embodiment of the present invention;

[0021] Figure 8 Flowchart of a video frame interpolation method according to an embodiment of the present invention;

[0022] Figure 9 Schematic diagram of a video frame interpolation device according to an embodiment of the present invention. Detailed implementation manners

[0023] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] First, some nouns or terms that appear in the process of describing the embodiments of the present application are applicable to the following explanations:

[0026] Frame rate: The frequency (rate) at which bitmap images in frames appear continuously on a display.

[0027] Frame interpolation algorithm: An algorithm is used to obtain the state frame at any moment between two given frames.

[0028] Adversarial Generative Network: A deep learning network model that generates quite good outputs through the mutual game learning of (at least) two modules in the framework: the Generative Model and the Discriminative Model.

[0029] Optical flow: It is used for the motion situation of objects in time-varying images. Specifically, it is the amount of movement of the same object in a video image from one frame to the next frame, and it is a method for analyzing moving images.

[0030] PWCNet (Pyramid, Warping, Cost Volune), a neural network optical flow prediction algorithm.

[0031] Embodiment 1

[0032] According to an embodiment of the present invention, an embodiment of a video frame interpolation method is further provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0033] The method embodiment provided by the first embodiment of this application can be executed in a mobile terminal, a computing device, or a similar computing device. Figure 1 A hardware structure block diagram of a computing device (or mobile device) for implementing the video frame interpolation method is shown. As Figure 1 shown, the computing device 10 (or mobile device 10) may include one or more (shown as 102a, 102b,..., 102n in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computing device 10 may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown. The computing device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), a PAD, etc.

[0034] It should be noted that one or more of the above-mentioned processors 102 and / or other data processing circuits can generally be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computing device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video frame interpolation method in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the vulnerability detection method of the above-mentioned application program. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 can further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the computing device 10 through a network. Examples of the above network include but are not limited to the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0036] The transmission module 106 is used to receive or send data via a network. Specific examples of the above network can include the wireless network provided by the communication provider of the computing device 10. In one instance, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computing device 10 (or mobile device).

[0038] It should be noted here that in some alternative embodiments, the above-mentioned Figure 1 shown computer device (or mobile device) can include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 1This is just an example of a specific concrete instance and is intended to illustrate the types of components that may exist in the above computer device (or mobile device).

[0039] Under the above operating environment, the present application provides a video frame interpolation method as Figure 2 shown. Figure 2 It is a flowchart of the video frame interpolation method according to Embodiment 1 of the present invention. As Figure 2 shown, the video frame interpolation method includes the following steps:

[0040] Step S201, obtain a first image and a second image in the video to be frame interpolated, where the first image and the second image are adjacent frames in the video to be frame interpolated.

[0041] The above video to be frame interpolated is a low-resolution video that needs to be frame interpolated, and the first image and the second image are respectively any two adjacent frame images in the video to be frame interpolated.

[0042] In an alternative embodiment, taking the frame rate of the video to be frame interpolated as 20fps as an example, that is, the video to be frame interpolated has 20 frame images per second, then the first image and the second image can be any two adjacent frame images among the 20 frame images. For example, the first image and the second image can be the 1st frame and the 2nd frame, or the 2nd frame and the 3rd frame, or the 3rd frame and the 4th frame, etc. Each two adjacent image frames in the video to be frame interpolated can be used as the above first image and second image to execute the following steps, and the target image for frame interpolation can be obtained.

[0043] In another alternative embodiment, after one frame interpolation of the video to be frame interpolated is completed, the newly inserted image and its adjacent image can also be used as the first image and the second image. Still taking the frame rate of the video to be frame interpolated as 20fps as an example, after one frame interpolation of the video to be frame interpolated is completed, there is a newly inserted frame image between the original 1st frame and the original 2nd frame. The original first frame and the newly inserted image are two adjacent frame images after one frame interpolation, so they can be used as the above first image and second image in sequence. The newly inserted image and the second image are also two adjacent frame images, so they can also be used as the above first image and second image in sequence.

[0044] It should be noted that the above first image and second image can be two adjacent frame images in the original video to be frame interpolated, or two adjacent frame images in the video after other processing on the video to be frame interpolated. For example, color restoration processing is performed on the video to be frame interpolated, etc.

[0045] Step S202, input the first image and the second image into two symmetric feature extraction units in the feature extraction network respectively, and predict the feature information of the target image through the feature extraction network, where the target image is the image for frame interpolating the first image and the second image at the target time.

[0046] Specifically, the above-mentioned feature extraction network can be a neural network model improved based on the U-net network. The feature extraction network can predict the feature information of the target image according to the input first image and second image. The feature information can be the optical flow or RGB pixel points of the target image.

[0047] The above-mentioned target time can be any time point between the first image and the second image on the time axis of the video to be interpolated. The target image is used to represent one or more frames inserted between the first image and the second image at the target time point. For example, if the time of the first image on the time axis of the video to be interpolated is 0.01s and the time of the second image on the time axis of the video to be interpolated is 0.05s, then the target time can be any time between 0.01s and 0.05s on the time axis of the video to be interpolated. The target time can be a single time point. For example, if the target time is 0.02s, it means inserting a frame of image at 0.02s on the time axis of the video to be interpolated; the target time can also include multiple time points. For example, if the target time is 0.02s and 0.03s, it means inserting a frame of image at 0.02s and 0.03s respectively on the time axis of the video to be interpolated, that is, inserting two new frames of images between the first image and the second image at the same time.

[0048] Figure 3 is a network framework diagram of a video interpolation method according to Embodiment 1 of the present application. In an optional embodiment, as Figure 3 shown, the feature extraction network includes two symmetric feature extraction units (the first feature extraction unit 311 and the second feature extraction unit 321). The two symmetric feature extraction units can be symmetric networks improved based on the U-net network. The structures of the first feature extraction unit 311 and the second feature extraction unit 321 of the two U-net networks are symmetric, and can include encoders and decoders with the same structure in the first feature extraction unit 311 and the second feature extraction unit 321. The number of downsampling layers in the encoders of the first feature extraction unit 311 and the second feature extraction unit 321 is equal, and the number of upsampling layers in the decoders of the first feature extraction unit 311 and the second feature extraction unit 321 is equal. For the traditional U-net network, it includes an encoder (i.e., the downsampling layer) for feature information extraction and a decoder (i.e., the upsampling layer) for fusing and outputting the extracted feature information, and the input of its decoder comes from its encoder. In this embodiment, the output of the encoder of the first feature extraction unit 311 is connected to the input of the decoder of the second feature extraction unit 321, and the output of the encoder of the second feature extraction unit 321 is connected to the input of the decoder of the first feature extraction unit 311, forming an improved symmetric network with the basic structure of the U-net, but the inputs and outputs of the two U-net networks are staggered.

[0049] When performing video frame interpolation, the first image I0 is input to the encoder of the first feature extraction unit 311 for multi-layer encoding, and the second image I1 is input to the encoder of the second feature extraction unit 321 for multi-layer encoding. The output of the encoder of the first feature extraction unit 311 is connected to the input of the decoder of the second feature extraction unit 321, and the output of the encoder of the second feature extraction unit 321 is connected to the input of the decoder of the first feature extraction unit 311. The outputs of the decoders of the first feature extraction unit 311 and the second feature extraction unit 321 are cascaded and then subjected to full convolution to output the feature information of the target image. Since the first feature extraction unit 311 and the second feature extraction unit 321 have a symmetric structure, the feature information of the target image is not affected by the input order of the first image and the second image. Furthermore, no matter in what order the first image and the second image are input to the feature extraction network, the feature information of the exactly same target image can be output.

[0050] Based on the target image obtained from the above feature extraction network, on the time axis of the video to be frame interpolated, it can be inserted at the time point of the target time in the order from the first image to the second image, or in the order from the second image to the first image. The frame interpolation of the target image is not affected by the order of the first image and the second image. For example, on the time axis of the video to be frame interpolated, the time point of the first image is 0.01 s, the time point of the second image is 0.05 s, and the target time is 0.02 s. Then the target image can be frame interpolated by adding 0.01 s starting from the time point of the first image, or by subtracting 0.03 s starting from the time point of the second image. The target images obtained by the two frame interpolation methods are exactly the same.

[0051] Step S203: Predict the target image based on the feature information.

[0052] After obtaining the target image, insert the target image between the first image and the second image according to the target time, and then one-time frame interpolation between the first image and the second image can be completed.

[0053] In this embodiment, the first image and the second image in the video to be interpolated are obtained, where the first image and the second image are adjacent frames in the video to be interpolated; the first image and the second image are respectively input into two symmetric feature extraction units in the feature extraction network; the feature extraction network predicts the feature information of the target image, where the target image is the image obtained by interpolating the first image and the second image at the target time; the target image is predicted based on the feature information. By using completely symmetric feature extraction units, the feature extraction network is not interfered by the input order of the first image and the second image, and the feature extraction network can output exactly the same feature information of the target image when the input orders of the first image and the second image are different, solving the technical problem that the interpolation effect is poor due to the influence of the input image order in the related art when interpolating a video.

[0054] As an optional embodiment, the feature extraction network includes: a first feature extraction unit and a second feature extraction unit, and the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures.

[0055] The first feature extraction unit and the second feature extraction unit can be symmetric networks improved based on the Unet network. The above encoder includes a downsampling layer, and the decoder includes an upsampling layer. Specifically, the number of downsampling layers of the encoder of the first feature extraction unit is equal to the number of downsampling layers of the encoder of the second feature extraction unit, and the number of upsampling layers of the decoder of the first feature extraction unit is equal to the number of upsampling layers of the decoder of the second feature extraction unit. For example, as Figure 3 shown, the encoders of the first feature extraction unit 311 and the second feature extraction unit 321 both use 3*3 downsampling layers (including 4 convolutional layers, that is, Figure 3 the first 4 rectangular modules at the start of the arrow lines in the first feature extraction unit 311 and the second feature extraction unit 321), and the decoders both use 3*3 upsampling layers (including 4 transposed convolutional layers, that is, Figure 3 the 4 rectangular modules pointed to by the arrow lines in the first feature extraction unit 311 and the second feature extraction unit 321).

[0056] As an alternative embodiment, the feature information of the target image includes: the first feature information output by the first feature extraction unit and the second feature information output by the second feature extraction unit. The first image and the second image are respectively input into two symmetric feature extraction units in the feature extraction network, and the feature information of the target image is predicted through the feature extraction network, including: the first image, the second image, and the time information are sequentially input into the optical flow prediction network to obtain the first optical flow information output by the optical flow prediction network, where the time information is used to represent the target time for which frame interpolation is required between the first image and the second image; the second image, the first image, and the time information are sequentially input into the optical flow prediction network to obtain the second optical flow information output by the optical flow prediction network; the encoding result of the second encoder of the second feature extraction unit and the first optical flow information are input into the first decoder in the first feature extraction unit to generate the first feature information; the encoding result of the first encoder of the first feature extraction unit and the second optical flow information are input into the first decoder in the second feature extraction unit to generate the second feature information.

[0057] Specifically, the output of the encoder of the first feature extraction unit is connected to the input of the decoder of the second feature extraction unit, and the output of the encoder of the second feature extraction unit is connected to the input of the decoder of the first feature extraction unit, so that the short connection feature layer of the first feature extraction unit comes from the second feature extraction unit. Correspondingly, the short connection feature layer of the second feature extraction unit comes from the first feature extraction unit.

[0058] Specifically, the optical flow prediction network is used to estimate the optical flow information of pixel points in a video image, that is, the motion of each pixel point in the video image over time. Therefore, it can predict the positions of each pixel point in the input image in the adjacent image at the specified time according to the input image and the specified time. In the above embodiment, the first image I0, the second image I1, and the target time are input into the optical flow prediction network. According to the first image I0 and the second image I1, the motion of the pixel points from the time point of the first image I0 to the time point of the second image I1 can be estimated (i.e., the first optical flow information). According to the first optical flow information, the positions of the pixel points at the target time point between the first image I0 and the second image I1 on the target image can be predicted, and then the target image can be determined. Correspondingly, the second image I1, the first image I0, and the target time are input into the optical flow prediction network. According to the second image I1 and the first image I0, the motion of the pixel points from the time point of the second image I1 to the time point of the first image I0 can be estimated (i.e., the second optical flow information). According to the second optical flow information, the positions of the pixel points at the target time point between the first image I0 and the second image I1 on the target image can be predicted, and then the target image can be determined.

[0059] It should be noted that the optical flow prediction network for generating the first optical flow information and the optical flow prediction network for generating the second optical flow information can be the same optical flow prediction network or two optical flow prediction networks with exactly the same structure. The difference between the first optical flow information and the second optical flow information lies in that the first image and the second image are input into the optical flow prediction network in different input orders to obtain different optical flow information outputs. Specifically, inputting the optical flow prediction network in the order of the first image, the second image, and the time information can obtain the first optical flow information, and inputting the optical flow prediction network in the order of the second image, the first image, and the time information can obtain the second optical flow information. In an alternative embodiment, the optical flow prediction network can be PWCNet.

[0060] As Figure 3 shown, the first feature extraction unit 311 includes a first encoder (including 4 convolutional layers, that is, Figure 3 the 4 rectangular modules at the start of the arrow line in the first feature extraction unit 311 of Figure 3 ) and a first decoder (including 4 deconvolutional layers, that is, Figure 3 the 4 rectangular modules pointed to by the arrow line in the first feature extraction unit 311 of Figure 3 ). The second feature extraction unit 321 includes a second encoder (including 4 convolutional layers, that is, Figure 3 the 4 rectangular modules at the start of the arrow line in the second feature extraction unit 321 of Figure 3 ) and a second decoder (including 4 deconvolutional layers, that is, Figure 3 the 4 rectangular modules pointed to by the arrow line in the second feature extraction unit 321 of Figure 3 ). The first image I0 is input into the first encoder, and the second image I1 is input into the second encoder. The first image I0, the second image I1, and the first time information t (that is, the above-mentioned time information, specifically the time for frame interpolation on the time axis of the video to be interpolated frames in the order of the first image and the second image) are input into the optical flow prediction network 31 in sequence, and the second image I1, the first image I0, and the second time information 1 - t (that is, the above-mentioned time information, specifically the time for frame interpolation on the time axis of the video to be interpolated frames in the order of the second image and the first image) are input into the optical flow prediction network 32 in sequence. The output of the first encoder is connected to the input of the second decoder (that is, Figure 3 the arrow connection line of the second feature extraction unit 321 pointed to by the first feature extraction unit 311 in Figure 3 ), and the output of the second encoder is connected to the input of the first decoder (that is, Figure 3(The arrow connection line of the first feature extraction unit 311 pointed to by the second feature extraction unit 321 in FIG. Since the first feature extraction unit 311 and the second feature extraction unit 321 have a symmetric structure, both the first encoder 311 and the second encoder 321 include a 3×3 upsampling layer, and both the first decoder 312 and the second decoder 322 include a 3×3 downsampling layer, enabling the input and output of the first feature extraction unit 311 and the second feature extraction unit 321 to be cross-input in a one-to-one correspondence. In addition, the input of the first decoder is also connected to the output of the optical flow prediction network 31, and the input of the second decoder is also connected to the output of the optical flow prediction network 32, such that the first feature information output by the first feature extraction unit 311 comes from both the encoder of the second feature extraction unit 321 and the optical flow prediction network 31 with the first image I0, the second image I1, and the first time information t as inputs, and the second feature information output by the second feature extraction unit 321 comes from both the encoder of the first feature extraction unit 311 and the optical flow prediction network 32 with the second image I1, the first image I0, and the second time information 1 - t as inputs.)

[0061] In this embodiment, by exchanging the inputs and outputs of the encoders and decoders of the first feature extraction unit and the second feature extraction unit, compared with the related art where the input of the decoder of the feature extraction unit comes from the output of its own encoder, this embodiment realizes changing the input order of the first image and the second image without changing the final prediction result output by the feature extraction network, solving the technical problem in the related art that the frame interpolation effect of a video is affected by the input image order, resulting in a poor frame interpolation effect.

[0062] Furthermore, in this embodiment, different optical flow information obtained by inputting the first image and the second image in different orders in each feature extraction unit is used to participate in the training process, realizing changing the input order of the first image and the second image in the optical flow prediction network without changing the final prediction result output by the feature extraction network.

[0063] As an alternative embodiment, inputting the first image, the second image, and the time information into the optical flow prediction network in sequence to obtain the first optical flow information output by the optical flow prediction network includes: obtaining the first time information of the target time relative to the first image; inputting the first image, the second image, and the first time information into the optical flow prediction network in sequence to obtain the first optical flow information output by the optical flow prediction network; inputting the second image, the first image, and the time information into the optical flow prediction network in sequence to obtain the second optical flow information output by the optical flow prediction network, including: obtaining the second time information of the target time relative to the second image; inputting the second image, the first image, and the time information into the optical flow prediction network in sequence to obtain the second optical flow information output by the optical flow prediction network.

[0064] Since the target image is any image inserted between the first image and the second image, the first time information is understood as the time point of the target image on the time axis of the video with frames to be inserted in the order from the time point of the first image to the time point of the second image, and the second time information is understood as the time point of the target image on the time axis of the video with frames to be inserted in the order from the time point of the second image to the time point of the first image (opposite to the above order from the time point of the first image to the time point of the second image).

[0065] On the time axis of the video with frames to be inserted, the target image interpolated from the time point of the first image to the first time information should be exactly the same as the target image interpolated in the reverse direction from the time point of the second image to the second time information. Therefore, the first image and the second image are input into the optical flow prediction network in different input orders, and the outputs of the optical flow prediction network under the two input orders are respectively input into the decoders of the first feature extraction unit and the second feature extraction unit, so that the feature information of the target image finally output by the feature extraction network is not interfered by the input order of the first image and the second image, realizing that the feature extraction network (i.e., the frame interpolation algorithm) can output exactly the same feature information of the target image when the input orders of the first image and the second image are different.

[0066] As an optional embodiment, obtaining the first time information of the target time relative to the first image includes: obtaining the time difference between the target time and the first image, and normalizing the time difference to obtain the first time information; obtaining the second time information of the target time relative to the second image includes: determining that the difference between 1 and the first time information is the second time information.

[0067] It should be noted that the first time information and the second time information after normalizing the time difference are not the time of the target image in the video with frames to be inserted, but the relative time with reference to the time difference between the first image and the second image. For example, if the time difference between the first image and the second image is 1, and the target image is in the time order from the first image to the second image, the first time information is determined as t, and in the time order from the second image to the first image, the second time information is determined as 1 - t.

[0068] As an optional embodiment, predicting the target image based on the feature information includes: obtaining the first feature information output by the first feature extraction unit and the second feature information output by the second feature extraction unit in a row concatenation to obtain the concatenated feature information; inputting the concatenated feature information into a fully convolutional layer to obtain the target image predicted by the fully convolutional layer.

[0069] As Figure 3As shown, the first feature information 313 output by the first feature extraction unit 311 and the second feature information 323 output by the second feature extraction unit 321 are concatenated and then input into the fully convolutional layer 33, and the fully convolutional layer 33 finally outputs the feature information of the target image It.

[0070] As an optional embodiment, the above method further includes: training the video interpolation model with sample data, where the video interpolation model at least includes a feature extraction network and a fully convolutional layer. The steps of training the video interpolation model with sample data include: inputting the sample data into the video interpolation model to obtain a prediction result output by the video interpolation model; inputting the prediction result into a preset discriminator to obtain a discrimination result output by the discriminator, where the discrimination result is used to indicate whether the prediction result is accurate; and adjusting the network parameters of the video interpolation model according to the discrimination result.

[0071] As Figure 3 As shown, the video interpolation model 30 includes a feature extraction network (including a first feature extraction unit 311 and a second feature extraction unit 321) and a fully convolutional layer 33. The feature information of the target image It output by the video interpolation model 30 is input into the discriminator 34. The discriminator 34 can be a pre-trained neural network model, which is used to discriminate whether the target image predicted by the feature extraction network is accurate, so as to obtain the loss function of this training according to the discrimination result, and then adjust the model parameters of the video interpolation model based on the loss functions of multiple trainings.

[0072] The above video interpolation model and the discriminator can form an adversarial generation network. The video interpolation model is the generation model in the adversarial generation network, and the discriminator is the discriminant model in the adversarial generation network, and they are trained through their confrontation.

[0073] In this embodiment, by adding a discriminator to discriminate the authenticity of the predicted image, compared with the frame interpolation algorithm guided by PSNR (Peak Signal to Noise Ratio) in the related art, the target image for frame interpolation generated based on the video interpolation model has more details, can effectively improve the perceptual quality of the image, and makes the finally output image more in line with human perception.

[0074] It should be noted that the discriminator is only used in the training process of the video interpolation model with sample data and does not participate in the actual frame interpolation process in the actual video interpolation application.

[0075] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0076] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0077] Embodiment 2

[0078] According to an embodiment of the present invention, an embodiment of a video frame interpolation method is further provided. Figure 4 For a video frame interpolation method according to Embodiment 2 of the present invention, as Figure 4 shown, the method includes the following steps:

[0079] Step S401, obtain a first image and a second image in the video to be frame interpolated, where the first image and the second image are adjacent frames in the video to be frame interpolated.

[0080] The above video to be frame interpolated is a low-definition video that needs to be frame interpolated, and the first image and the second image are respectively any two adjacent frame images in the video to be frame interpolated.

[0081] In an optional embodiment, taking the frame rate of the video to be frame interpolated as 20fps as an example, that is, the video to be frame interpolated has 20 frame images per second, then the first image and the second image can be any two adjacent frame images among the 20 frame images. For example, the first image and the second image can be the first frame and the second frame, or the second frame and the third frame, or the third frame and the fourth frame, etc. Each two adjacent image frames in the video to be frame interpolated can be used as the above first image and second image to execute the following steps, and the target image for frame interpolation can be obtained.

[0082] In another alternative embodiment, after one frame interpolation of the video to be interpolated is completed, the newly inserted frame image and its adjacent images can also be used as the first image and the second image. Still taking the frame rate of the video to be interpolated as 20fps as an example, after one frame interpolation of the video to be interpolated is completed, there is a newly inserted frame image between the original first frame and the original second frame. The original first frame and the newly inserted image are two adjacent frames after one frame interpolation. Therefore, they can be used as the above-mentioned first image and second image in sequence. The newly inserted image and the second image are also two adjacent frames. Therefore, they can also be used as the above-mentioned first image and second image in sequence.

[0083] It should be noted that the above-mentioned first image and second image can be two adjacent frames in the original video to be interpolated, or two adjacent frames in the video after other processing is performed on the video to be interpolated. For example, color restoration processing is performed on the video to be interpolated, etc.

[0084] Step S402, obtain optical flow information through an optical flow prediction network based on the first image, the second image, and time information, where the time information is used to represent the target time at which frame interpolation needs to be performed between the first image and the second image.

[0085] The above-mentioned target time can be any time point between the first image and the second image on the time axis of the video to be interpolated. The target image is used to represent one or more frames inserted between the first image and the second image at the target time point. For example, the time of the first image on the time axis of the video to be interpolated is 0.01s, and the time of the second image on the time axis of the video to be interpolated is 0.05s. Then, the target time can be any time between 0.01s and 0.05s on the time axis of the video to be interpolated. The target time can be a time point. For example, the target time is 0.02s, indicating that one frame image is inserted at 0.02s on the time axis of the video to be interpolated; the target time can also include multiple time points. For example, the target time is 0.02s and 0.03s, indicating that one frame image is inserted at 0.02s and 0.03s on the time axis of the video to be interpolated respectively, that is, two new frames are inserted between the first image and the second image at the same time.

[0086] In an alternative embodiment, the optical flow prediction network obtains optical flow information based on the first image, the second image, and time information, including: sequentially inputting the first image, the second image, and the first time information into the optical flow prediction network to obtain the first optical flow information output by the optical flow prediction network, where the first time information is the time point of the target image on the time axis of the video to be interpolated in the order from the time point of the first image to the time point of the second image; sequentially inputting the second image, the first image, and the time information into the optical flow prediction network to obtain the second optical flow information output by the optical flow prediction network, where the time point of the target image on the time axis of the video to be interpolated is in the order from the time point of the second image to the time point of the first image (opposite to the order from the time point of the first image to the time point of the second image).

[0087] Specifically, the optical flow prediction network is used to estimate the optical flow information of pixel points in video images, that is, the motion of each pixel point in the video image over time. Therefore, it can predict the positions of each pixel point in the input image in the adjacent image at the specified time according to the input image and the specified time. In the above embodiment, as Figure 3 shown, input the first image I0, the second image I1, and the target time into the optical flow prediction network. According to the first image I0 and the second image I1, the motion of the pixel points from the time point of the first image I0 to the time point of the second image I1 can be estimated (i.e., the first optical flow information). According to the first optical flow information, the positions of the pixel points corresponding to the target time point between the first image I0 and the second image I1 on the target image can be predicted, and then the target image can be determined. Correspondingly, input the second image I1, the first image I0, and the target time into the optical flow prediction network. According to the second image I1 and the first image I0, the motion of the pixel points from the time point of the second image I1 to the time point of the first image I0 can be estimated (i.e., the second optical flow information). According to the second optical flow information, the positions of the pixel points corresponding to the target time point between the first image I0 and the second image I1 on the target image can be predicted, and then the target image can be determined.

[0088] Step S403: Input the first image and the second image into the first feature extraction unit and the second feature extraction unit respectively to obtain the first encoding result of the first image and the second encoding result of the second image, where the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures.

[0089] Specifically, the first feature extraction unit and the second feature extraction unit may be symmetric networks improved based on the Unet network. The above encoder includes a downsampling layer, and the decoder includes an upsampling layer. The number of downsampling layers of the encoder of the first feature extraction unit is equal to the number of downsampling layers of the encoder of the second feature extraction unit, and the number of upsampling layers of the decoder of the first feature extraction unit is equal to the number of upsampling layers of the decoder of the second feature extraction unit. The output of the encoder of the first feature extraction unit is connected to the input of the decoder of the second feature extraction unit, and the output of the encoder of the second feature extraction unit is connected to the input of the decoder of the first feature extraction unit, so that the short connection feature layer of the first feature extraction unit comes from the second feature extraction unit. Correspondingly, the short connection feature layer of the second feature extraction unit comes from the first feature extraction unit.

[0090] For example, as Figure 3 shown, the encoders of the first feature extraction unit 311 and the second feature extraction unit 321 both adopt 3*3 downsampling layers (including 4 convolutional layers, that is, Figure 3 the first 4 rectangular modules at the starting points of the arrow lines in the first feature extraction unit 311 and the second feature extraction unit 321), and the decoders both adopt 3*3 upsampling layers (including 4 transposed convolutional layers, that is, Figure 3 the 4 rectangular modules pointed by the arrow lines in the first feature extraction unit 311 and the second feature extraction unit 321).

[0091] Step S404: Input the second coding result and the optical flow information into the decoder of the first feature extraction unit to output the first feature information, and input the first coding result and the optical flow information into the decoder of the second feature extraction unit to output the second feature information.

[0092] The above optical flow information may include the first optical flow information obtained by sequentially inputting the first image, the second image, and the above first time information into the optical flow prediction network, and the second optical flow information output by the optical flow prediction network obtained by sequentially inputting the second image, the first image, and the first time information into the optical flow prediction network. It should be noted that the optical flow prediction network for generating the first optical flow information and the optical flow prediction network for generating the second optical flow information may be the same optical flow prediction network, or two optical flow prediction networks with exactly the same structure. The difference between the first optical flow information and the second optical flow information is that the first image and the second image are input into the optical flow prediction network in different input orders to obtain different optical flow information outputs.

[0093] In an alternative embodiment, as Figure 3As shown in the figure, the input of the decoder of the first feature extraction unit 311 comes from the second encoding result output by the encoder of the second feature extraction unit 321 and the optical flow prediction network 31. Specifically, the output of the optical flow prediction network 31 is the first optical flow information obtained by sequentially inputting the first image, the second image, and the above-mentioned first time information into the optical flow prediction network 31. The input of the decoder of the second feature extraction unit 321 comes from the first encoding result output by the encoder of the first feature extraction unit 311 and the optical flow prediction network 32. The output of the optical flow prediction network 32 is the second optical flow information output by the optical flow prediction network obtained by sequentially inputting the second image, the first image, and the first time information into the optical flow prediction network.

[0094] Step S405: Predict the target image to be inserted based on the first feature information and the second feature information.

[0095] Specifically, the first feature information and the second feature information can be concatenated and then input into a fully convolutional layer to obtain the feature information of the target image. As Figure 3 shown in the figure, the first feature information 313 output by the first feature extraction unit 311 and the second feature information 323 output by the second feature extraction unit 321 are concatenated and then input into the fully convolutional layer 33. The fully convolutional layer 33 finally outputs the feature information of the target image It.

[0096] As an optional embodiment, after predicting the target image to be inserted based on the first feature information and the second feature information, the above method further includes: inserting the target image between the first image and the second image.

[0097] After obtaining the target image, the target image is inserted between the first image and the second image according to the target time, and the frame interpolation between the first image and the second image is completed.

[0098] In this embodiment, by cross - inputting and outputting the encoders and decoders of the first feature extraction unit and the second feature extraction unit, and inputting the first image and the second image into the optical flow prediction network in different orders to obtain the first optical flow information and the second optical flow information, and respectively inputting them into the decoder inputs of the first feature extraction unit and the second feature extraction unit. Compared with the related technology where the input of the decoder of the feature extraction unit comes from the output of its own encoder, this embodiment realizes changing the order of the first image and the second image in the optical flow prediction network without changing the final prediction result output by the feature extraction network, and solves the technical problem that the frame interpolation effect is poor due to the influence of the input image order when performing frame interpolation on a video in the related technology.

[0099] Embodiment 3

[0100] According to an embodiment of the present invention, there is also provided a device for implementing the above video frame interpolation method. Figure 5A video frame interpolation device according to Embodiment 3 of the present invention, as Figure 5 shown, the device 500 includes:

[0101] An acquisition module 51, configured to acquire a first image and a second image in the video to be frame interpolated, where the first image and the second image are adjacent frames in the video to be frame interpolated; an input module 52, configured to respectively input the first image and the second image into two symmetric feature extraction units in the feature extraction network, and predict the feature information of the target image through the feature extraction network, where the target image is an image for frame interpolating the first image and the second image at a target time; a prediction module 53, configured to predict the target image based on the feature information.

[0102] It should be noted here that the above acquisition module 51, input module 52, and prediction module 53 correspond to steps S201 to S203 in Embodiment 1. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computing device 10 provided in Embodiment 1.

[0103] As an optional embodiment, the feature extraction network includes: a first feature extraction unit and a second feature extraction unit, and the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures.

[0104] As an optional embodiment, the feature information of the target image includes: first feature information output by the first feature extraction unit and second feature information output by the second feature extraction unit. The above input module includes: a first input sub-module, configured to sequentially input the first image, the second image, and time information into the optical flow prediction network to obtain first optical flow information output by the optical flow prediction network, where the time information is used to represent the target time at which frame interpolation is required between the first image and the second image; a second input sub-module, configured to sequentially input the second image, the first image, and time information into the optical flow prediction network to obtain second optical flow information output by the optical flow prediction network; a third input sub-module, configured to input the encoding result of the second encoder of the second feature extraction unit and the first optical flow information into the first decoder in the first feature extraction unit to generate first feature information; a fourth input sub-module, configured to input the encoding result of the first encoder of the first feature extraction unit and the second optical flow information into the first decoder in the second feature extraction unit to generate second feature information.

[0105] As an alternative embodiment, the first input sub-module includes: a first acquisition sub-module, configured to acquire first time information of a target time relative to a first image; a first optical flow information output sub-module, configured to sequentially input the first image, a second image, and the first time information into an optical flow prediction network to obtain first optical flow information output by the optical flow prediction network; the second input sub-module includes: a second acquisition sub-module, configured to acquire second time information of the target time relative to the second image; a first optical flow information output sub-module, configured to sequentially input the second image, the first image, and the time information into the optical flow prediction network to obtain second optical flow information output by the optical flow prediction network.

[0106] As an alternative embodiment, the first acquisition sub-module includes: a normalization sub-module, configured to acquire a time difference between the target time and the first image, and perform normalization processing on the time difference to obtain the first time information; the second acquisition sub-module includes: a difference determination sub-module, configured to determine a difference between 1 and the first time information as the second time information.

[0107] As an alternative embodiment, the above prediction module includes: a concatenation sub-module, configured to perform row concatenation on first feature information output by a first feature extraction unit and second feature information output by a second feature extraction unit to obtain concatenated feature information; a fully convolutional sub-module, configured to input the concatenated feature information into a fully convolutional layer to obtain a target image predicted by the fully convolutional layer.

[0108] As an alternative embodiment, the above device further includes: a training module, configured to train a video interpolation model by using sample data, where the video interpolation model at least includes a feature extraction network and a fully convolutional layer; the above training module includes: a prediction sub-module, configured to input the sample data into the video interpolation model to obtain a prediction result output by the video interpolation model; a discrimination sub-module, configured to input the prediction result into a preset discriminator to obtain a discrimination result output by the discriminator, where the discrimination result is used to indicate whether the prediction result is accurate; an adjustment sub-module, configured to adjust network parameters of the video interpolation model according to the discrimination result.

[0109] It should be noted that the optional or preferred implementation manners of this embodiment can refer to the relevant descriptions in Embodiments 1 to 2, which will not be elaborated here.

[0110] Embodiment 4

[0111] According to an embodiment of the present invention, there is also provided a device for implementing the above video interpolation method. Figure 6 A video interpolation device according to Embodiment 4 of the present invention, as Figure 6 shown, the device 600 includes:

[0112] An image acquisition module 61, configured to acquire a first image and a second image in the video to be interpolated, where the first image and the second image are adjacent frames in the video to be interpolated; an optical flow acquisition module 62, configured to obtain optical flow information based on the first image, the second image, and time information through an optical flow prediction network, where the time information is used to represent the target time at which interpolation is required between the first image and the second image; an encoding module 63, configured to input the first image and the second image into a first feature extraction unit and a second feature extraction unit respectively, to obtain a first encoding result of the first image and a second encoding result of the second image, where the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures; a decoding module 64, configured to input the second encoding result and the optical flow information into the decoder of the first feature extraction unit, output first feature information, and input the first encoding result and the optical flow information into the decoder of the second feature extraction unit, output second feature information; an insertion module 65, configured to predict a target image to be inserted based on the first feature information and the second feature information.

[0113] It should be noted here that the above-mentioned image acquisition module 61, optical flow acquisition module 62, encoding module 63, decoding module 64, and insertion module 65 correspond to steps S401 to S405 in Embodiment 2. The functions of the five modules and the corresponding steps are the same in terms of implementation examples and application scenarios, but are not limited to the content disclosed in the above-mentioned Embodiment 2. It should be noted that the above modules, as part of the device, can run in the computing device 10 provided in Embodiment 1.

[0114] As an optional embodiment, the above device further includes: an insertion module, configured to insert the target image between the first image and the second image.

[0115] It should be noted that the optional or preferred implementation manners of this embodiment can refer to the relevant descriptions in Embodiments 1 to 2, and will not be elaborated here.

[0116] Embodiment 5

[0117] An embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the above storage medium can be used to store the program code executed by the video interpolation method provided in the above-mentioned Embodiment 1.

[0118] Optionally, in this embodiment, the above storage medium can be located in any one of the computing devices in the computing device group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0119] Optionally, the storage medium is configured to store program code for performing the following steps: obtaining a first image and a second image in the video to be interpolated, where the first image and the second image are adjacent frames in the video to be interpolated; respectively inputting the first image and the second image into two symmetric feature extraction units in the feature extraction network, and predicting the feature information of the target image through the feature extraction network, where the target image is an image obtained by interpolating the first image and the second image at the target time; predicting the target image based on the feature information.

[0120] Optionally, the storage medium is configured to store program code for performing the following steps: the feature extraction network includes: a first feature extraction unit and a second feature extraction unit, and the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures.

[0121] Optionally, the storage medium is configured to store program code for performing the following steps: the feature information of the target image includes: first feature information output by the first feature extraction unit and second feature information output by the second feature extraction unit. Respectively inputting the first image and the second image into two symmetric feature extraction units in the feature extraction network, and predicting the feature information of the target image through the feature extraction network includes: sequentially inputting the first image, the second image, and time information into the optical flow prediction network to obtain first optical flow information output by the optical flow prediction network, where the time information is used to represent the target time at which interpolation is required between the first image and the second image; sequentially inputting the second image, the first image, and time information into the optical flow prediction network to obtain second optical flow information output by the optical flow prediction network; inputting the encoding result of the second encoder of the second feature extraction unit and the first optical flow information into the first decoder in the first feature extraction unit to generate first feature information; inputting the encoding result of the first encoder of the first feature extraction unit and the second optical flow information into the first decoder in the second feature extraction unit to generate second feature information.

[0122] Optionally, the storage medium is configured to store program code for performing the following steps: sequentially inputting the first image, the second image, and time information into the optical flow prediction network to obtain first optical flow information output by the optical flow prediction network includes: obtaining first time information of the target time relative to the first image; sequentially inputting the first image, the second image, and the first time information into the optical flow prediction network to obtain first optical flow information output by the optical flow prediction network; sequentially inputting the second image, the first image, and time information into the optical flow prediction network to obtain second optical flow information output by the optical flow prediction network includes: obtaining second time information of the target time relative to the second image; sequentially inputting the second image, the first image, and time information into the optical flow prediction network to obtain second optical flow information output by the optical flow prediction network.

[0123] Optionally, the storage medium is configured to store program code for performing the following steps: obtaining first time information of a target time relative to a first image, including: obtaining a time difference between the target time and the first image, and normalizing the time difference to obtain the first time information; obtaining second time information of the target time relative to a second image, including: determining a difference between 1 and the first time information as the second time information.

[0124] Optionally, the storage medium is configured to store program code for performing the following steps: predicting a target image based on feature information, including: obtaining concatenated feature information by concatenating first feature information output by a first feature extraction unit and second feature information output by a second feature extraction unit; inputting the concatenated feature information into a fully convolutional layer to obtain the target image predicted by the fully convolutional layer.

[0125] Optionally, the storage medium is configured to store program code for performing the following steps: the above method further includes: training a video frame interpolation model with sample data, where the video frame interpolation model at least includes a feature extraction network and a fully convolutional layer, and the steps of training the video frame interpolation model with sample data include: inputting the sample data into the video frame interpolation model to obtain a prediction result output by the video frame interpolation model; inputting the prediction result into a preset discriminator to obtain a discrimination result output by the discriminator, where the discrimination result is used to indicate whether the prediction result is accurate; adjusting network parameters of the video frame interpolation model according to the discrimination result.

[0126] Embodiment 6

[0127] According to an embodiment of the present application, an embodiment of a computer terminal is further provided, and the computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above computer terminal may also be replaced with a terminal device such as a mobile terminal.

[0128] Optionally, in this embodiment, the above computer terminal may be located in at least one of multiple network devices of a computer network.

[0129] In this embodiment, the above computer terminal may execute program code for the following steps in a video frame interpolation method of an application program: obtaining a first image and a second image in a video to be frame interpolated, where the first image and the second image are adjacent frames in the video to be frame interpolated; respectively inputting the first image and the second image into two symmetric feature extraction units in a feature extraction network, and predicting feature information of a target image through the feature extraction network, where the target image is an image obtained by frame interpolating the first image and the second image at a target time; predicting the target image based on the feature information.

[0130] Optionally, Figure 7It is a structural block diagram of a computer terminal according to Embodiment 6 of the present application. As Figure 7 shown, the computer terminal 700 may include: one or more (only one is shown in the figure) processors 702, a memory 704, and a peripheral interface 706.

[0131] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video frame interpolation method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned video frame interpolation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely disposed relative to the processor, and these remote memories may be connected to the computer terminal 700 through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The processor is used to run a program, and can call the information and application programs stored in the memory through a transmission device to execute the following steps: obtaining a first image and a second image in the video to be frame interpolated, where the first image and the second image are adjacent frames in the video to be frame interpolated; respectively inputting the first image and the second image into two symmetric feature extraction units in the feature extraction network, and predicting the feature information of the target image through the feature extraction network, where the target image is an image used for frame interpolation of the first image and the second image at the target time; predicting the target image based on the feature information.

[0133] In this embodiment, by respectively inputting the first image and the second image into two symmetric feature extraction units in the feature extraction network, predicting the feature information of the target image through the feature extraction network, and predicting the target image based on the feature information, by using completely symmetric feature extraction units, the feature extraction network is not interfered by the input order of the first image and the second image, and it is realized that the feature extraction network can output exactly the same feature information of the target image when the input order of the first image and the second image is different, solving the technical problem in the related art that the frame interpolation effect is poor due to the influence of the input image order when performing frame interpolation on a video.

[0134] Those of ordinary skill in the art can understand that Figure 7 the structure shown is only schematic, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a mobile Internet device (Mobile Internet Devices, MID), a PAD and other terminal devices. Figure 7It does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 700 may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 7 or have a different configuration from that shown in Figure 7 .

[0135] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0136] Embodiment 7

[0137] According to an embodiment of the present invention, there is also provided an embodiment of a video frame interpolation method. Figure 8 A video frame interpolation method according to Embodiment 7 of the present invention, as shown in Figure 8 , the method includes the following steps:

[0138] Step S801, receiving a video to be frame interpolated and a frame interpolation instruction.

[0139] The above-mentioned video to be frame interpolated is a low-definition video that needs to be frame interpolated.

[0140] The above-mentioned frame interpolation instruction can be triggered by a user's corresponding operation on frame interpolation software on a computing device. The frame interpolation instruction should at least include the identifier of the video to be frame interpolated (for example, the video name) and the frame rate after frame interpolation. For example, when a user needs to increase the frame rate of a low-definition video (i.e., the video to be frame interpolated) with a frame rate of 20fps to 40fps, by performing corresponding operations (such as mouse and keyboard operations on the interface of the frame interpolation software) on the computing device, a frame interpolation instruction is issued. The frame interpolation instruction at least includes the name of the low-definition video and the frame rate of 40fps after frame interpolation. The frame interpolation software can directly obtain the frame rate of 20fps before frame interpolation based on the obtained low-definition video. According to the frame rate of 40fps after frame interpolation in the frame interpolation instruction, the number of images that need to be frame interpolated can be determined.

[0141] Step S802, predicting a target image between any two adjacent frames in the video to be frame interpolated according to the frame interpolation instruction. Among them, any two adjacent frames are respectively input into two symmetric feature extraction units in the feature extraction network, the feature information of the target image is predicted through the feature extraction network, and the target image is predicted based on the feature information.

[0142] The target image is used to represent one or more images inserted between any two adjacent frames. The number of target images can be determined according to the frame interpolation instruction. For example, when increasing the frame rate of a low-definition video with a frame rate of 20fps to 40fps, the number of target images is equal to the number of original images in the low-definition video, and they are inserted between the corresponding two adjacent frames accordingly.

[0143] Any two adjacent frames are two adjacent frames in the original video to be frame interpolated, or two adjacent frames in the video after other processing on the video to be frame interpolated. For example, color restoration processing is performed on the video to be frame interpolated, etc.

[0144] The above feature extraction network can be a neural network model improved based on the U-net network. The feature extraction network can predict the feature information of the target image according to the images of any two adjacent frames input. The feature information can be the optical flow or RGB pixel points of the target image.

[0145] In an alternative embodiment, Figure 3 is the network framework diagram of a video frame interpolation method according to Embodiment 7 of the present application. As Figure 3 shown, the feature extraction network includes two symmetric feature extraction units (the first feature extraction unit 311 and the second feature extraction unit 321). The two symmetric feature extraction units can be symmetric networks improved based on the U-net network. The structures of the first feature extraction unit 311 and the second feature extraction unit 321 of the two U-net networks are symmetric, and can include an encoder and a decoder with the same structure in the first feature extraction unit 311 and the second feature extraction unit 321. The number of downsampling layers in the encoder of the first feature extraction unit 311 is equal to that of the encoder of the second feature extraction unit 321, and the number of upsampling layers in the decoder of the first feature extraction unit 311 is equal to that of the decoder of the second feature extraction unit 321. For the traditional U-net network, it includes an encoder (i.e., the downsampling layer) for feature information extraction and a decoder (i.e., the upsampling layer) for fusing and outputting the extracted feature information, and the input of its decoder comes from its encoder. In this embodiment, the output of the encoder of the first feature extraction unit 311 is connected to the input of the decoder of the second feature extraction unit 321, and the output of the encoder of the second feature extraction unit 321 is connected to the input of the decoder of the first feature extraction unit 311, forming an improved symmetric network with the basic structure of the U-net but with the input and output of the two U-net networks interleaved.

[0146] Two adjacent frames are the first image I0 and the second image I1 respectively. When performing video frame interpolation, the first image I0 is input to the encoder of the first feature extraction unit 311 for multi-layer encoding, and the second image I1 is input to the encoder of the second feature extraction unit 321 for multi-layer encoding. The output of the encoder of the first feature extraction unit 311 is connected to the input of the decoder of the second feature extraction unit 321, and the output of the encoder of the second feature extraction unit 321 is connected to the input of the decoder of the first feature extraction unit 311. The outputs of the decoders of the first feature extraction unit 311 and the second feature extraction unit 321 are cascaded and then subjected to full convolution to output the feature information of the target image as the target image. Since the first feature extraction unit 311 and the second feature extraction unit 321 have a symmetric structure, the feature information of the target image is not affected by the input order of the first image and the second image. Furthermore, no matter in what order the first image and the second image are input into the feature extraction network, the feature information of the exactly same target image can be output.

[0147] Step S803: Perform frame interpolation on the video to be frame-interpolated according to the target image.

[0148] After obtaining the target image, insert the target image between the corresponding two adjacent frames, and the frame interpolation between the two adjacent frames can be completed.

[0149] In this embodiment, the video to be frame-interpolated and the frame interpolation instruction are obtained, and any two adjacent frames are respectively input into two symmetric feature extraction units in the feature extraction network. The feature information of the target image is predicted through the feature extraction network, and the target image is predicted based on the feature information. Then, frame interpolation is performed on the video to be frame-interpolated according to the target image. By using completely symmetric feature extraction units, the feature extraction network is not interfered by the input order of the two adjacent frames, realizing that the feature extraction network can output the feature information of the exactly same target image when the input orders of the two adjacent frames are different, and solving the technical problem in the related art that the frame interpolation effect is poor due to the influence of the input image order when performing frame interpolation on the video.

[0150] It should be noted that the optional or preferred implementation manners of this embodiment can refer to the relevant descriptions in Embodiments 1 to 2, which will not be elaborated here.

[0151] Embodiment 8

[0152] According to an embodiment of the present invention, there is also provided a device for implementing the above video frame interpolation method. Figure 9 For a video frame interpolation device according to Embodiment 8 of the present invention, as Figure 9 shown, the device 900 includes:

[0153] A receiving module 91 for receiving the video to be interpolated and an interpolation instruction; a feature information prediction module 92 for predicting a target image between any two adjacent frames in the video to be interpolated according to the interpolation instruction. Specifically, the two adjacent frames are respectively input into two symmetric feature extraction units in a feature extraction network, the feature information of the target image is predicted by the feature extraction network, and the target image is predicted based on the feature information; an interpolation module 93 for performing interpolation processing on the video to be interpolated according to the target image.

[0154] It should be noted here that the above-mentioned receiving module 91, feature information prediction module 92, and interpolation module 93 correspond to steps S801 to S803 in Embodiment 7. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 7. It should be noted that the above modules, as part of the device, can run in the computing device 10 provided in Embodiment 1.

[0155] It should be noted that the optional or preferred implementation manners of this embodiment can refer to the relevant descriptions in Embodiments 1 to 2, which will not be elaborated here.

[0156] The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0157] In the above-mentioned embodiments of the present invention, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces, and the indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0159] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, in each embodiment of the present invention, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0161] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0162] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A video frame interpolation method, characterized in that, Including: Obtain a first image and a second image in the video to be interpolated, where the first image and the second image are adjacent frames in the video to be interpolated; Input the first image and the second image into two symmetric feature extraction units in a feature extraction network respectively, and predict the feature information of the target image through the feature extraction network, where the target image is an image used for interpolating the first image and the second image at a target time, the feature extraction network includes: a first feature extraction unit and a second feature extraction unit, the structures of the first feature extraction unit and the second feature extraction unit are symmetric, and the output end of the encoder of the first feature extraction unit is connected to the input end of the decoder of the second feature extraction unit, and the output end of the encoder of the second feature extraction unit is connected to the input end of the decoder of the first feature extraction unit; Predict the target image based on the feature information.

2. The method according to claim 1, wherein The first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures.

3. The method according to claim 2, characterized in that, The feature information of the target image includes: the first feature information output by the first feature extraction unit and the second feature information output by the second feature extraction unit. Inputting the first image and the second image into two symmetric feature extraction units in the feature extraction network respectively and predicting the feature information of the target image through the feature extraction network includes: Input the first image, the second image and time information into an optical flow prediction network in sequence, and obtain the first optical flow information output by the optical flow prediction network, where the time information is used to represent the target time when interpolation is required between the first image and the second image; Input the second image, the first image and the time information into the optical flow prediction network in sequence, and obtain the second optical flow information output by the optical flow prediction network; Input the encoding result of the second encoder of the second feature extraction unit and the first optical flow information into the first decoder in the first feature extraction unit to generate the first feature information; Input the encoding result of the first encoder of the first feature extraction unit and the second optical flow information into the first decoder in the second feature extraction unit to generate the second feature information.

4. The method according to claim 3, characterized in that, Inputting the first image, the second image and time information into the optical flow prediction network in sequence and obtaining the first optical flow information output by the optical flow prediction network includes: obtaining the first time information of the target time relative to the first image; inputting the first image, the second image and the first time information into the optical flow prediction network in sequence to obtain the first optical flow information output by the optical flow prediction network; Input the second image, the first image, and the time information into the optical flow prediction network in sequence to obtain the second optical flow information output by the optical flow prediction network, including: obtaining second time information of the target time relative to the second image; inputting the second image, the first image, and the time information into the optical flow prediction network in sequence to obtain the second optical flow information output by the optical flow prediction network.

5. The method according to claim 4, wherein obtaining first time information of the target time relative to the first image includes: obtaining a time difference between the target time and the first image, and performing normalization processing on the time difference to obtain the first time information; obtaining second time information of the target time relative to the second image includes: determining a difference between 1 and the first time information as the second time information.

6. The method according to claim 2, wherein Predicting the target image based on the feature information includes: obtaining a row concatenation of the first feature information output by the first feature extraction unit and the second feature information output by the second feature extraction unit to obtain concatenated feature information; inputting the concatenated feature information into a fully convolutional layer to obtain the target image predicted by the fully convolutional layer.

7. The method according to claim 6, wherein The method further includes: training a video interpolation model with sample data, where the video interpolation model at least includes the feature extraction network and the fully convolutional layer, and the steps of training the video interpolation model with sample data include: inputting the sample data into the video interpolation model to obtain a prediction result output by the video interpolation model; inputting the prediction result into a preset discriminator to obtain a discrimination result output by the discriminator, where the discrimination result is used to indicate whether the prediction result is accurate; adjusting network parameters of the video interpolation model according to the discrimination result.

8. A video frame interpolation method, characterized in that including: obtaining a first image and a second image in an video to be interpolated, where the first image and the second image are adjacent frames in the video to be interpolated; obtaining optical flow information based on the first image, the second image, and time information through an optical flow prediction network, where the time information is used to represent a target time at which interpolation is required between the first image and the second image; inputting the first image and the second image into a first feature extraction unit and a second feature extraction unit respectively to obtain a first coding result of the first image and a second coding result of the second image, where the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures, and an output end of an encoder of the first feature extraction unit is connected to an input end of a decoder of the second feature extraction unit, and an output end of an encoder of the second feature extraction unit is connected to an input end of a decoder of the first feature extraction unit; inputting the second coding result and the optical flow information into a decoder of the first feature extraction unit to output first feature information, and inputting the first coding result and the optical flow information into a decoder of the second feature extraction unit to output second feature information; Predict a target image to be inserted based on the first feature information and the second feature information.

9. The method according to claim 8, characterized in that, After predicting the target image to be inserted based on the first feature information and the second feature information, the method further includes: Insert the target image between the first image and the second image.

10. A video frame interpolation device, characterized in that, Includes: An acquisition module, configured to acquire a first image and a second image in a video with frames to be inserted, where the first image and the second image are adjacent frames in the video with frames to be inserted; An input module, configured to respectively input the first image and the second image into two symmetric feature extraction units in a feature extraction network, and predict feature information of a target image through the feature extraction network, where the target image is an image for frame interpolation between the first image and the second image at a target time, the feature extraction network includes: a first feature extraction unit and a second feature extraction unit, the structures of the first feature extraction unit and the second feature extraction unit are symmetric, and an output end of an encoder of the first feature extraction unit is connected to an input end of a decoder of the second feature extraction unit, and an output end of an encoder of the second feature extraction unit is connected to an input end of a decoder of the first feature extraction unit; A prediction module, configured to predict the target image based on the feature information.

11. A video frame interpolation device, characterized in that, Includes: An image acquisition module, configured to acquire a first image and a second image in a video with frames to be inserted, where the first image and the second image are adjacent frames in the video with frames to be inserted; An optical flow acquisition module, configured to acquire optical flow information based on the first image, the second image, and time information through an optical flow prediction network, where the time information is used to represent a target time at which frame interpolation needs to be performed between the first image and the second image; An encoding module, configured to respectively input the first image and the second image into a first feature extraction unit and a second feature extraction unit to obtain a first encoding result of the first image and a second encoding result of the second image, where the first feature extraction unit and the second feature extraction unit are symmetric encoder-decoder structures, and an output end of an encoder of the first feature extraction unit is connected to an input end of a decoder of the second feature extraction unit, and an output end of an encoder of the second feature extraction unit is connected to an input end of a decoder of the first feature extraction unit; A decoding module, configured to input the second encoding result and the optical flow information into a decoder of the first feature extraction unit to output first feature information, and input the first encoding result and the optical flow information into a decoder of the second feature extraction unit to output second feature information; An insertion module, configured to predict a target image to be inserted based on the first feature information and the second feature information.

12. A storage medium, characterized in that, The storage medium includes a stored program, where when the program runs, it controls the device where the storage medium is located to execute the video frame interpolation method according to any one of claims 1 to 9.

13. A processor, characterized in that, The processor is used to run a program, wherein, when the program runs, it executes the video frame interpolation method described in any one of claims 1 to 9.

14. A video frame interpolation method, characterized in that, It includes: Receiving a video to be frame interpolated and a frame interpolation instruction; Predicting a target image between any two adjacent frames in the video to be frame interpolated according to the frame interpolation instruction, wherein the two adjacent frames are respectively input into two symmetric feature extraction units in a feature extraction network, predicting the feature information of the target image through the feature extraction network, and predicting the target image based on the feature information, wherein the feature extraction network includes: a first feature extraction unit and a second feature extraction unit, the structure of the first feature extraction unit is symmetric to the structure of the second feature extraction unit, and the output end of the encoder of the first feature extraction unit is connected to the input end of the decoder of the second feature extraction unit, and the output end of the encoder of the second feature extraction unit is connected to the input end of the decoder of the first feature extraction unit; Performing frame interpolation processing on the video to be frame interpolated according to the target image.

Images