Interpolation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an interpolation method and device, an electronic device and a storage medium. The interpolation method comprises: obtaining an input source, an input frame rate and an output frame rate of the input source; obtaining corresponding interpolation configuration information according to the input frame rate, the output frame rate and an interpolation configuration, the interpolation configuration information being associated with an interpolation for realizing frame rate conversion from the input frame rate to the output frame rate; and performing interpolation processing on the input source by using the interpolation configuration information to obtain a target source with the output frame rate. Such an interpolation method can realize uniform and smooth interpolation of a video, improve visual effects and enhance user viewing experience.

Description

Technical Field

[0001] This disclosure belongs to the field of video frame interpolation technology, and in particular relates to frame interpolation methods and apparatus, electronic devices and storage media. Background Technology

[0002] Frame interpolation is a technique that increases the frame rate of a video by inserting new frames between existing frames. It is widely used in television, film, and games. The core goal of frame interpolation is to improve the smoothness and visual effects of video, making fast-moving scenes appear more coherent and natural. Frame interpolation is particularly important in high-speed motion scenes.

[0003] With improvements in hardware performance and optimization of algorithms, frame interpolation technology has been widely applied across various devices. From high-definition televisions to virtual reality devices, the application scope of frame interpolation technology continues to expand, meeting users' demands for high frame rate video. In short, video frame interpolation technology significantly improves the frame rate and viewing experience by inserting new frames between existing video frames. Its widespread application in various scenarios demonstrates its importance in enhancing visual effects and improving user experience. Summary of the Invention

[0004] The purpose of this disclosure is to provide frame interpolation methods, apparatus, electronic devices, and storage media to solve the problems of uneven frame interpolation intervals and poor visual effects in the prior art.

[0005] In a first aspect, embodiments of this disclosure provide a frame interpolation method. The frame interpolation method includes: acquiring an input source, an input frame rate of the input source, and an output frame rate; acquiring corresponding frame interpolation configuration information based on the input frame rate, the output frame rate, and an interpolation configuration, wherein the frame interpolation configuration information is associated with frame interpolation that implements frame rate conversion from the input frame rate to the output frame rate; and performing frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source having the output frame rate.

[0006] In one implementation of the first aspect, obtaining interpolation configuration information based on the input frame rate, the output frame rate, and the interpolation configuration includes: analyzing the input frame rate and the output frame rate to obtain the interpolation configuration; obtaining corresponding compensation frame information based on the interpolation configuration, the input frame rate, and the output frame rate; and obtaining the interpolation configuration information using the compensation frame information and the interpolation configuration.

[0007] In one implementation of the first aspect, analyzing the input frame rate and the output frame rate to obtain the interpolation configuration includes: obtaining the corresponding interpolation network model and the number of calls based on the ratio of the output frame rate to the input frame rate; and obtaining the interpolation configuration based on the input frame rate, the output frame rate, the interpolation network model, and the number of calls.

[0008] In one implementation of the first aspect, performing frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source having the output frame rate includes: performing frame interpolation processing between the current frame and the next frame of the input source according to the frame interpolation configuration information to obtain a target source having the output frame rate.

[0009] In one implementation of the first aspect, the frame interpolation method further includes: obtaining inter-frame motion information between frames of the input source based on the input source; and performing motion compensation for frame interpolation processing using the inter-frame motion information and the frame interpolation configuration information.

[0010] In one implementation of the first aspect, obtaining inter-frame motion information between frames of the input source based on the input source includes: obtaining an interpolation ratio using the interpolation configuration information; obtaining the interpolation time between frames based on the interpolation ratio; and using a convolutional neural network to predict the motion between frames of the input source to obtain the inter-frame motion information at the interpolation time.

[0011] In one implementation of the first aspect, using a convolutional neural network to predict the motion between frames of the input source to obtain inter-frame motion information at the interpolation time includes: obtaining an interpolation frame according to the interpolation frame configuration information, the interpolation frame including a reference frame and a non-reference frame; performing downsampling processing on the current frame and the previous frame to obtain multiple layers of current frames and previous frames with different resolutions; inputting the current frame and the previous frame into the current layer of the convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame in the current layer; and inputting the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame, along with the current frame and the previous frame of the next layer at the next resolution, into the next layer of the convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame in the next layer.

[0012] In one implementation of the first aspect, concatenating the current frame and the previous frame and inputting them into the current layer convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame includes: using the current layer convolutional neural network to obtain the optical flow and mask of the reference frame and the offset and mask of the non-reference frame; obtaining the optical flow of the non-reference frame based on the optical flow of the reference frame and the offset of the non-reference frame; and obtaining the inter-frame motion information of the reference frame based on the optical flow and mask of the reference frame, and obtaining the inter-frame motion information of the non-reference frame based on the optical flow and offset of the non-reference frame.

[0013] In one implementation of the first aspect, the frame interpolation method further includes: using the inter-frame motion information at the current moment as input to obtain the inter-frame motion information at the next moment, so as to transmit the temporal information of the input source.

[0014] In one implementation of the first aspect, the frame interpolation method further includes: using an encoder to obtain motion vectors between frames of the input source to obtain inter-frame basic motion information of the input source.

[0015] In one implementation of the first aspect, the frame interpolation method further includes: detecting the input source using an image detection network to obtain the foreground region of the input source; and performing protection processing on the foreground region of the input source according to its location.

[0016] In one implementation of the first aspect, the frame interpolation method further includes: determining the true frame rate of the video from the input source, and using the true frame rate as the input frame rate.

[0017] Secondly, embodiments of this disclosure provide a frame interpolation apparatus. The frame interpolation apparatus includes: an input parameter module configured to acquire an input source, an input frame rate of the input source, and an output frame rate; a frame interpolation configuration module configured to acquire corresponding frame interpolation configuration information based on the input frame rate, the output frame rate, and the frame interpolation configuration, wherein the frame interpolation configuration information is associated with frame interpolation that implements frame rate conversion from the input frame rate to the output frame rate; and a frame interpolation processing module configured to perform frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source having the output frame rate.

[0018] In one implementation of the second aspect, the frame interpolation processing module is configured to: obtain the foreground region of the Nth frame, which is the current frame, through a first processing unit and perform protection processing on the foreground region; perform downsampling processing on the Nth frame through a second processing unit; and simultaneously obtain inter-frame motion information of the (N-1)th frame and the Nth frame through a third processing unit; and obtain inter-frame motion information of the (N-2)th frame and the (N-1)th frame through the third processing unit, and simultaneously perform motion compensation on the (N-2)th frame and the (N-1)th frame using the inter-frame motion information of the (N-2)th frame and the (N-1)th frame through the second processing unit, where N is an integer greater than 2.

[0019] Thirdly, embodiments of this disclosure provide an electronic device. The electronic device includes: a memory configured to store an executable program; and a processor configured to execute the program to cause the electronic device to perform the frame interpolation method according to any one of the first aspects.

[0020] Fourthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed, implements the frame interpolation method according to any one of the first aspects.

[0021] According to embodiments of this disclosure, the input source, input frame rate, and output frame rate can be used to obtain corresponding frame interpolation configuration information. This frame interpolation configuration information is then used to perform smooth frame interpolation processing on the input source to obtain the target source. This frame interpolation method can achieve uniform and smooth frame interpolation in video, improving visual effects and enhancing the user's viewing experience. Attached Figure Description

[0022] Figure 1 The diagram shows an application scenario of the frame interpolation method described in this disclosure.

[0023] Figure 2 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0024] Figure 3 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0025] Figure 4 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0026] Figure 5 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0027] Figure 6A The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0028] Figure 6BThe diagram shown is a schematic diagram of obtaining inter-frame motion information according to an embodiment of this disclosure.

[0029] Figure 7 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0030] Figure 8 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure.

[0031] Figure 9 The diagram shown is a structural schematic of the frame interpolation device according to an embodiment of this disclosure.

[0032] Figure 10 The diagram shown is a structural schematic of the frame interpolation processing module described in an embodiment of this disclosure.

[0033] Figure 11 The diagram shown is a structural schematic of the electronic device described in an embodiment of this disclosure. Detailed Implementation

[0034] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0035] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0036] With the rapid development of the consumer electronics industry in recent years, the screen size and resolution of electronic products have become increasingly larger. However, due to limitations in transmission bandwidth and historical shooting techniques, most local and streaming online videos have a frame rate of 30 frames per second or lower (mostly 24 / 25 / 30 frames per second). When these frame rate videos are played on mainstream playback devices with a refresh rate of 60 frames per second, they are prone to dragging and stuttering, affecting the visual viewing experience.

[0037] Current commonly used solutions typically only handle interpolation rates that are powers of 2, failing to meet the upsampling requirements of arbitrary frame rates. For example, a simple 2x interpolation cannot meet the requirements for interpolating from 25 to 60 frames per second. Some solutions utilize linear combinations of power-of-2 interpolation rates, such as using a combination of four 2x and one 4x interpolation loop for 25 to 60 frames per second, but the overall time interval is not uniform enough, resulting in a visually inconsistent speed. Some solutions can support interpolation at any point between two frames, but interpolating multiple frames requires multiple repeated calls, which does not meet the real-time requirements of mobile devices.

[0038] Furthermore, traditional frame interpolation techniques primarily rely on motion compensation algorithms, which generate new intermediate frames by estimating motion vectors between adjacent frames. However, this method has significant limitations. First, motion compensation algorithms have limited processing power, especially in scenes with fast motion and drastic changes. Because simple linear models cannot accurately match nonlinear motion, they cannot accurately capture motion trajectories. This results in blurred, ghosting, and misaligned intermediate frames, severely impacting the video's visual quality. Second, due to hardware performance limitations, practical applications often require a trade-off between frame rate improvement and processing time, making it impossible to achieve smooth and uniform frame interpolation results.

[0039] Furthermore, most deep learning frame interpolation schemes do not take into account the computing power constraints of mobile terminal devices, making them unsuitable for deployment on general mobile terminals and greatly limiting the practical application scenarios of this technology.

[0040] At least to address the aforementioned problems, this disclosure provides a frame interpolation method. The frame interpolation method includes: acquiring an input source, an input frame rate of the input source, and an output frame rate; acquiring corresponding frame interpolation configuration information based on the input frame rate, the output frame rate, and an interpolation configuration, wherein the frame interpolation configuration information is associated with frame interpolation that implements frame rate conversion from the input frame rate to the output frame rate; and performing frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source having the output frame rate.

[0041] In this embodiment, the input source, input frame rate, and output frame rate can be used to obtain corresponding frame interpolation configuration information. This frame interpolation configuration information is then used to perform smooth frame interpolation processing on the input source to obtain the target source. This frame interpolation method can achieve uniform and smooth frame interpolation in video, improving visual effects and enhancing the user's viewing experience.

[0042] This disclosure proposes a method for real-time arbitrary frame rate upsampling on mobile devices using deep learning. To achieve real-time performance and arbitrary frame rate upsampling on mobile devices, a combination of interpolation neural networks with different interpolation ratios is proposed. For example, this invention uses a combination of 4x and 5x networks. A 4x network is one that, for every two input frames (frame t and frame t+1), outputs three intermediate frames (frames t+0.25 / t+0.5 / t+0.75), thus increasing the frame rate by 4 times. Similarly, a 5x network is one that, for every two input frames (frame t and frame t+1), outputs four intermediate frames (frames t+0.2 / t+0.4 / t+0.6 / t+0.8), thus increasing the frame rate by 5 times. These two ratios are used because they are sufficient to meet the frame rate requirements of common videos. It should be understood that the proposed network structure is not only applicable to 4x and 5x interpolation networks, but theoretically applicable to any Nx (N is a positive integer) network structure design. Therefore, the networks that can be combined are not limited to 4x and 5x networks.

[0043] In this embodiment, to make the interpolated frames more coherent and smooth in the temporal domain, a temporal information transmission path is added to the interpolation network. Motion data output from the previous frame is added as additional information to the input of the current frame. This allows the network to selectively reference motion information from the previous moment based on actual conditions, reducing flickering artifacts caused by single-frame errors. Simultaneously, the motion vector output from the decoder is combined to improve motion vector initialization and enhance the network's robustness.

[0044] In this embodiment, in the logo and subtitle area, since the subtitles and logo are usually stationary, there is often a significant difference in motion between the foreground and background, which can easily lead to motion defects. By introducing an additional logo and subtitle protection module, a neural network is used to detect the logo and subtitle areas in the current frame, and the area information is passed into the frame interpolation network, thereby protecting the logo and subtitles.

[0045] Figure 1 The diagram illustrates an application scenario of the frame interpolation method described in this disclosure. Communication can occur between the terminal device and the cloud server, and the communication method is not limited to wired or wireless. In some embodiments, the frame interpolation method described in this disclosure can be executed solely by the terminal device, while the terminal device and the cloud server perform other data communication besides the frame interpolation method. In this embodiment, the frame interpolation method described in this disclosure can be executed by a system chip in the terminal device, which may include a CPU, NPU, and GPU, etc. In other embodiments, the frame interpolation method described in this disclosure can be executed jointly by the terminal device and the cloud server.

[0046] The terminal device can be mobile or fixed. For example, it can be a wireless terminal or a wired terminal. A wireless terminal can refer to a device with wireless transceiver capabilities, which can be deployed indoors and outdoors. The terminal device can be a mobile phone, tablet computer, laptop computer, etc., and is not limited thereto. The cloud server can include one or more servers, or one or more processing nodes, or one or more virtual machines running on the server. The cloud server can also be referred to as a server cluster, management platform, data processing center, etc., and is not limited thereto in this disclosure.

[0047] The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0048] The following embodiments of this disclosure provide a frame interpolation method. Figure 2 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 2 As shown, the frame interpolation method includes steps S11 to S13.

[0049] Step S11: Obtain the input source, its input frame rate, and its output frame rate. The input source's output frame rate is a low frame rate, and its output frame rate is a high frame rate.

[0050] Step S12: Obtain corresponding interpolation configuration information based on the input frame rate, the output frame rate, and the interpolation configuration. The interpolation configuration information is associated with the interpolation used to implement frame rate conversion from the input frame rate to the output frame rate. The interpolation configuration information is used to determine the interpolation frame of the input source, and to perform interpolation on the input source based on the interpolation frame of the input source to obtain a target source with the output frame rate.

[0051] Step S13: Use the frame interpolation configuration information to perform frame interpolation processing on the input source to obtain a target source with the output frame rate.

[0052] In this embodiment, the input source, input frame rate, and output frame rate can be used to obtain corresponding frame interpolation configuration information. This frame interpolation configuration information is then used to perform smooth frame interpolation processing on the input source to obtain the target source. This frame interpolation method can achieve uniform and smooth frame interpolation in video, improving visual effects and enhancing the user's viewing experience.

[0053] Figure 3 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 3 As shown, step S12 includes steps S121 to S123.

[0054] Step S121: Analyze the input frame rate and the output frame rate to obtain the frame interpolation configuration.

[0055] Step S122: Obtain the corresponding compensation frame information according to the interpolation configuration, the input frame rate, and the output frame rate.

[0056] Step S123: Obtain the interpolation configuration information using the compensation frame information and the interpolation configuration.

[0057] In some possible implementations, the frame interpolation configuration includes the frame interpolation network model used and the number of calls. Based on the frame interpolation network model used, the number of calls, the input frame rate, and the output frame rate, the corresponding compensation frame information of the input source can be obtained. The compensation frame information and the frame interpolation configuration together serve as the frame interpolation configuration information, which is used to implement frame interpolation processing of the input source from the input frame rate to the output frame rate.

[0058] In some other possible implementations, assuming the interpolation network library stores n different frame interpolation network models with varying magnifications, the formula for calculating the number of calls to achieve the conversion between input and output frame rates is:

[0059]

[0060] Where, k i Let t be the scaling factor of the i-th interpolation network model. i Let r be the number of times the i-th frame interpolation network model is called. in Input frame rate, r out This is the output frame rate.

[0061] It should be noted that the above formula for calculating the number of calls can theoretically achieve uniform frame interpolation when n is large enough, but this requires maintaining a sufficient number of frame interpolation network models with different magnifications.

[0062] Figure 4 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 4 As shown, step S121 includes steps S1211 to S1212.

[0063] Step S1211: Obtain the corresponding frame interpolation network model and the number of calls based on the ratio of the output frame rate to the input frame rate.

[0064] Step S1212: Obtain the frame interpolation configuration based on the input frame rate, the output frame rate, the frame interpolation network model, and the number of calls.

[0065] In some possible implementations, the input frame rate of regular streaming video is typically 24, 25, 30, or 60 FPS (Frames Per Second). The output frame rate is typically 60 or 120 FPS. For example, using a 4x interpolation network model and a 5x interpolation network model for frame interpolation processing, for the two frames t and t+1 of the 4x interpolation network model, outputting the three intermediate frames at time t+0.25, t+0.5, and t+0.75 can increase the overall frame rate by 4 times. For the two frames t and t+1 of the 5x interpolation network model, outputting the three intermediate frames at time t+0.2, t+0.4, t+0.6, and t+0.8 can increase the overall frame rate by 5 times. The interpolated frame is the frame obtained using the interpolation network model and used to insert between frames; the interpolated frame is the frame actually used during the frame interpolation processing. The following sections will explain how to obtain frame interpolation configuration information using several common input and output frame rates.

[0066] Input frame rate is 60 FPS, output frame rate is 60 FPS: no frame interpolation is required.

[0067] The input frame rate is 60 FPS, and the output frame rate is 120 FPS: a 4x interpolation network model is used. For the current frame n and the next frame n+1, the 4x interpolation network model is used to obtain interpolated frames at times n+0.25, n+0.5, and n+0.75. The interpolated frames at times n+0.25 and n+0.75 are discarded, and the interpolated frame at time n+0.5 is retained as the interpolated frame. One frame is added for each subsequent frame. By performing frame interpolation processing on the input source, the target source with an output frame rate of 120 FPS can be obtained.

[0068] Input frame rate is 30 FPS, output frame rate is 60 FPS: A 4x frame interpolation network model is used, which is similar to the frame interpolation method with an input frame rate of 60 FPS and an output frame rate of 120 FPS.

[0069] The input frame rate is 30 FPS and the output frame rate is 120 FPS: Using a 4x interpolation network model, the interpolated frames at times n+0.25, n+0.5 and n+0.75 are obtained as interpolated frames. By performing interpolation processing on the input source, the target source with an output frame rate of 120 FPS can be obtained.

[0070] The input frame rate is 25 FPS and the output frame rate is 60 FPS. Theoretically, the time interval for uniform frame interpolation from 25 FPS to 60 FPS is 25 / 60≈0.417. Therefore, 4x and 5x interpolation network models are used alternately to achieve an interpolation time interval close to 0.417, resulting in smooth video in actual use. Using a 5x frame interpolation network model, interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8 are obtained. The interpolated frames at times n+0.2 and n+0.6 are discarded, while the interpolated frames at times n+0.4 and n+0.8 are retained as interpolated frames. Then, using a 4x frame interpolation network model, interpolated frames at times n+0.25, n+0.5, and n+0.75 are obtained. The interpolated frame at time n+0.5 is discarded, while the interpolated frames at times n+0.25 and n+0.75 are retained as interpolated frames. Then, using a 5x frame interpolation network model, interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8 are obtained. The interpolated frames at times n+0.4 and n+0.8 are discarded, while the interpolated frames at times n+0.2 and n+0.6 are retained as interpolated frames. Finally, using a 5x frame interpolation network model... The network model is used to obtain interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8, discarding the interpolated frames at times n+0.2 and n+0.6, and retaining the interpolated frames at times n+0.4 and n+0.8 as interpolated frames. Then, a 5x interpolation network model is used to obtain interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8, discarding the interpolated frames at times n+0.4 and n+0.8, and retaining the interpolated frames at times n+0.2 and n+0.6 as interpolated frames. In this way, interpolation processing is performed from time 0 to 5 using interpolated frames at times 0.4, 0.8, 1.25, 1.75, 2.2, 2.6, 3.0, 3.4, 3.8, 4.2, and 4.6, adding 11 frames every 5 frames to obtain the target source of an output frame rate of 60 FPS.

[0071] Input frame rate 25 FPS, output frame rate 120 FPS: Similar to the frame interpolation method with an input frame rate of 25 FPS and an output frame rate of 120 FPS, each round consists of 5 calls, with 4 rounds using a 5x interpolation network model and 1 round using a 4x interpolation network model. All interpolated frames are retained as interpolation frames, and every 5 frames are interpolated to 24 frames. For example: Implement frame interpolation processing from time 0 to 5 using interpolated frames at times 0.2, 0.4, 0.6, 0.8, 1.2, 1.4, 1.6, 1.8, 2.2, 2.4, 2.6, 2.8, 3.2, 3.4, 3.6, 3.8, 4.25, 4.5, 4.75, increasing by 19 frames every 5 frames to obtain the target source of 120 FPS output frame rate.

[0072] The input frame rate is 24 FPS, and the output frame rate is 60 FPS. Using a 5x interpolation network model, interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8 are obtained. The interpolated frames at times n+0.2 and n+0.6 are discarded, and the interpolated frames at times n+0.4 and n+0.8 are retained as interpolated frames. This is repeated again using the same 5x interpolation network model, again using interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8, discarding the interpolated frames at times n+0.4 and n+0.8, and retaining the interpolated frames at times n+0.2 and n+0.6. This achieves frame interpolation from time 0 to 2 using interpolated frames at times 0.4, 0.8, 1.2, and 1.6, adding 4 frames every 2 frames to obtain the target source for an output frame rate of 60 FPS.

[0073] The input frame rate is 24 FPS, and the output frame rate is 120 FPS: Using a 5x interpolation network model, interpolated frames at times n+0.2, n+0.4, n+0.6, and n+0.8 are obtained. All interpolated frames at all times are retained as interpolated frames. This achieves frame interpolation processing from time 0 to 1 using interpolated frames at times 0.2, 0.4, 0.6, and 0.8. Four frames are added for each frame to obtain the target source of an output frame rate of 120 FPS.

[0074] It should be noted that the above is only the conventional interpolation frame rate configuration information according to the technical solution of the embodiments of this disclosure, and the specific method of implementing smooth interpolation between the input frame rate and the output frame rate is not limited to this.

[0075] In one embodiment of this disclosure, performing frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source having the output frame rate includes: performing frame interpolation processing between the current frame and the next frame of the input source according to the frame interpolation configuration information to obtain the target source.

[0076] In some other possible implementations, in order to achieve smooth frame interpolation between the input frame rate and the output frame rate, frame interpolation is performed in groups of several frames that can be repeatedly implemented, wherein the starting frame in the group is the current frame and the ending frame is the next frame, and the frame interpolation is performed in this cyclical manner to obtain the target source with the output frame rate.

[0077] Figure 5 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 5 As shown, the frame interpolation method further includes steps S14 to S15.

[0078] Step S14: Obtain inter-frame motion information between frames of the input source based on the input source.

[0079] Step S15: Motion compensation for frame interpolation processing is performed using the inter-frame motion information and the frame interpolation configuration information.

[0080] In some possible implementations, inter-frame motion information between frames of the input source is obtained based on the input source, and frame interpolation compensation is performed based on the inter-frame motion information. The calculation formula for frame interpolation compensation is as follows:

[0081] I n =M*Warp(I0,F n→0 )+(1-M)*Warp(I1,F n→1 ),

[0082] Where 1 and 0 represent the current time and the previous time, respectively, n is a time between 0 and 1, Warp is the inverse interpolation transform function, and I n Let I0 be the interpolation frame at time n, I1 be the input frame at the previous time, and F be the input frame at the current time. n→0 Let M be the optical flow from time n to time 0, and M be the mask.

[0083] Figure 6A The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 6A As shown, step S14 includes steps S141 to S143.

[0084] Step S141: Obtain the interpolation ratio using the interpolation configuration information. The interpolation ratio is a multiple of the interpolation network model, such as a 4x interpolation network model or a 5x interpolation network model.

[0085] Step S142: Obtain the interpolation time between frames according to the interpolation ratio.

[0086] Step S143: Use a convolutional neural network to predict the motion between frames of the input source to obtain the inter-frame motion information at the interpolation time.

[0087] In some possible implementations, the interpolation ratio is obtained based on the interpolation configuration information, thus revealing the interpolation network model used for interpolation. The interpolation time between frames is obtained based on the adopted interpolation network model, the input frame rate, and the output frame rate. A convolutional neural network is used to predict the motion between each frame of the input source, obtaining the optical flow and mask between each frame. The inter-frame motion information at the interpolation time is then obtained using the optical flow and mask.

[0088] In some other possible implementations, Figure 6B The diagram shown is a schematic representation of the acquisition of inter-frame motion information as described in an embodiment of this disclosure. Figure 6BAs shown, taking a 4x frame interpolation network model as an example, the interpolation times are the interpolated frames at times n+0.25, n+0.5, and n+0.75 of the 4x frame interpolation network model. The current frame n and the previous frame n-1 represent the image frames at times t=1 and t=0, respectively. The current frame and the previous frame are sampled twice by 1 / 2. The first-stage convolutional neural network is used to predict the 1 / 4 downsampled current frame and the previous frame to obtain the inter-frame motion information at times n+0.25, n+0.5, and n+0.75. The 1 / 4 downsampled current frame and the previous frame are concatenated with the inter-frame motion information predicted by the first-stage convolutional neural network and then input into the second-stage convolutional neural network. The second-stage convolutional neural network predicts the inter-frame motion information at times n+0.25, n+0.5, and n+0.75 by using a 1 / 2 downsampled current frame and the previous frame. This inter-frame motion information is then concatenated with the inter-frame motion information predicted by the first-stage convolutional neural network and input into the third-stage convolutional neural network. The third-stage convolutional neural network then predicts the inter-frame motion information at times n+0.25, n+0.5, and n+0.75 by using the first-stage convolutional neural network. Finally, this concatenation of the inter-frame motion information at the interpolation time is used to obtain the inter-frame motion information at the interpolation time.

[0089] It should be noted that the above is only one possible implementation of the embodiments of this disclosure, and this disclosure is not limited thereto.

[0090] Figure 7 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 7 As shown, step S143 includes steps S1431 to S1434.

[0091] Step S1431: Obtain interpolation frames according to the interpolation configuration information. The interpolation frames include reference frames and non-reference frames. The reference frame is any frame that the interpolation network model interpolates between two frames, and the non-reference frame is any frame other than the reference frame when the interpolation network model performs interpolation between two frames.

[0092] Step S1432: Downsample the current frame and the previous frame to obtain multiple layers of the current frame and the previous frame with different resolutions.

[0093] Step S1433: Input the current frame and the previous frame into the current layer convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frames in the current layer.

[0094] Step S1434: Input the inter-frame motion information of the reference frame of the current layer, the inter-frame motion information of the non-reference frame, and the current frame and the previous frame of the resolution of the next layer into the next layer convolutional neural network, so as to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame of the next layer.

[0095] In some possible implementations, the required interpolation network model is obtained based on the interpolation configuration information, and the reference frame and non-reference frames of the interpolated frames are obtained based on the interpolation network model. Taking a 4x interpolation network model as an example, the intermediate time n+0.5 is the reference frame, and the times n+0.25 and n+0.75 are the non-reference frames. The current frame and the previous frame are downsampled twice to obtain three layers of current and previous frames at different resolutions: the original resolution current and previous frames, the 1 / 2 resolution current and previous frames, and the 1 / 4 resolution current and previous frames. The 1 / 4 resolution current and previous frames are input into the current layer of the convolutional neural network to output the inter-frame motion information of the reference frame at time n+0.5, and the inter-frame motion information of the non-reference frames at times n+0.25 and n+0.75. The inter-frame motion information of the reference frame at time n+0.5 of the current layer, the inter-frame motion information of the non-reference frames at times n+0.25 and n+0.75, and the current frame and the previous frame at half resolution of the next layer are input into the next layer convolutional neural network to output the inter-frame motion information of the reference frame at time n+0.5 of the next layer, as well as the inter-frame motion information of the non-reference frames at times n+0.25 and n+0.75. This disclosure is not limited thereto.

[0096] Figure 8 The diagram shown is a flowchart illustrating the frame interpolation method described in an embodiment of this disclosure. Figure 8 As shown, step S1433 includes steps S14331 to S14333.

[0097] Step S14331: Use the current layer convolutional neural network to obtain the optical flow and mask of the reference frame and the offset and mask of the non-reference frame in the current layer.

[0098] Step S14332: Obtain the optical flow of the non-reference frame based on the optical flow of the reference frame and the offset of the non-reference frame.

[0099] Step S14333: Obtain the inter-frame motion information of the reference frame based on the optical flow and mask of the reference frame, and obtain the inter-frame motion information of the non-reference frame based on the optical flow and offset of the non-reference frame.

[0100] In some possible implementations, taking a 4x frame interpolation network model as an example, the optical flow and mask of the reference frame at time n+0.5, and the offsets and masks of the non-reference frames at times n+0.25 and n+0.75 are obtained using the current layer convolutional neural network. The optical flow of the non-reference frames at times n+0.25 and n+0.75 is obtained based on the optical flow of the reference frame at time n+0.5 and the offsets of the non-reference frames at times n+0.25 and n+0.75. The inter-frame motion information of the reference frame is obtained based on the optical flow and mask of the reference frame at time n+0.5, and the inter-frame motion information of the non-reference frames is obtained based on the optical flow and offset of the non-reference frames at times n+0.25 and n+0.75. The formula for calculating the optical flow of the non-reference frames is:

[0101] F 0.25→0 =(0.5+Δ 0.25→0 )×F 0.5→0

[0102] F 0.25→1 =(1.5+Δ 0.25→1 )×F 0.5→1

[0103] F 0.75→0 =(1.5+Δ 0.75→0 )×F 0.5→0 ,

[0104] F 0.75→1 =(0.5+Δ 0.75→1 )×F 0.5→1

[0105] Among them, F 0.5→0 For optical flow from 0.5 to 0 frames, F 0.5→1 The optical flow is 0.5 to 1 frame, Δ is the offset, and F is the optical flow. 0.25→0 and F 0.25→1 For the optical flow of the non-reference frame at time n+0.25, F 0.75→0 and F 0.75→1 This is the offset of the non-reference frame at time n+0.75.

[0106] Assuming uniform motion between the previous and current frames, the optical flow from frame 0.25 to frame 0 should be half the optical flow from frame 0.5 to frame 0, with an offset Δ. 0.25→0 Approaching zero. For the reference frame, two optical flows and one mask are predicted using a convolutional neural network; for non-reference frames, two offsets and one mask are predicted using the same convolutional neural network.

[0107] In other possible implementations, the method described in this disclosure, which inputs the current frame and the previous frame into the current layer of a convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frames, is not limited to a 4x interpolation network model and can be extended to any integer multiple interpolation network model. Taking a 5x interpolation network model as an example, the current frame and the previous frame are input, and time n+0.4 is selected as the reference frame, while times n+0.2, n+0.6, and n+0.8 are selected as non-reference frames. The inter-frame motion information of the reference frame and non-reference frames is then predicted using the convolutional neural network. Taking an arbitrary nx interpolation network model as an example, if n is even, the interpolated frame at time n+0.5 is selected as the reference frame; if n is odd, the interpolated frame closest to time n+0.5 is selected as the reference frame. Assuming the reference frame is b, the formula for calculating the optical flow at time t relative to time k (k is time 0 or 1) is:

[0108]

[0109] F t→k Let Δ be the optical flow at time t relative to time k. t→k F is the offset at time t relative to time k. b→k Let b be the optical flow relative to k.

[0110] In one embodiment of this disclosure, the frame interpolation method further includes: using the inter-frame motion information at the current moment as input to obtain the inter-frame motion information at the next moment, so as to transmit the temporal information of the input source.

[0111] In some possible implementations, inter-frame motion information at time n and time n+1 is acquired, and the inter-frame motion information at time n and time n+1 is used as input to acquire inter-frame motion information at time n+1 and time n+2, so as to transmit the temporal information of the input source.

[0112] In this embodiment of the disclosure, using the inter-frame motion information at the current moment as the input for obtaining the inter-frame motion information at the next moment can increase the stability in the temporal domain and prevent the visual effect of flickering caused by the prediction error of the convolutional neural network in a certain frame.

[0113] In one embodiment of this disclosure, the frame interpolation method further includes: using an encoder to obtain motion vectors between frames of the input source to obtain inter-frame basic motion information of the input source.

[0114] In some possible implementations, the encoder is used to obtain the motion vectors between each frame of the input source as the reference motion information between each frame, so as to obtain the inter-frame motion information of each frame of the input source.

[0115] In one embodiment of this disclosure, the frame interpolation method further includes: detecting the input source using an image detection network to obtain the foreground region of the input source; and performing protection processing on the foreground region of the input source according to its location.

[0116] In some possible implementations, to prevent static subtitles and logos from moving with the background in videos with significant background motion, an image detection network is used to detect the input source and obtain its foreground region. When using a convolutional neural network to predict inter-frame motion information, the optical flow of the foreground region during the first stage of convolutional neural network prediction corresponding to the current frame (downsampled by 1 / 4) and the previous frame, as well as the optical flow during the second stage of convolutional neural network prediction corresponding to the current frame (downsampled by 1 / 2) and the previous frame, is set to 0. This eliminates large motion in the foreground region, and only the optical flow predicted by the third stage of convolutional neural network corresponding to the original current frame and the previous frame is used as the motion result of the foreground region, thus protecting it.

[0117] In one embodiment of this disclosure, the frame interpolation method further includes: determining the true frame rate of the video from the input source, and using the true frame rate as the input frame rate.

[0118] In some possible implementations, the actual frame rate of video sources existing in the current network may differ from the true frame rate due to secondary processing. For example, a video that is actually 60fps may only have a true frame rate of 30fps. In the actual video, each frame is repeated once, resulting in an arrangement like 0-0-1-1-2-2-3-3-… Before performing frame interpolation using the aforementioned method, the true frame rate is first determined, and then frame interpolation is performed based on both the true frame rate and the current frame rate.

[0119] At time n, input frames n and n-1, and determine the similarity between the two frames. The similarity determination includes, but is not limited to, the difference between the histograms of frame n and frame n-1. If the similarity between the two frames is less than a certain threshold, the two frames are considered to be the same, denoted as S (Same); otherwise, the two frames are considered to be different, denoted as D (Different).

[0120] For the first N frames, only the similarity markers between two frames are counted and recorded. After N markers have been recorded, the true frame rate of the video source can be determined based on the recorded patterns. These patterns include, but are not limited to:

[0121] a) The actual frame rate is 60fps, but if SD loop appears in the identifier arrangement, the actual frame rate is 30fps.

[0122] b) The actual frame rate is 30fps. If the DDDDDS loop appears in the identifier arrangement, the actual frame rate is 25fps.

[0123] c) The actual frame rate is 30fps. If the DDDDS loop appears in the identifier arrangement, the actual frame rate is 24fps.

[0124] For frames after the Nth frame, if the same identification cycle is maintained, if the current frame is an S-frame, skip this frame and proceed to the judgment of the next frame; if the current frame is a D-frame, use the current frame and the previous D-frame to apply the frame interpolation method described in this disclosure.

[0125] Figure 9 The diagram shown is a structural schematic of the frame interpolation device described in an embodiment of this disclosure, as follows: Figure 9 As shown, the frame interpolation device 100 includes an input parameter module 110, a frame interpolation configuration module 120, and a frame interpolation processing module 130.

[0126] The input parameter module 110 is configured to acquire the input source, the input frame rate of the input source, and the output frame rate.

[0127] The frame interpolation configuration module 120 is configured to obtain corresponding frame interpolation configuration information based on the input frame rate, the output frame rate, and the frame interpolation configuration. The frame interpolation configuration information is associated with frame interpolation that implements frame rate conversion from the input frame rate to the output frame rate.

[0128] The frame interpolation processing module 130 is configured to perform frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source with the output frame rate.

[0129] It should be noted that the actions or steps performed by the modules 110 to 130 included in the frame interpolation device 100 can correspond one-to-one with the steps in the frame interpolation methods in the embodiments described above, and will not be elaborated here. Figure 10 The diagram shown is a structural schematic of the frame interpolation processing module described in an embodiment of this disclosure. Figure 10 As shown, the frame interpolation processing module 130 includes a first processing unit 131, a second processing unit 132, and a third processing unit 133.

[0130] In some embodiments, the frame interpolation module 130 is configured to obtain the foreground region of the Nth frame (the current frame) through a first processing unit 131 and perform protection processing on the foreground region, perform downsampling processing on the Nth frame through a second processing unit 132, and simultaneously obtain inter-frame motion information of the (N-1)th frame and the Nth frame through a third processing unit 133. Furthermore, the frame interpolation module 130 is configured to obtain inter-frame motion information of the (N-2)th frame and the (N-1)th frame through the third processing unit 133, and simultaneously perform motion compensation on the (N-2)th frame and the (N-1)th frame using the inter-frame motion information of the (N-2)th frame and the (N-1)th frame through the second processing unit 132, where N is an integer greater than 2.

[0131] In some possible implementations, the frame interpolation device 100 may be included in a chip. In other possible implementations, the frame interpolation device 100 may include a chip. In some embodiments, the first processing unit 131 is a CPU, the second processing unit 132 is a GPU, and the third processing unit 133 is an NPU. The execution of each frame can be divided into two parts: the Nth frame and the (N-1)th frame, and the (N-1)th frame and the (N-2)th frame. The two parts are executed asynchronously in parallel, each calling different hardware resources to maximize utilization.

[0132] In some possible implementations, the first processing unit 131 acquires the foreground region of the Nth frame (the current frame) and performs protection processing on the foreground region. The second processing unit downsamples the Nth frame, and the third processing unit acquires the inter-frame motion information of the (N-1)th and Nth frames. This can be implemented on the CPU using image detection networks such as NCNN or MNN. Alternatively, the GPU can downsample the input data of the Nth frame by 1 / 2 and 1 / 4, respectively. Inter-frame motion information is then acquired using a first-stage convolutional neural network corresponding to the 1 / 4 downsampled Nth and (N-1)th frames, and a second-stage convolutional neural network corresponding to the 1 / 2 downsampled Nth and (N-1)th frames, respectively.

[0133] In some possible implementations, the inter-frame motion information of frames N-2 and N-1 is obtained by the third processing unit 133, and the second processing unit uses the inter-frame motion information of frames N-2 and N-1 to perform motion compensation on frames N-2 and N-1, where N is an integer greater than 2. Alternatively, the NPU is directly invoked to obtain the inter-frame motion information from the third-stage convolutional neural network corresponding to frames N and N-1, and the GPU is invoked to perform motion compensation on frames N-2 and N-1 using the inter-frame motion information of frames N-2 and N-1, where N is an integer greater than 2.

[0134] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0135] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0136] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0137] This disclosure also provides an electronic device. Figure 11 The diagram shown is a structural schematic of the electronic device 300 according to an embodiment of this disclosure. Figure 3 As shown, in this embodiment, the electronic device 300 includes a memory 310 and a processor 320.

[0138] The memory 310 is used to store computer programs; the memory 310 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0139] Specifically, memory 310 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 300 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 310 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0140] The processor 320 is connected to the memory 310 and is used to execute the computer program stored in the memory 310 so that the electronic device 300 performs the frame interpolation method described in any embodiment of the present disclosure.

[0141] In some embodiments, processor 320 may be a general-purpose processor, including a central processing unit (CPU) or a network processor (NP). In other embodiments, processor 320 may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0142] In this embodiment, the electronic device 300 may further include a display 330. The display 330 is communicatively connected to the memory 310 and the processor 320, and is used to display the graphical user interface (GUI) of the frame interpolation method described in this embodiment.

[0143] This disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program implements the frame interpolation method described in any embodiment of this disclosure.

[0144] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0145] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. A frame interpolation method, characterized in that, include: Obtain the input source, the input frame rate of the input source, and the output frame rate; The corresponding interpolation configuration information is obtained based on the input frame rate, the output frame rate, and the interpolation configuration. The interpolation configuration information is associated with the interpolation that realizes the frame rate conversion from the input frame rate to the output frame rate. The preset interpolation network model is called based on the input frame rate and the output frame rate to obtain the multiplier and the number of calls of at least one interpolation network model in the interpolation configuration. The number of calls aims to achieve a uniform distribution of inter-frame interpolation times. Based on the input source, obtain the inter-frame motion information between each frame of the input source; Motion compensation for frame interpolation processing is performed using the inter-frame motion information and the frame interpolation configuration information. as well as The input source is subjected to frame interpolation processing using the frame interpolation configuration information to obtain a target source with the specified output frame rate. The process of obtaining inter-frame motion information between frames of the input source includes: obtaining an interpolation ratio using the interpolation configuration information; obtaining the interpolation time between frames based on the interpolation ratio; and predicting the motion between frames of the input source using a convolutional neural network to obtain the inter-frame motion information at the interpolation time. The method of using a convolutional neural network to predict the motion between frames of the input source to obtain inter-frame motion information at the interpolation time includes: obtaining interpolation frames according to the interpolation configuration information, the interpolation frames including reference frames and non-reference frames; performing downsampling processing on the current frame and the previous frame to obtain multiple layers of current frames and previous frames with different resolutions; inputting the current frame and the previous frame into the current layer convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frames in the current layer; and inputting the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frames, along with the current frame and the previous frame of the next layer at the next resolution, into the next layer convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frames in the next layer.

2. The frame interpolation method according to claim 1, characterized in that, Obtaining the corresponding interpolation configuration information based on the input frame rate, the output frame rate, and the interpolation configuration includes: The interpolation configuration is obtained by analyzing the input frame rate and the output frame rate. Obtain corresponding compensation frame information based on the interpolation configuration, the input frame rate, and the output frame rate; and The interpolation configuration information is obtained using the compensation frame information and the interpolation configuration.

3. The frame interpolation method according to claim 2, characterized in that, Analyzing the input frame rate and the output frame rate to obtain the frame interpolation configuration includes: The corresponding frame interpolation network model and the number of calls are obtained based on the ratio of the output frame rate to the input frame rate; and The frame interpolation configuration is obtained based on the input frame rate, the output frame rate, the frame interpolation network model, and the number of calls.

4. The frame interpolation method according to claim 1, characterized in that, Using the frame interpolation configuration information to perform frame interpolation processing on the input source to obtain a target source with the output frame rate includes: The current frame and the next frame of the input source are interpolated according to the interpolation configuration information to obtain a target source with the output frame rate.

5. The frame interpolation method according to claim 1, characterized in that, The current frame and the previous frame are concatenated and input into the current layer of the convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frames, including: The optical flow and mask of the reference frame and the offset and mask of the non-reference frame are obtained using the current layer convolutional neural network. The optical flow of the non-reference frame is obtained based on the optical flow of the reference frame and the offset of the non-reference frame; and The inter-frame motion information of the reference frame is obtained based on the optical flow and mask of the reference frame, and the inter-frame motion information of the non-reference frame is obtained based on the optical flow and offset of the non-reference frame.

6. The frame interpolation method according to claim 1, characterized in that, Also includes: The inter-frame motion information at the current moment is used as input to obtain the inter-frame motion information at the next moment, so as to transmit the temporal domain information of the input source.

7. The frame interpolation method according to claim 1, characterized in that, Also includes: The encoder is used to obtain the motion vectors between each frame of the input source in order to obtain the basic inter-frame motion information of the input source.

8. The frame interpolation method according to claim 1, characterized in that, Also includes: The input source is detected using an image detection network to obtain the foreground region of the input source; The foreground region of the input source is protected based on its location.

9. The frame interpolation method according to claim 1, characterized in that, Also includes: The actual frame rate of the input source video is determined, and the actual frame rate is used as the input frame rate.

10. A frame insertion device, characterized in that, include: The input parameter module is configured to acquire the input source, the input frame rate of the input source, and the output frame rate; The frame interpolation configuration module is configured to call a preset frame interpolation network model according to the input frame rate and the output frame rate to obtain the multiplier and the number of calls of at least one frame interpolation network model in the frame interpolation configuration, and to obtain corresponding frame interpolation configuration information according to the input frame rate, the output frame rate and the frame interpolation configuration, wherein the frame interpolation configuration information is associated with the frame interpolation that realizes the frame rate conversion from the input frame rate to the output frame rate, and the number of calls aims to achieve a uniform distribution of inter-frame interpolation times; as well as The frame interpolation module is configured to obtain inter-frame motion information between frames of the input source based on the input source, perform motion compensation for frame interpolation using the inter-frame motion information and the frame interpolation configuration information, and perform frame interpolation processing on the input source using the frame interpolation configuration information to obtain a target source with the output frame rate. The frame interpolation processing module is configured to: obtain an interpolation ratio using the frame interpolation configuration information; obtain the interpolation time between frames based on the interpolation ratio; and predict the motion between frames of the input source using a convolutional neural network to obtain inter-frame motion information at the interpolation time. The frame interpolation processing module is configured to: obtain interpolated frames according to the frame interpolation configuration information, the interpolated frames including reference frames and non-reference frames; perform downsampling processing on the current frame and the previous frame to obtain multiple layers of current frames and previous frames with different resolutions; input the current frame and the previous frame into the current layer convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame in the current layer; The inter-frame motion information of the reference frame in the current layer, the inter-frame motion information of the non-reference frame, and the current frame and the previous frame of the next layer at the specified resolution are input into the next layer convolutional neural network to output the inter-frame motion information of the reference frame and the inter-frame motion information of the non-reference frame in the next layer.

11. The frame interpolation apparatus according to claim 10, characterized in that, The frame interpolation processing module is configured as follows: The first processing unit obtains the foreground region of the Nth frame, which is the current frame, and performs protection processing on the foreground region. The second processing unit performs downsampling processing on the Nth frame. At the same time, the third processing unit obtains the inter-frame motion information of the (N-1)th frame and the Nth frame. as well as The third processing unit obtains the inter-frame motion information of the (N-2)th frame and the (N-1)th frame, and the second processing unit uses the inter-frame motion information of the (N-2)th frame and the (N-1)th frame to perform motion compensation on the (N-2)th frame and the (N-1)th frame, where N is an integer greater than 2.

12. An electronic device, characterized in that, include: The memory is configured to store executable programs; as well as A processor is configured to execute the program to cause the electronic device to perform the frame interpolation method according to any one of claims 1 to 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the frame interpolation method according to any one of claims 1 to 9.

Images