A video frame interpolation method and apparatus

Abstract

The application relates to a video frame interpolation method and device, and relates to the technical field of media, and can improve the effect of video frame interpolation. The method comprises the following steps: obtaining a first matching cost and a second matching cost according to a first video frame and a second video frame; performing feature extraction on the first video frame to obtain first features of the first video frame; performing feature extraction on the second video frame to obtain first features of the second video frame; obtaining a first optical flow and a second optical flow according to the first matching cost and the first features of the first video frame; obtaining a first target optical flow from the third video frame to the first video frame according to the first optical flow and the second optical flow; obtaining a third optical flow and a fourth optical flow according to the second matching cost and the first features of the second video frame; obtaining a second target optical flow from the third video frame to the second video frame according to the third optical flow and the fourth optical flow; and obtaining the third video frame according to the first video frame, the second video frame, the first target optical flow and the second target optical flow.

Description

Technical Field

[0001] This application relates to the field of media technology, and in particular to a video frame interpolation method and apparatus. Background Technology

[0002] Video frame interpolation is a technique that uses information from adjacent frames to synthesize one or more intermediate frames and inserts these intermediate frames between adjacent frames. Based on this technique, high frame rate videos can be generated, resulting in smoother playback. Video frame interpolation can solve image quality problems such as stuttering and jitter in videos. It can be applied to scenarios such as old film (photo or video) restoration, ultra-high-definition video production, and slow-motion video production.

[0003] Currently, the commonly used video frame interpolation method is based on optical flow. The idea behind optical flow-based video frame interpolation is to predict the optical flow from the intermediate frame to two adjacent frames (also known as two reference frames), and then obtain the intermediate frame based on the optical flow from the two adjacent frames and from the intermediate frame to the two adjacent frames.

[0004] In optical flow-based video interpolation methods, accurate estimation of optical flow is crucial to the interpolation effect. Existing optical flow-based video interpolation methods often produce intermediate frames of poor quality. Summary of the Invention

[0005] This application provides a video frame interpolation method and apparatus, which can synthesize intermediate frames with higher image quality by accurately estimating the optical flow from the intermediate frame to be inserted to two reference frames, thereby improving the effect of video frame interpolation.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a video frame interpolation method for inserting a third video frame between a first video frame and a second video frame. The method includes: obtaining a first matching cost and a second matching cost based on the first video frame and the second video frame; then performing feature extraction on the first video frame to obtain a first feature of the first video frame; and performing feature extraction on the second video frame to obtain a first feature of the second video frame; and obtaining a first optical flow and a second optical flow based on the first matching cost and the first feature of the first video frame, wherein the first optical flow is the optical flow from the first video frame to the second video frame, and the second optical flow is the optical flow from the third video frame to the first video frame. The optical flow of the video frame; based on the first optical flow and the second optical flow, the first target optical flow is obtained, which is the optical flow from the third video frame to the first video frame; then based on the second matching cost and the first feature of the second video frame, the third optical flow and the fourth optical flow are obtained, whereby the third optical flow is the optical flow from the second video frame to the first video frame and the fourth optical flow is the optical flow from the third video frame to the second video frame; based on the third optical flow and the fourth optical flow, the second target optical flow is obtained, whereby the second target optical flow is the optical flow from the third video frame to the second video frame; finally, based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow, the third video frame is obtained.

[0008] In the video frame interpolation method provided in this application embodiment, since the method by which the electronic device obtains the first target optical flow and the second target optical flow can overcome the problems of holes or multiple mappings, the accuracy of the predicted first target optical flow and the second target optical flow is relatively high. Thus, it is possible to synthesize intermediate frames with higher image quality (i.e., the third video frame mentioned above), thereby improving the effect of video frame interpolation.

[0009] In one possible implementation, the method for obtaining the first target optical flow based on the first optical flow and the second optical flow includes: determining a fifth optical flow based on the first optical flow, which is the optical flow from the first video frame to the third video frame, based on the linear motion characteristics of the optical flow; and performing reverse distortion on the fifth optical flow using the second optical flow to obtain a first reverse-distorted optical flow and a first optical flow mask, the first optical flow mask being used to indicate whether the optical flow value in the fifth optical flow has been reverse-distorted; and updating the second optical flow based on the first optical flow mask and the first reverse-distorted optical flow to obtain the first target optical flow.

[0010] It should be understood that the motion of optical flow is usually linear. Based on the linear motion characteristics of optical flow, the optical flow from the first video frame to the second video frame (i.e., the first optical flow F) is calculated. 0→1 It is possible to estimate the optical flow from the first video frame to the third video frame (i.e., the fifth optical flow F). 0→t First, the fifth optical flow can be determined using the following expression:

[0011] F 0→t =t×F 0→1

[0012] Among them, F 0→t For the fifth optical flow, F 0→1 Let t represent the first optical flow and t represent the insertion time of the third video frame.

[0013] Secondly, the fifth optical flow can be reverse-distorted using the following expression:

[0014]

[0015] Among them, F″ t→0 V1 represents the first reverse twisted optical flow, and F represents the first optical flow mask. 0→t For the fifth optical flow, F′ t→0 For the second optical flow, This indicates a reverse twist operation. Detailed procedures for the reverse twist operation can be found in existing techniques in this field, and will not be elaborated further in the embodiments of this application.

[0016] It should be understood that the first optical flow mask V1 is an optical flow mask matrix composed of elements 0 and 1. 0 indicates that the pixel at the corresponding position is not covered during the reverse warp process (the uncovered position is usually an edge or occlusion area), that is, the optical flow value at the corresponding position is not reverse warped during the reverse warp process of the fifth optical flow, i.e., the optical flow value does not change (not updated); 1 indicates that the pixel at the corresponding position is covered during the reverse warp process, that is, the optical flow value at the corresponding position is reverse warped during the reverse warp process of the fifth optical flow, i.e., the optical flow value has changed (updated).

[0017] Finally, the optical flow of the first target can be determined using the following expression:

[0018] F t→0 =F″ t→0 ⊙V1+F′ t→0 ⊙(1-V1)

[0019] Among them, F t→0 F″ represents the optical flow of the first target (the optical flow from the third video frame to the first video frame). t→0 V1 represents the first reverse twisted optical flow, and F′ represents the first optical flow mask. t→0 For the second optical flow, ⊙ represents element-wise multiplication at corresponding positions.

[0020] The meaning of the above expression for determining the first target optical flow is: for the position where the optical flow value is reverse-distorted during the reverse distortion process, the first reverse-distorted optical flow F″ is used. t→0 The optical flow value at this position is used as the optical flow value at the corresponding position of the first target optical flow. For positions where the optical flow value is not reverse-distorted during the reverse distortion process, the second optical flow F′ is used. t→0The optical flow value at that position is taken as the optical flow value at the corresponding position of the first target optical flow.

[0021] In one possible implementation, the method for obtaining the second target optical flow based on the third and fourth optical flows includes: determining a sixth optical flow based on the third optical flow, which is the optical flow from the second video frame to the third video frame, based on the linear motion characteristics of the optical flow; and inversely twisting the sixth optical flow using the fourth optical flow to obtain a second inversely twisted optical flow and a second optical flow mask, the second optical flow mask being used to indicate whether the optical flow value in the sixth optical flow has been inversely twisted; and updating the fourth optical flow based on the second optical flow mask and the second inversely twisted optical flow to obtain the second target optical flow.

[0022] Based on the linear motion characteristics of optical flow, according to the optical flow from the second video frame to the first video frame (i.e., the first optical flow F) 1→0 It is possible to estimate the optical flow from the second video frame to the third video frame (i.e., the sixth optical flow F). 1→t First, the sixth optical flow can be determined using the following expression:

[0023] F 1→t =t×F 1→0

[0024] Among them, F 1→t For the sixth optical flow, F 1→0 The third optical flow is denoted by t, which represents the insertion time of the third video frame.

[0025] Secondly, the sixth optical flow can be reverse-torsional using the following expression:

[0026]

[0027] Among them, F″ t→1 V2 represents the second reverse twisted optical flow, and F represents the second optical flow mask. 1→t For the sixth optical flow, F′ t→1 It is the fourth optical flow. This indicates a reverse twist operation.

[0028] Finally, the optical flow of the second target can be determined using the following expression:

[0029] F t→1 =F″ t→1 ⊙V2+F′ t→1 ⊙(1-V2)

[0030] Among them, F t→1 Indicates the second target optical flow (optical flow from the third video frame to the second video frame), F″ t→1 V2 represents the second reverse twisted optical flow, and F′ represents the first optical flow mask. t→1 This represents the fourth optical flow, and ⊙ indicates element-wise multiplication at corresponding positions.

[0031] The above expression for determining the second target optical flow means that, for the position where the optical flow value is reverse-distorted during the reverse distortion process, the second reverse-distorted optical flow F″ is used. t→1 The optical flow value at this position is used as the optical flow value at the corresponding position of the second target optical flow. For positions where the optical flow value is not reverse-distorted during the reverse distortion process, the fourth optical flow F′ is used. t→1 The optical flow value at that position is used as the optical flow value at the corresponding position of the second target optical flow.

[0032] In one possible implementation, the first matching cost is the matching cost of feature matching from the first video frame to the second video frame, and the second matching cost is the matching cost of feature matching from the second video frame to the first video frame.

[0033] It should be understood that the matching cost between video frames is the matching cost (or similarity) between a feature vector in the feature map (composed of feature vectors) of one video frame and a possible corresponding feature vector in the feature map of another video frame; it can also be called the stereo matching cost. Optionally, different methods can be used to obtain the first and second matching costs mentioned above. Commonly used methods for calculating stereo matching costs include local matching algorithms based on region constraints and optimization algorithms based on global constraints. Local matching algorithms based on region constraints include, but are not limited to, cost aggregation algorithms for matching windows (Square Difference (SSD), Absolute Difference (SAD), Normalization (NCC), etc.), feature point matching algorithms, phase matching matching algorithms, etc.; optimization algorithms based on global constraints include, but are not limited to, graph cut (GC) algorithms, artificial intelligence algorithms (e.g., neural network algorithms, genetic algorithms), belief propagation (BP) algorithms, dynamic programming (DP) algorithms, etc.

[0034] In one possible implementation, the method for obtaining the first matching cost and the second matching cost based on the first video frame and the second video frame includes: performing feature extraction on the first video frame to obtain a second feature of the first video frame; and performing feature extraction on the second video frame to obtain a second feature of the second video frame; and determining the first matching cost and the second matching cost based on the second feature of the first video frame and the second feature of the second video frame.

[0035] The process of obtaining the first matching cost includes: obtaining the correlation between the feature vector of the second feature of the first video frame and the possible corresponding feature vector in the second feature of the second video frame to obtain the first matching cost. The possible corresponding feature vector in the second feature of the second video frame can be determined based on the optical flow from the initial first video frame to the second video frame. It should be understood that a certain feature in the second feature of the first video frame can correspond to at least one feature vector in the second feature of the second video frame.

[0036] The process of obtaining the second matching cost includes: obtaining the correlation between the feature vector of the second feature of the second video frame and the feature vector that may correspond to the second feature point of the first video frame, and obtaining the second matching cost. A certain feature in the second feature of the second video frame can correspond to at least one feature vector in the second feature of the first video frame.

[0037] Optionally, the first feature and the second feature of the video frame can be different features or the same feature, and this application embodiment does not limit this.

[0038] In one possible implementation, the first target optical flow and the second target optical flow are obtained by a coupled optical flow network, which is trained based on a first training sample set, which includes a first sample video frame, a second sample video frame, optical flow from the first sample video frame to the second sample video frame, optical flow from the second sample video frame to the first sample video frame, optical flow from the third sample video frame to the first sample video frame, and optical flow from the third sample video frame to the second sample video frame.

[0039] In this embodiment, the coupled optical flow network can be trained based on artificial intelligence technology (machine learning, deep learning). The coupled optical flow network can be trained on the server (e.g., in the cloud). When the electronic device executes the video frame interpolation method of this embodiment, the electronic device obtains the model of the coupled optical flow network from the server and predicts the first target optical flow and the second target optical flow based on the model. The coupled optical flow network can also be trained online by the electronic device. This embodiment does not limit this.

[0040] Optionally, the method for obtaining training samples in the first training sample set is as follows: Optical flow from the first sample video frame to the second sample video frame, from the second sample video frame to the first sample video frame, from the third sample video frame to the first sample video frame, and from the third sample video frame to the second sample video frame are generated based on the pre-trained RAFT optical flow model. Since the optical flow obtained according to the RAFT optical flow model may not be accurate, this first training sample set can be called a training sample set with pseudo-optical flow labels.

[0041] Optionally, the training sample set used for training the coupled optical flow network may further include a training sample set with accurate optical flow labels. The training sample set with accurate optical flow labels includes a first sample video frame, a second sample video frame, the optical flow from the first sample video frame to the second sample video frame, and the optical flow from the second sample video frame to the first sample video frame. The training sample set does not include a third sample video frame, the optical flow from the third sample video frame to the first sample video frame, and the optical flow from the third sample video frame to the second sample video frame.

[0042] In this embodiment of the application, a coupled optical flow network can be trained based on a training sample set with pseudo optical flow labels and a training sample set with accurate optical flow labels. During the training process of the coupled optical flow network, the two training sample sets can be combined into one training sample set.

[0043] In one possible implementation, the third video frame is acquired by a video interpolation network trained on a second training sample set. This second training sample set includes the first sample video frame, the second sample video frame, the optical flow from the third sample video frame (output from the coupled optical flow network) to the first sample video frame, the optical flow from the third sample video frame (output from the coupled optical flow network) to the second sample video frame, and the third sample video frame itself. The video interpolation network can be a convolutional neural network such as UNet, or other network structures suitable for image processing, such as adversarial neural networks.

[0044] Optionally, during the process of updating the video interpolation network based on the predicted third sample video frame and the actual third sample video frame, the trained coupled optical flow network can also be fine-tuned based on the predicted third sample video frame and the actual third sample video frame.

[0045] Secondly, embodiments of this application provide a video frame interpolation device for inserting a third video frame between a first video frame and a second video frame. The device includes a matching cost calculation module, a first feature extraction module, an optical flow estimation module, and a video frame interpolation module. The matching cost calculation module calculates a first matching cost and a second matching cost based on the first and second video frames. The first feature extraction module extracts features from the first video frame to obtain a first feature of the first video frame and extracts features from the second video frame to obtain a first feature of the second video frame. The optical flow estimation module calculates a first optical flow and a second optical flow based on the first matching cost and the first feature of the first video frame. The first optical flow is the optical flow from the first video frame to the second video frame, and the second optical flow is the optical flow from the third video frame to the first video frame. Based on the first and second optical flows, a third video frame is inserted between the first and second video frames. The optical flow estimation module is further configured to obtain a third optical flow and a fourth optical flow based on a second matching cost and a first feature of a second video frame. The third optical flow is the optical flow from the second video frame to the first video frame, and the fourth optical flow is the optical flow from the third video frame to the second video frame. The second target optical flow is obtained based on the third optical flow and the fourth optical flow. The second target optical flow is the optical flow from the third video frame to the second video frame. The video frame interpolation module is configured to obtain a third video frame based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow.

[0046] In one possible implementation, the aforementioned optical flow estimation module is specifically used to determine a fifth optical flow based on the linear motion characteristics of the optical flow, which is the optical flow from the first video frame to the third video frame; and to reverse-distort the fifth optical flow using a second optical flow to obtain a first reverse-distorted optical flow and a first optical flow mask, the first optical flow mask being used to indicate whether the optical flow value in the fifth optical flow has been reverse-distorted; and to update the second optical flow based on the first optical flow mask and the first reverse-distorted optical flow to obtain a first target optical flow.

[0047] In one possible implementation, the aforementioned optical flow estimation module is specifically used to determine a sixth optical flow based on the linear motion characteristics of the optical flow, which is the optical flow from the second video frame to the third video frame; and to reverse-distort the sixth optical flow using a fourth optical flow to obtain a second reverse-distorted optical flow and a second optical flow mask, the second optical flow mask being used to indicate whether the optical flow value in the sixth optical flow has been reverse-distorted; and to update the fourth optical flow based on the second optical flow mask and the second reverse-distorted optical flow to obtain a second target optical flow.

[0048] In one possible implementation, the first matching cost is the matching cost of feature matching from the first video frame to the second video frame, and the second matching cost is the matching cost of feature matching from the second video frame to the first video frame.

[0049] In one possible implementation, the video frame interpolation device provided in this application embodiment further includes a second feature extraction module; the second feature extraction module is used to extract features from the first video frame to obtain a second feature of the first video frame; and to extract features from the second video frame to obtain a second feature of the second video frame; the matching cost calculation module is specifically used to determine a first matching cost and a second matching cost based on the second feature of the first video frame and the second feature of the second video frame.

[0050] In one possible implementation, the optical flow estimation module is a coupled optical flow network, which is used to obtain the first target optical flow and the second target optical flow based on the first video frame and the second video frame. This coupled optical flow network is trained based on a first training sample set, which includes the first sample video frame, the second sample video frame, the optical flow from the first sample video frame to the second sample video frame, the optical flow from the second sample video frame to the first sample video frame, the optical flow from the third sample video frame to the first sample video frame, and the optical flow from the third sample video frame to the second sample video frame.

[0051] In one possible implementation, the aforementioned video frame interpolation module is a video frame interpolation network used to obtain a third video frame based on the first target optical flow and the second target optical flow. This video frame interpolation network is trained based on a second training sample set, which includes the first sample video frame, the second sample video frame, the optical flow from the third sample video frame output by the coupled optical flow network to the first sample video frame, the optical flow from the third sample video frame output by the coupled optical flow network to the second sample video frame, and the third sample video frame.

[0052] Thirdly, embodiments of this application provide an electronic device, including a memory and at least one processor connected to the memory. The memory is used to store instructions, which, after being read by the at least one processor, execute the method described in the first aspect and any of its possible implementations.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the method described in the first aspect and any of its possible implementations.

[0054] Fifthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a computer, execute the method described in the first aspect and any of its possible implementations.

[0055] Sixthly, embodiments of this application provide a chip including a communication interface and a processing circuit. The communication interface is used for communicating with other devices or components. The processing circuit is used to execute computer instructions to perform the method described in the first aspect and any of its possible implementations.

[0056] It should be understood that the beneficial effects achieved by the second to sixth aspects of the technical solutions and the corresponding possible implementations of the embodiments of this application can be referred to the above-described technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0057] Figure 1 A schematic diagram of a video frame interpolation process based on optical flow is provided for an embodiment of this application;

[0058] Figure 2 A hardware schematic diagram of a mobile phone provided for an embodiment of this application;

[0059] Figure 3 This is a schematic diagram illustrating an application scenario of a video frame interpolation method provided in an embodiment of this application.

[0060] Figure 4 This is a schematic diagram of a video frame interpolation method provided in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of another video frame interpolation method provided in an embodiment of this application;

[0062] Figure 6 This is a schematic diagram of another video frame interpolation method provided in an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of another video frame interpolation method provided in an embodiment of this application;

[0064] Figure 8 A schematic diagram illustrating a video frame interpolation process based on a video interpolation network, provided in an embodiment of this application;

[0065] Figure 9 This is a schematic diagram illustrating the effect of a video frame interpolation method provided in an embodiment of this application;

[0066] Figure 10 This is a schematic diagram of a coupled optical flow network provided in an embodiment of this application;

[0067] Figure 11 This is a schematic diagram of the structure of a video frame interpolation device provided in an embodiment of this application;

[0068] Figure 12 This is a schematic diagram of another video frame interpolation device provided in an embodiment of this application. Detailed Implementation

[0069] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0070] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first video frame" and "second video frame," etc., are used to distinguish different video frames, not to describe a specific order of video frames.

[0071] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0072] First, some concepts involved in the video frame interpolation method and apparatus provided in the embodiments of this application will be explained.

[0073] Video frame interpolation: A technique that uses information from adjacent frames to synthesize one or more intermediate frames and inserts the synthesized intermediate frames between adjacent frames. Based on video frame interpolation technology, high frame rate videos can be generated, making video playback smoother. Video frame interpolation can solve image quality problems such as stuttering and jitter in videos.

[0074] Video frame interpolation can be applied to scenarios such as old film (photo or video) restoration, ultra-high-definition video production, and slow-motion video production. For example, a video with a frame rate of 30 frames per second (f / s) can be doubled and then played at a frame rate of 30 f / s to achieve a slow-motion playback effect without stuttering.

[0075] Optical flow refers to the displacement of pixels representing the same object (or entity) from one frame to the next in a video frame sequence. It should be understood that, given the optical flow from one video frame to another, the optical flow between video frames can be used to estimate the optical flow of one of the video frames.

[0076] For example, given video frame I0, and to estimate video frame I1, if the optical flow from video frame I0 to video frame I1 is calculated (denoted as F... 0→1 ), can be used for F 0→1 Video frame I0 is forward-distorted to obtain video frame I1; or, the optical flow from video frame I1 to video frame I0 (denoted as F) is calculated. 1→0 ), can be used for F 1→0 Video frame I0 is twisted in reverse to obtain video frame I1.

[0077] Forward warping refers to mapping each pixel in the source image (e.g., I0 above) to an appropriate position in the target image (e.g., I1 above). Specifically, it involves mapping each pixel in the source image to an appropriate position in the target image based on the optical flow from the source image to the target image. Forward warping can also be called forward mapping. It should be noted that forward warping may have issues with holes or multiple mappings. Holes occur when some pixels in the target image are not mapped, meaning that after forward warping, some pixel values ​​in the target image are missing. Multiple mapping occurs when some pixels in the target image are mapped multiple times.

[0078] Inverse warping involves iterating through every pixel in the target image and sampling pixels at appropriate locations in the source image to create pixels for the target image. Inverse mapping avoids the problems of holes or multiple mappings mentioned above, ensuring that every pixel in the target image can be mapped to a pixel in the source image.

[0079] The video frame interpolation method provided in this application is an optical flow-based method for interpolating video frames, inserting video frame I0 between adjacent video frames I0 and I1.t For example, refer to Figure 1 The process of video frame interpolation based on optical flow includes: First, predicting I t Optical flow F to I0 t→0 , and I t Optical flow F to I1 t→1 Secondly, according to optical flow F t→0 Inverse warping of I0 yields an intermediate frame, denoted as I. t ', and according to optical flow F t→1 Inversely warping I1 yields an intermediate frame, denoted as I. t Finally, the estimated I t 'and I t "Perform weighted fusion to obtain the intermediate frame I to be inserted." t .

[0080] In the aforementioned video frame interpolation process, accurately predicting the optical flow impact on the synthesized intermediate frame from the intermediate frame to the two reference frames still allows for the insertion of video frame I between adjacent video frames I0 and I1. t For example, one existing method for predicting the optical flow from the intermediate frame to two reference frames is to first calculate the bidirectional optical flow between the two reference frames, that is, estimate the optical flow F from I0 to I1. 0→1 and the optical flow F from I1 to I0 1→0 Then, according to F 0→1 and F 1→0 , to obtain F t→0 and F t→1 For example, by using a positive twisting method, for F 0→1 Perform a positive twist operation (a type of linear interpolation) to obtain F t→0 , for F 1→0 F is obtained by performing a positive twist operation. t→1 However, due to the presence of holes or multiple mappings in the forward distortion, the accuracy of the predicted optical flow from the intermediate frame to the two reference frames is low, resulting in poor quality of the synthesized intermediate frame.

[0081] To address the problems existing in the prior art, embodiments of this application provide a video frame interpolation method. The device for video frame interpolation obtains a first matching cost and a second matching cost based on a first video frame and a second video frame. Then, based on the first matching cost and a first feature of the first video frame, it obtains a first optical flow and a second optical flow, where the first optical flow is the optical flow from the first video frame to the second video frame, and the second optical flow is the optical flow from the third video frame to the first video frame. Furthermore, based on the first and second optical flows, it obtains a first target optical flow, where the first target optical flow is the optical flow from the third video frame to the first video frame. The device for video frame interpolation also obtains a third optical flow and a fourth optical flow based on the second matching cost and the first feature of the second video frame, where the third optical flow is the optical flow from the second video frame to the first video frame, and the fourth optical flow is the optical flow from the third video frame to the second video frame. Furthermore, based on the third and fourth optical flows, it obtains a second target optical flow, where the second target optical flow is the optical flow from the third video frame to the second video frame. Finally, based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow, it obtains a third video frame. The technical solution provided by the embodiments of this application can synthesize intermediate frames with high image quality by accurately estimating the optical flow from the intermediate frame to be inserted to two reference frames, thereby improving the effect of video frame interpolation.

[0082] The video frame interpolation method provided in this application can be applied to electronic devices such as mobile phones, tablets, or ultra-mobile personal computers (UMPCs). Alternatively, it can also be applied to other electronic devices such as desktop devices, laptops, handheld devices, wearable devices, smart home devices, and in-vehicle devices, including netbooks, smartwatches, smart cameras, personal digital assistants (PDAs), portable multimedia players (PMPs), and dedicated media players. This application does not limit the specific type and structure of the electronic device.

[0083] Taking mobile phones as an example, Figure 2This is a schematic diagram of the hardware structure of a mobile phone 200 provided in an embodiment of this application. The mobile phone 200 includes a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an accelerometer sensor 280E, a distance sensor 280F, a proximity sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0084] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 200. In other embodiments of this application, the mobile phone 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0085] Processor 210 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0086] The controller can serve as the central nervous system and command center of the mobile phone 200. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0087] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0088] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0089] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple I2C buses. The processor 210 can couple to the touch sensor 280K, charger, flash, camera 293, etc., through different I2C bus interfaces. For example, the processor 210 can couple to the touch sensor 280K through the I2C interface, enabling the processor 210 and the touch sensor 280K to communicate through the I2C bus interface, thus realizing the touch function of the mobile phone 200.

[0090] The I2S interface can be used for audio communication. In some embodiments, the processor 210 may include multiple I2S buses. The processor 210 can be coupled to the audio module 270 via the I2S bus to enable communication between the processor 210 and the audio module 270. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0091] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 270 and the wireless communication module 260 can be coupled via the PCM bus interface. In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0092] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 210 and the wireless communication module 260. For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 via the UART interface to enable music playback through Bluetooth headphones.

[0093] The MIPI interface can be used to connect the processor 210 to peripheral devices such as the display screen 294 and the camera 293. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 210 and the camera 293 communicate via the CSI interface to enable the mobile phone 200 to take pictures. The processor 210 and the display screen 294 communicate via the DSI interface to enable the mobile phone 200 to display.

[0094] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 to a camera 293, a display screen 294, a wireless communication module 260, an audio module 270, a sensor module 280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0095] USB port 230 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, or USB Type-C port. USB port 230 can be used to connect a charger to charge mobile phone 200, and can also be used for data transfer between mobile phone 200 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0096] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the mobile phone 200. In other embodiments of this application, the mobile phone 200 may also adopt different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0097] The charging management module 240 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 receives charging input from the wired charger via a USB interface 230. In some wireless charging embodiments, the charging management module 240 receives wireless charging input via the wireless charging coil of the mobile phone 200. While charging the battery 242, the charging management module 240 can also supply power to the electronic device via the power management module 241.

[0098] The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, external memory, display screen 294, camera 293, and wireless communication module 260. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 241 may also be located within the processor 210. In other embodiments, the power management module 241 and the charging management module 240 may be housed in the same device.

[0099] The wireless communication function of mobile phone 200 can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0100] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 200 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0101] The mobile communication module 250 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on the mobile phone 200. The mobile communication module 250 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some modules of the processor 210 may be housed in the same device.

[0102] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 270A, receiver 270B, etc.) or displays images or videos through the display screen 294. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 210 and may be housed in the same device as the mobile communication module 250 or other functional modules.

[0103] The wireless communication module 260 can provide solutions for wireless communication applications on the mobile phone 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0104] In some embodiments, antenna 1 of mobile phone 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling mobile phone 200 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0105] The mobile phone 200 implements its display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0106] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, mobile phone 200 may include one or N displays 294, where N is a positive integer greater than 1.

[0107] The mobile phone 200 can achieve shooting functions through ISP, camera 293, video codec, GPU, display 294 and application processor.

[0108] The ISP (Image Signal Processor) is used to process data fed back from the camera 293. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0109] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, mobile phone 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0110] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals (such as audio signals). For example, when the mobile phone 200 is selecting a frequency, the DSP performs Fourier transforms on the frequency energy.

[0111] Video codecs are used to compress or decompress digital video. Mobile phone 200 can support one or more video codecs. Thus, mobile phone 200 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0112] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in mobile phones, such as image recognition, facial recognition, speech recognition, and text understanding.

[0113] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 200. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0114] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of mobile phone 200 by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of mobile phone 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0115] The mobile phone 200 can perform audio functions, such as music playback and recording, through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, and an application processor.

[0116] The audio module 270 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0117] The speaker 270A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The mobile phone 200 can listen to music or make hands-free calls through the speaker 270A.

[0118] The receiver 270B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the mobile phone 200 answers a call or voice message, the receiver 270B can be brought close to the user's ear to listen to the voice.

[0119] Microphone 270C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 270C, inputting the sound signal into microphone 270C. Mobile phone 200 may have at least one microphone 270C. In some embodiments, mobile phone 200 may have two microphones 270C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, mobile phone 200 may also have three, four, or more microphones 270C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0120] The headphone jack 270D is used to connect wired headphones. The headphone jack 270D can be a USB 230 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0121] Pressure sensor 280A is used to sense pressure signals and convert them into electrical signals. Gyroscope sensor 280B is used to determine the motion posture of mobile phone 200. Barometric pressure sensor 280C is used to measure air pressure. Magnetism sensor 280D includes a Hall sensor. Accelerometer sensor 280E can detect the magnitude of acceleration of mobile phone 200 in various directions (generally three axes). Proximity sensor 280G can include, for example, a light-emitting diode (LED) and a photodetector, such as a photodiode. The LED can be an infrared LED. Ambient light sensor 280L is used to sense ambient light brightness. Fingerprint sensor 280H is used to collect fingerprints. Temperature sensor 280J is used to detect temperature. Touch sensor 280K, also known as a "touch panel," can be set on display screen 294. Touch sensor 280K and display screen 294 together form a touch screen, also known as a "touchscreen." Touch sensor 280K is used to detect touch operations applied to or near it. Bone conduction sensor 280M can acquire vibration signals. In some embodiments, the bone conduction sensor 280M can acquire vibration signals from the vibrating bone fragments of the human vocal cords.

[0122] Keypad 290 includes a power button, volume buttons, etc. Keypad 290 can be a mechanical keypad or a touch-sensitive keypad. Mobile phone 200 can receive keypad input and generate key signal inputs related to user settings and function control of mobile phone 200.

[0123] Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 291 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 294. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0124] Indicator 292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0125] The SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to make contact with and separate from the mobile phone 200. The mobile phone 200 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 295 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 295 is also compatible with different types of SIM cards. The SIM card interface 295 is also compatible with external memory cards. The mobile phone 200 interacts with the network through the SIM card to achieve functions such as calls and data communication. In some embodiments, the mobile phone 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone 200 and cannot be separated from the mobile phone 200.

[0126] It is understood that in the embodiments of this application, the electronic device (e.g., the mobile phone 200 described above) can execute some or all of the steps in the embodiments of this application. These steps or operations are merely examples, and the electronic device can also perform other operations or variations thereof. Furthermore, the steps can be executed in different orders as presented in the embodiments of this application, and it is not necessary to execute all the operations in the embodiments of this application. The embodiments of this application can be implemented individually or in any combination, and this application does not limit this.

[0127] The video frame interpolation method provided in this application embodiment can be applied to conditions such as... Figure 2 The illustrated hardware structure is applicable to electronic devices or electronic devices with a similar structure. Alternatively, it can be applied to electronic devices with other structures, but this application does not limit the scope of the embodiments.

[0128] Optionally, the video frame interpolation method provided in this application embodiment can be applied in the following situations, one of which is: referring to Figure 3 For a low frame rate video involving multiple scene scenarios (such as indoor scenes, outdoor scenes, etc.), the video can first be divided (segmented) into different video segments according to different scenes, such as the video segment corresponding to scene one, the video segment corresponding to scene two, and the video segment corresponding to scene three. Then, the video frame interpolation method provided in the embodiments of this application is applied to the video segments of different scenes to obtain high frame rate videos corresponding to the video segments of different scenes. Finally, the high frame rate videos corresponding to the video segments of different scenes are merged according to the scene order to obtain a complete high frame rate video.

[0129] Another scenario is: for low frame rate videos in a single scene, the video frame interpolation method provided in the embodiments of this application is directly executed to obtain high frame rate videos.

[0130] like Figure 4 As shown in the figure, this application provides a video frame interpolation method, which can be applied to electronic devices. The method is used to insert a third video frame between a first video frame and a second video frame, and includes steps 401 to 408.

[0131] Step 401: The electronic device obtains the first matching cost and the second matching cost based on the first video frame and the second video frame.

[0132] The first matching cost is the cost of feature matching from the first video frame to the second video frame, and the second matching cost is the cost of feature matching from the second video frame to the first video frame. It should be noted that the calculation of the matching cost is directional; that is, the first matching cost is from the first video frame to the second video frame, and the second matching cost is from the second video frame to the first video frame. The first and second matching costs are different.

[0133] The matching cost between video frames is the matching cost (or similarity) between a feature vector in the feature map (composed of feature vectors) of one video frame and a possible corresponding feature vector in the feature map of another video frame; it can also be called the stereo matching cost. Optionally, different methods can be used to obtain the first and second matching costs mentioned above. Common methods for calculating stereo matching costs include local matching algorithms based on region constraints and optimization algorithms based on global constraints. Local matching algorithms based on region constraints include, but are not limited to, cost aggregation algorithms for matching windows (Square Difference (SSD), Absolute Difference (SAD), Normalization (NCC), etc.), feature point matching algorithms, and phase matching algorithms. Optimization algorithms based on global constraints include, but are not limited to, graph cut (GC) algorithms, artificial intelligence algorithms (e.g., neural network algorithms, genetic algorithms), belief propagation (BP) algorithms, and dynamic programming (DP) algorithms.

[0134] In this embodiment of the application, a feature point matching algorithm is used as an example, combined with... Figure 4 ,like Figure 5 As shown, the process of calculating the first matching cost and the second matching cost may include steps 4011 to 4013.

[0135] Step 4011: The electronic device extracts features from the first video frame to obtain the second feature of the first video frame.

[0136] Step 4012: The electronic device extracts features from the second video frame to obtain the second feature of the second video frame.

[0137] Optionally, the second feature of the image can be a feature extracted based on a second feature extractor. The second feature of the image is a feature map composed of feature vectors. In this embodiment, the second feature extractor can be a feature extractor learned based on a deep learning algorithm, and the feature extractor can be a convolutional neural network.

[0138] Step 4013: The electronic device determines the first matching cost and the second matching cost based on the second feature of the first video frame and the second feature of the second video frame.

[0139] Specifically, the process of obtaining the first matching cost includes: obtaining the correlation between the feature vector of the second feature of the first video frame and the feature vector that may correspond to the second feature of the second video frame, to obtain the first matching cost. The feature vector that may correspond to the second feature of the second video frame can be determined based on the optical flow from the initial first video frame to the second video frame. It should be understood that a certain feature in the second feature of the first video frame can correspond to at least one feature vector in the second feature of the second video frame. Similarly, the process of obtaining the second matching cost includes: obtaining the correlation between the feature vector of the second feature of the second video frame and the feature vector that may correspond to the second feature point of the first video frame, to obtain the second matching cost. A certain feature in the second feature of the second video frame can correspond to at least one feature vector in the second feature of the first video frame.

[0140] Step 402: The electronic device extracts features from the first video frame to obtain the first feature of the first video frame.

[0141] Step 403: The electronic device extracts features from the second video frame to obtain the first feature of the second video frame.

[0142] Optionally, the first feature of the image can be a feature extracted based on a first feature extractor. In this embodiment, the first feature extractor is a feature extractor learned based on a deep learning algorithm.

[0143] Optionally, the first feature and the second feature mentioned above can be different features or the same features. That is, the model structure and parameters of the first feature extractor can be different from or the same as the model structure and parameters of the second feature extractor. This application does not limit this.

[0144] Step 404: The electronic device obtains a first optical flow and a second optical flow based on the first matching cost and the first feature of the first video frame. The first optical flow is the optical flow from the first video frame to the second video frame, and the second optical flow is the optical flow from the third video frame to the first video frame.

[0145] In this embodiment of the application, the first video frame is denoted as I0, the second video frame as I1, and the third video frame as I... tLet the first matching cost be R1 and the first feature of the first video frame be T1. The specific process of obtaining the first optical flow and the second optical flow based on the first matching cost and the first feature of the first video frame is as follows: Input the first matching cost R1 and the first feature T1 of the second video frame into the pre-trained optical flow prediction model (also known as an iterator). It should be understood that the input of the optical flow prediction model also includes the initialized state variable H and the initialized optical flow from the first video frame to the second video frame. And the optical flow from the initial third video frame to the first video frame. The first optical flow (denoted as F) is obtained by the optical flow prediction model. 0→1 ) and the second optical flow (denoted as F′) t→0 ).

[0146] Among them, the initial value of the state variable H and the initial value of the optical flow from the first video frame to the second video frame. and the initial value of optical flow from the third video frame to the first video frame. It can be 0 or a random value; this application does not limit it in its embodiments.

[0147] Optionally, the initial value of the aforementioned state variable H can also be the first feature of the first video frame.

[0148] It should be noted that the optical flow prediction model described above can also be a model trained based on a deep learning algorithm. For example, the structure of the optical flow prediction model can be a gated recurrent unit (GRU) structure. The training process of the optical flow prediction model will be described in detail in the following embodiments.

[0149] Step 405: The electronic device obtains the first target optical flow based on the first optical flow and the second optical flow. The first target optical flow is the optical flow from the third video frame to the first video frame.

[0150] Optionally, such as Figure 6 As shown, step 405 can be implemented through the following steps 4051 to 4053.

[0151] Step 4051: Based on the linear motion characteristics of optical flow, the electronic device determines the fifth optical flow according to the first optical flow. The fifth optical flow is the optical flow from the first video frame to the third video frame.

[0152] It should be understood that the motion of optical flow is usually linear. Based on the linear motion characteristics of optical flow, the optical flow from the first video frame to the second video frame (i.e., the first optical flow F) is calculated. 0→1 It is possible to estimate the optical flow from the first video frame to the third video frame (i.e., the fifth optical flow F). 0→t Specifically, the fifth optical flow can be determined using the following expression:

[0153] F 0→t =t×F 0→1

[0154] Among them, F 0→t For the fifth optical flow, F 0→1 Let t represent the first optical flow and t represent the insertion time of the third video frame.

[0155] Step 4052: The electronic device uses the second optical flow to reverse twist the fifth optical flow to obtain a first reverse twisted optical flow and a first optical flow mask. The first optical flow mask is used to indicate whether the optical flow value in the fifth optical flow has been reverse twisted.

[0156] In this embodiment of the application, the fifth optical flow can be reverse-distorted using the following expression:

[0157]

[0158] Among them, F″ t→0 V1 represents the first reverse twisted optical flow, and F represents the first optical flow mask. 0→t For the fifth optical flow, F′ t→0 For the second optical flow, This indicates a reverse twist operation. Detailed procedures for the reverse twist operation can be found in existing techniques in this field, and will not be elaborated further in the embodiments of this application.

[0159] It should be understood that the first optical flow mask V1 is an optical flow mask matrix composed of elements 0 and 1. 0 indicates that the pixel at the corresponding position is not covered during the reverse warp process (the uncovered position is usually an edge or occlusion area), that is, the optical flow value at the corresponding position is not reverse warped during the reverse warp process of the fifth optical flow, i.e., the optical flow value does not change (not updated); 1 indicates that the pixel at the corresponding position is covered during the reverse warp process, that is, the optical flow value at the corresponding position is reverse warped during the reverse warp process of the fifth optical flow, i.e., the optical flow value has changed (updated).

[0160] Step 4053: The electronic device updates the second optical flow based on the first optical flow mask and the first reverse twisted optical flow to obtain the first target optical flow.

[0161] Specifically, the optical flow of the first target can be determined using the following expression:

[0162] F t→0 =F″ t→0 ⊙V1+F′ t→0 ⊙(1-V1)

[0163] Among them, F t→0 F″ represents the optical flow of the first target (the optical flow from the third video frame to the first video frame). t→0V1 represents the first reverse twisted optical flow, and F′ represents the first optical flow mask. t→0 For the second optical flow, ⊙ represents element-wise multiplication at corresponding positions.

[0164] The meaning of the above expression for determining the first target optical flow is: for the position where the optical flow value is reverse-distorted during the reverse distortion process, the first reverse-distorted optical flow F″ is used. t→0 The optical flow value at this position is used as the optical flow value at the corresponding position of the first target optical flow. For positions where the optical flow value is not reverse-distorted during the reverse distortion process, the second optical flow F′ is used. t→0 The optical flow value at that position is taken as the optical flow value at the corresponding position of the first target optical flow.

[0165] In summary, the process of obtaining the first target optical flow based on the first optical flow and the second optical flow described in steps 4051 to 4053 above can be understood as coupling the first optical flow and the second optical flow to obtain the first target optical flow.

[0166] Step 406: The electronic device obtains a third optical flow and a fourth optical flow based on the second matching cost and the first feature of the second video frame. The third optical flow is the optical flow from the second video frame to the first video frame, and the fourth optical flow is the optical flow from the third video frame to the second video frame.

[0167] Similar to step 404 above, the first video frame is denoted as I0, the second video frame as I1, and the third video frame as I... t Let the second matching cost be R2 and the first feature of the second video frame be T2. The specific process of obtaining the third and fourth optical flows based on the second matching cost and the first feature of the second video frame is as follows: Input the second matching cost R2 and the first feature T2 of the second video frame into the pre-trained optical flow prediction model (also known as an iterator). It should be understood that the input of the optical flow prediction model also includes the initialized state variable H and the initialized optical flow from the second video frame to the first video frame. And the optical flow from the initial third video frame to the second video frame. The third optical flow (denoted as F) is obtained through the prediction of the optical flow prediction model. 1→0 ) and the fourth optical flow (denoted as F′) t→1 ).

[0168] Among them, the initial value of the state variable H and the initial value of the optical flow from the second video frame to the first video frame. And the initial value of optical flow from the third video frame to the second video frame. The value can be 0 or a random value. Optionally, the initial value of the state variable H can also be the first feature of the second video frame.

[0169] Step 407: The electronic device obtains the second target optical flow based on the third optical flow and the fourth optical flow. The second target optical flow is the optical flow from the third video frame to the second video frame.

[0170] Optionally, such as Figure 7 As shown, step 407 above can be implemented by steps 4071 to 4073 below.

[0171] Step 4071: Based on the linear motion characteristics of optical flow, the electronic device determines the sixth optical flow according to the third optical flow. The sixth optical flow is the optical flow from the second video frame to the third video frame.

[0172] Based on the linear motion characteristics of optical flow, according to the optical flow from the second video frame to the first video frame (i.e., the first optical flow F) 1→0 It is possible to estimate the optical flow from the second video frame to the third video frame (i.e., the sixth optical flow F). 1→t Specifically, the sixth optical flow can be determined using the following expression:

[0173] F 1→t =t×F 1→0

[0174] Among them, F 1→t For the sixth optical flow, F 1→0 The third optical flow is denoted by t, which represents the insertion time of the third video frame.

[0175] Step 4072: The electronic device uses the fourth optical flow to reverse the sixth optical flow to obtain the second reverse-distorted optical flow and the second optical flow mask. The second optical flow mask is used to indicate whether the optical flow value in the sixth optical flow has been reverse-distorted.

[0176] In this embodiment of the application, the sixth optical flow can be reverse-torsional using the following expression:

[0177]

[0178] Among them, F″ t→1 V2 represents the second reverse twisted optical flow, and F represents the second optical flow mask. 1→t For the sixth optical flow, F′ t→1 It is the fourth optical flow. This indicates a reverse twist operation.

[0179] Similarly, the second optical flow mask V2 is an optical flow mask matrix composed of elements 0 and 1. For a description of the second optical flow mask V2, please refer to the detailed description of the first optical flow mask V1 in the above embodiments, which will not be repeated here.

[0180] Step 4073: The electronic device updates the fourth optical flow based on the second optical flow mask and the second reverse twisted optical flow to obtain the second target optical flow.

[0181] Specifically, the optical flow of the second target can be determined using the following expression:

[0182] F t→1 =F″ t→1 ⊙V2+F′ t→1 ⊙(1-V2)

[0183] Among them, F t→1 Indicates the second target optical flow (optical flow from the third video frame to the second video frame), F″ t→1 V2 represents the second reverse twisted optical flow, and F′ represents the first optical flow mask. t→1 This represents the fourth optical flow, and ⊙ indicates element-wise multiplication at corresponding positions.

[0184] The above expression for determining the second target optical flow means that, for the position where the optical flow value is reverse-distorted during the reverse distortion process, the second reverse-distorted optical flow F″ is used. t→1 The optical flow value at this position is used as the optical flow value at the corresponding position of the second target optical flow. For positions where the optical flow value is not reverse-distorted during the reverse distortion process, the fourth optical flow F′ is used. t→1 The optical flow value at that position is used as the optical flow value at the corresponding position of the second target optical flow.

[0185] Step 408: The electronic device obtains the third video frame based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow.

[0186] In this embodiment, the electronic device can input a first video frame, a second video frame, a first target optical flow, and a second target optical flow into a video frame interpolation network, thereby outputting a third video frame. Optionally, the video frame interpolation network can be a convolutional neural network such as Unet, or other network structures that can be used for image processing, such as adversarial neural networks. The video frame interpolation network can be a pre-trained network, and the training process of the video frame interpolation network will be described in detail in the following embodiments.

[0187] For details, please refer to Figure 8The first video frame, the second video frame, the first target optical flow, and the second target optical flow are input into a video interpolation network. In this network, the first video frame is inversely warped according to the first target optical flow to obtain a first intermediate frame (this first intermediate frame is an intermediate prediction result for the third video frame). Then, the first intermediate frame is adjusted according to the first occlusion map predicted by the video interpolation network to obtain a first prediction value for the third video frame. The first occlusion map indicates information that the third video frame exists but is not present in the first video frame. Similarly, the second video frame is inversely warped according to the second target optical flow to obtain a second intermediate frame (this second intermediate frame is also an intermediate prediction result for the third video frame). Then, the second intermediate frame is adjusted according to the second occlusion map predicted by the video interpolation network to obtain a second prediction value for the third video frame. The second occlusion map indicates information that the third video frame exists but is not present in the second video frame. Finally, the first and second prediction values ​​of the third video frame are weighted and fused. The fused result is then added to the residual frame predicted by the video interpolation network to obtain the synthesized third video frame.

[0188] In the video frame interpolation method provided in this application embodiment, the electronic device can obtain the first optical flow from the first video frame to the second video frame and the second optical flow from the third video frame to the first video frame based on the first feature of the first video frame and the first matching cost of matching from the first video frame to the second video frame. Based on the first and second optical flows, a first target optical flow from the third video frame to the first video frame is obtained. Similarly, the electronic device can obtain the third optical flow from the second video frame to the first video frame and the fourth optical flow from the third video frame to the second video frame based on the first feature of the second video frame and the second matching cost of matching from the second video frame to the first video frame. Based on the third and fourth optical flows, a second target optical flow from the third video frame to the second video frame is obtained. Furthermore, based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow, the third video frame is obtained. Since the method by which the electronic device obtains the first and second target optical flows can overcome the problems of holes or multiple mappings, the accuracy of the predicted first and second target optical flows is relatively high. Thus, a high-quality intermediate frame (i.e., the aforementioned third video frame) can be synthesized, thereby improving the effect of video frame interpolation.

[0189] refer to Figure 9 Through experimental research, the third video frame obtained according to the video frame interpolation method provided in the embodiments of this application ( Figure 9 (b) in the example is compared with existing methods for obtaining the third video frame. Figure 9 In (a) of the image, the problems of distorted lines and disordered textures have been significantly improved.

[0190] It is understood that in the embodiments of this application, the model for video frame interpolation includes a coupled optical flow network, which is used to obtain a first target optical flow and a second target optical flow based on a first video frame and a second video frame. That is, the coupled optical flow network is used to implement steps 401 to 407 in the above embodiments. Figure 10 As shown, the coupled optical flow network may include the first feature extractor for extracting a first feature of an image, the second feature extractor for extracting a second feature of an image, the matching cost calculation module, the optical flow prediction model, and the coupling processing module as described in the above embodiments.

[0191] Optionally, the coupled optical flow network can be trained based on artificial intelligence technology (machine learning, deep learning). The coupled optical flow network can be trained on the server (e.g., in the cloud). When the electronic device executes the video frame interpolation method of this application embodiment, the electronic device obtains the model of the coupled optical flow network from the server and predicts the first target optical flow and the second target optical flow based on the model. The coupled optical flow network can also be trained online by the electronic device. This application embodiment does not limit this.

[0192] It should be noted that, in the embodiments of this application, during the training and updating of the coupled optical flow network, the matching cost calculation module and the coupling processing module in the coupled optical flow network do not need to be updated. What needs to be updated are the first feature extractor, the second feature extractor, and the optical flow prediction model.

[0193] In this embodiment of the application, the coupled optical flow network is trained based on a first training sample set, which includes a first sample video frame, a second sample video frame, optical flow from the first sample video frame to the second sample video frame, optical flow from the second sample video frame to the first sample video frame, optical flow from the third sample video frame to the first sample video frame, and optical flow from the third sample video frame to the second sample video frame.

[0194] Optionally, the method for obtaining training samples in the first training sample set may be: generating optical flow from the first sample video frame to the second sample video frame, optical flow from the second sample video frame to the first sample video frame, optical flow from the third sample video frame to the first sample video frame, and optical flow from the third sample video frame to the second sample video frame based on the pre-trained RAFT (recurrent all-pairs field transforms for optical flow) optical flow model.

[0195] Suppose a training sample set includes the first sample video frame I0, the second sample video frame I1, and the third sample video frame I... tBased on the RAFT optical flow model, and according to the first sample video frame I0 and the second sample video frame I1, the optical flow F from the first sample video frame I0 to the second sample video frame I1 can be obtained. 0→1 Optical flow F from the second sample video frame I1 to the first sample video frame I0 1→0 Based on the first sample video frame and the third sample video frame, we can obtain the first sample video frame I0 to the third sample video frame I0. t Optical Flow F 0→t and the third sample video frame I t Optical flow F to the first sample video frame I0 t→0 Based on the second and third sample video frames, we can obtain the second sample video frame I1 to the third sample video frame I. t Optical Flow F 1→t and the third sample video frame I t Optical flow F to the second sample video frame I1 t→1 .

[0196] After obtaining three sets of optical flows from multiple training samples using the above method, due to potential factors such as occlusion, motion blur, and large motion, this embodiment of the application performs a consistency check on the three sets of optical flows based on the spatial consistency of the bidirectional optical flows (also known as consistency before and after), filtering out (or eliminating) unqualified samples. The consistency of the bidirectional optical flows refers to the fact that the magnitudes of the bidirectional optical flows between two video frames are the same, but their directions are opposite. For example, the aforementioned F... 0→1 With F 1→0 They should be the same size and opposite in direction, i.e., F 1→0 +F 1→0 =0, if F 1→0 +F 1→0 If the sum of the two-way optical flows is not equal to 0, it indicates that the optical flow estimation is inaccurate. Based on this, the above-mentioned screening (or filtering) of unqualified samples specifically means removing samples in the training samples whose sum of the two-way optical flows is not equal to 0.

[0197] After spatial consistency verification, I0, I1, and I2 are selected from the three optical flows of multiple qualified training samples. t F 0→1 F 1→0 F t→0 and F t→1 The coupled optical flow network is trained using these samples. It should be noted that the optical flow obtained according to the RAFT optical flow model may not be the accurate optical flow. Therefore, the training sample set based on the RAFT optical flow model can be called a training sample set with pseudo-optical flow labels.

[0198] It should be understood that the training process of the coupled optical flow network involves continuous optimization of the first feature extractor, the second feature extractor, and the optical flow prediction model within the network. The following details the process of training the coupled optical flow network based on the first training sample set; for specifics, please refer to [reference needed]. Figure 10 The schematic diagram of the coupled optical flow network shown is obtained by repeatedly executing the following steps 1 to 5.

[0199] Step 1: For the i-th training sample, extract the first feature of the first sample video frame I0 using the first feature extractor.

[0200] Step 2: Obtain the first matching cost R1 by using the second feature extractor to extract the second features of the first sample video frame I0 and the second features of the second sample video frame I1.

[0201] Step 3: Combine the first feature and the first matching cost R1 of the first sample video frame I0. as well as The input is given to the optical flow prediction model, which first predicts... as well as And according to and The optical flow from the first sample video frame to the second sample video frame was obtained in this estimation. Optical flow from the third sample video frame to the first sample video frame

[0202] in, as well as It was obtained from the previous iteration (corresponding to the (i-1)th training sample). For this estimate of F 0→1 Compared with the previous estimate The residuals between For this estimate of F t→0 Compared with the previous estimate The residuals between

[0203] It should be understood that in the embodiments of this application, when step 3 is executed for the first time, the state variable H1 and the optical flow F from the first sample video frame to the second sample video frame are... 0→1 The optical flow F from the third sample video frame to the first sample video frame t→0 The value is the initial value, where the state variable H is initialized to... Optical flow F from the first sample video frame to the second sample video frame 0→1 Initialize to And the optical flow F from the third sample video frame to the first sample video frame t→0 Initialize to

[0204] Step 4: Based on the optical flow from the first sample video frame to the second sample video frame Optical flow from the third sample video frame to the first sample video frame Obtain the target optical flow from the third sample video frame to the first sample video frame.

[0205] Specifically, the target optical flow can be obtained from the following three expressions.

[0206]

[0207]

[0208]

[0209] For a detailed description of step 4, please refer to the description of steps 4051 to 4053 in the above embodiments, which will not be repeated here.

[0210] It should be understood that the optical flow from the first sample video frame to the second sample video frame can be obtained through steps 1 to 4 above. Optical flow from the third sample video frame to the first sample video frame

[0211] Step 5: Calculate the optical flow from the first sample video frame to the second sample video frame obtained from this estimation. The optical flow F from the first sample video frame to the second sample video frame 0→1 And the optical flow from the third sample video frame to the first sample video frame obtained in this estimation. The optical flow F from the real third sample video frame to the first sample video frame t→0 Update the coupled optical flow network.

[0212] Specifically, based on the above estimates The real F 0→1 estimated And the real F t→0 Calculate the loss value of the preset loss function, and update the first feature extractor, the second feature extractor, and the optical flow prediction model in the coupled optical flow network according to the loss value.

[0213] In summary, by repeatedly executing steps 1 to 5 until a preset number of times (e.g., 12 times) or until the loss value of the coupled optical flow network reaches a preset condition, a trained coupled optical flow network is obtained.

[0214] Alternatively, steps a through e below can be repeated to obtain a coupled optical flow network.

[0215] Step a: For the i-th training sample, extract the first feature of the second sample video frame I1 using the first feature extractor.

[0216] Step b: Using the second feature extractor, extract the second features of the first sample video frame I0 and the second features of the second sample video frame I1, and obtain the second matching cost R2 based on the second features of the first sample video frame I0 and the second features of the second sample video frame I1.

[0217] Step c: Combine the first feature of the second sample video frame I1 with the second matching cost R2. as well as The input is given to the optical flow prediction model, which first predicts... as well as And according to and The optical flow from the second sample video frame to the first sample video frame was obtained in this estimation. Optical flow from the third sample video frame to the second sample video frame

[0218] in, and It was obtained from the previous iteration (corresponding to the (i-1)th training sample). For this estimate of F 1→0 Compared with the previous estimate The residuals between For this estimate of F t→1 Compared with the previous estimate The residuals between

[0219] Step d: Based on the optical flow from the second sample video frame to the first sample video frame Optical flow from the third sample video frame to the second sample video frame Obtain the target optical flow from the third sample video frame to the first sample video frame.

[0220] Specifically, the target optical flow can be obtained from the following three expressions.

[0221]

[0222]

[0223]

[0224] For a detailed description of step d, please refer to the description of steps 4071 to 4073 in the above embodiments, which will not be repeated here.

[0225] Step e: Based on the optical flow from the second sample video frame to the first sample video frame obtained in this estimation. The optical flow F from the real second sample video frame to the first sample video frame 1→0 And the optical flow from the third sample video frame to the second sample video frame obtained in this estimation. The optical flow F from the real third sample video frame to the second sample video frame t→1 Update the coupled optical flow network.

[0226] Specifically, based on the above estimates The real F 1→0 estimated And the real F t→1 Calculate the loss value of the preset loss function, and update the first feature extractor, the second feature extractor, and the optical flow prediction model in the coupled optical flow network according to the loss value.

[0227] In summary, by repeatedly executing steps a to e until a preset number of times (e.g., 12 times) or until the loss value of the coupled optical flow network reaches a preset condition, a trained coupled optical flow network can be obtained.

[0228] Optionally, the training sample set used for training the coupled optical flow network may further include a training sample set with accurate optical flow labels. The training sample set with accurate optical flow labels includes a first sample video frame, a second sample video frame, the optical flow from the first sample video frame to the second sample video frame, and the optical flow from the second sample video frame to the first sample video frame. The training sample set does not include a third sample video frame, the optical flow from the third sample video frame to the first sample video frame, and the optical flow from the third sample video frame to the second sample video frame.

[0229] In this embodiment of the application, a coupled optical flow network can be trained based on a training sample set with pseudo optical flow labels and a training sample set with accurate optical flow labels. During the training process of the coupled optical flow network, the two training sample sets can be combined into one training sample set.

[0230] Specifically, for training samples from a training sample set with pseudo-optical flow labels, the coupled optical flow network is updated according to the process described in steps 1 to 5 above. For training samples from a training sample set with accurate optical flow labels, since there is no third sample video frame, the optical flow from the third sample video frame to the first sample video frame, or the optical flow from the third sample video frame to the second sample video frame, the coupled optical flow network is updated only based on the estimated optical flow from the first sample video frame to the second sample video frame and the actual optical flow from the first sample video frame to the second sample video frame, as well as the estimated optical flow from the second sample video frame to the first sample video frame and the actual optical flow from the second sample video frame to the first sample video frame.

[0231] Optionally, in this embodiment of the application, the model for video frame interpolation further includes a video frame interpolation network, which is used to obtain a third video frame based on the first target optical flow and the second target optical flow, that is, the video frame interpolation network is used to implement step 408 in the above embodiment.

[0232] The video frame interpolation network can be trained based on a second training sample set, which includes a first sample video frame, a second sample video frame, the optical flow from the third sample video frame output by the coupled optical flow network to the first sample video frame, the optical flow from the third sample video frame output by the coupled optical flow network to the second sample video frame, and a third sample video frame, which is a real third sample video frame.

[0233] It is understandable that the training principle of the video frame interpolation network is similar to that of the coupled optical flow network described above. The optical flow from the first sample video frame, the second sample video frame, the third sample video frame output by the coupled optical flow network to the first sample video frame, and the optical flow from the third sample video frame output by the coupled optical flow network to the second sample video frame are input into a preset video frame interpolation network. The video frame interpolation network outputs a predicted third sample video frame. Then, the loss value is calculated based on the predicted third sample video frame and the real third sample video frame, and the video frame interpolation network is updated based on the loss value. By repeatedly executing the above process, a trained video frame interpolation network is obtained.

[0234] Optionally, during the process of updating the video frame interpolation network based on the predicted third sample video frames and the actual third sample video frames, the trained coupled optical flow network can also be fine-tuned based on the predicted and actual third sample video frames. When training the video frame interpolation network and the coupled optical flow network based on the second training sample set, the ratio of the learning rate of the video frame interpolation network to the learning rate of the coupled optical flow network can be set as needed, for example, to 1:10.

[0235] Accordingly, this application provides a video frame interpolation device. The video frame interpolation device can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0236] When dividing each function into modules according to its corresponding function. Figure 11 A schematic diagram of a possible structure of the video frame interpolation device involved in the above embodiments is shown. For example... Figure 11 As shown, the video frame interpolation device includes a matching cost calculation module 1101, a first feature extraction module 1102, an optical flow estimation module 1103, and a video frame interpolation module 1104.

[0237] The matching cost calculation module 1101 is used to obtain the first matching cost and the second matching cost based on the first video frame and the second video frame, for example, by performing step 401 in the above method embodiment.

[0238] The first feature extraction module 1102 is used to extract features from the first video frame to obtain the first feature of the first video frame, for example, by executing step 402 in the above method embodiment. Furthermore, the first feature extraction module 1102 is used to extract features from the second video frame to obtain the first feature of the second video frame, for example, by executing step 403 in the above method embodiment.

[0239] The optical flow estimation module 1103 is used to obtain a first optical flow and a second optical flow based on a first matching cost and a first feature of a first video frame. The first optical flow is the optical flow from the first video frame to the second video frame, and the second optical flow is the optical flow from the third video frame to the first video frame. The module also obtains a first target optical flow based on the first and second optical flows. The first target optical flow is the optical flow from the third video frame to the first video frame. For example, steps 404 and 405 in the above method embodiment are executed.

[0240] The optical flow estimation module 1103 is further configured to obtain a third optical flow and a fourth optical flow based on the second matching cost and the first feature of the second video frame, wherein the third optical flow is the optical flow from the second video frame to the first video frame and the fourth optical flow is the optical flow from the third video frame to the second video frame; and to obtain a second target optical flow based on the third optical flow and the fourth optical flow, wherein the second target optical flow is the optical flow from the third video frame to the second video frame, for example, by performing steps 406 and 407 in the above method embodiment.

[0241] The video frame interpolation module 1104 is used to obtain a third video frame based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow, for example, by performing step 408 in the above method embodiment.

[0242] Optionally, the optical flow estimation module 1103 is specifically used to determine a fifth optical flow based on the linear motion characteristics of the optical flow, which is the optical flow from the first video frame to the third video frame; and to reverse-distort the fifth optical flow using the second optical flow to obtain a first reverse-distorted optical flow and a first optical flow mask, the first optical flow mask being used to indicate whether the optical flow value in the fifth optical flow has been reverse-distorted; and to update the second optical flow based on the first optical flow mask and the first reverse-distorted optical flow to obtain a first target optical flow, for example, by performing steps 4051 to 4053 in the above method embodiment.

[0243] Optionally, the optical flow estimation module 1103 is specifically used to determine a sixth optical flow based on the linear motion characteristics of the optical flow, which is the optical flow from the second video frame to the third video frame; and to reverse-distort the sixth optical flow using a fourth optical flow to obtain a second reverse-distorted optical flow and a second optical flow mask, the second optical flow mask being used to indicate whether the optical flow value in the sixth optical flow has been reverse-distorted; and to update the fourth optical flow based on the second optical flow mask and the second reverse-distorted optical flow to obtain a second target optical flow, for example, by performing steps 4071 to 4073 in the above method embodiment.

[0244] Optionally, the video frame interpolation apparatus provided in this application embodiment includes a second feature extraction module 1105, which is used to extract features from a first video frame to obtain a second feature of the first video frame; and to extract features from a second video frame to obtain a second feature of the second video frame, for example, by executing steps 4011 and 4012 in the above method embodiment. The matching cost calculation module 1101 is specifically used to determine a first matching cost and a second matching cost based on the second features of the first and second video frames, for example, by executing step 4013 in the above method embodiment.

[0245] Each module of the above-mentioned video frame interpolation device can also be used to perform other actions in the above-mentioned method embodiments. All relevant content of each step involved in the above-mentioned method embodiments can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0246] When using integrated units, Figure 12 A schematic diagram of another possible structure of the video frame interpolation device involved in the above embodiments is shown. For example... Figure 12As shown, the video frame interpolation device provided in this application embodiment may include a processing module 1201 and a communication module 1202. The processing module 1201 can be used to control and manage the operation of the video frame interpolation device. For example, the processing module 1201 can be used to support the video frame interpolation device in executing steps 401 to 408 in the above method embodiments, and / or other processes used in the technology described herein. The communication module 1202 can be used to support communication between the video frame interpolation device and other network entities. Optionally, as... Figure 12 As shown, the video frame interpolation device may also include a storage module 1203 for storing the device's program code and data.

[0247] The processing module 1201 may be a processor or a controller (e.g., as described above). Figure 2 The processor 210 shown may be, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of this invention. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 1202 may be a transceiver, transceiver circuitry, or communication interface, etc. (e.g., as described above). Figure 2 The mobile communication module 250 or wireless communication module 260 shown. Storage module 1203 can be a memory (e.g., the one described above). Figure 2 The internal memory 221 shown.

[0248] When the processing module 1201 is a processor, the communication module 1202 is a transceiver, and the storage module 1203 is a memory, the processor, transceiver, and memory can be connected via a bus. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc.

[0249] For more details on how the modules included in the above video frame interpolation device achieve the above functions, please refer to the descriptions in the previous method embodiments, which will not be repeated here.

[0250] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0251] This application provides an electronic device, including a memory and at least one processor connected to the memory. The memory stores instructions, which, after being read by the at least one processor, are executed as follows: Figures 4 to 7 The method described in any embodiment.

[0252] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the following... Figures 4 to 7 The method described in any embodiment.

[0253] This application provides a computer program product containing instructions that, when run on a computer, execute the following... Figures 4 to 7 The method described in any embodiment.

[0254] This application provides a chip including a communication interface and a processing circuit. The communication interface is used to communicate with other devices or components. The processing circuit is used to execute computer instructions to perform tasks such as... Figures 4 to 7 The method described in any embodiment.

[0255] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0256] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0258] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0259] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0260] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0261] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video frame interpolation method, characterized in that, The method is used to insert a third video frame between a first video frame and a second video frame, the method comprising: Based on the first video frame and the second video frame, the first matching cost and the second matching cost are obtained; Feature extraction is performed on the first video frame to obtain the first feature of the first video frame; Feature extraction is performed on the second video frame to obtain the first feature of the second video frame; Based on the first matching cost and the first feature of the first video frame, a first optical flow and a second optical flow are obtained, wherein the first optical flow is the optical flow from the first video frame to the second video frame, and the second optical flow is the optical flow from the third video frame to the first video frame; based on the first optical flow and the second optical flow, a first target optical flow is obtained, wherein the first target optical flow is the optical flow from the third video frame to the first video frame; Based on the second matching cost and the first feature of the second video frame, a third optical flow and a fourth optical flow are obtained, wherein the third optical flow is the optical flow from the second video frame to the first video frame, and the fourth optical flow is the optical flow from the third video frame to the second video frame; based on the third optical flow and the fourth optical flow, a second target optical flow is obtained, wherein the second target optical flow is the optical flow from the third video frame to the second video frame; The third video frame is obtained based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow.

2. The method according to claim 1, characterized in that, The step of obtaining the first target optical flow based on the first optical flow and the second optical flow includes: Based on the linear motion characteristics of optical flow, a fifth optical flow is determined according to the first optical flow, wherein the fifth optical flow is the optical flow from the first video frame to the third video frame; The second optical flow is used to reverse twist the fifth optical flow to obtain a first reverse twisted optical flow and a first optical flow mask. The first optical flow mask is used to indicate whether the optical flow value in the fifth optical flow is reverse twisted. The second optical flow is updated based on the first optical flow mask and the first reverse twisted optical flow to obtain the first target optical flow.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the second target optical flow based on the third optical flow and the fourth optical flow includes: Based on the linear motion characteristics of optical flow, a sixth optical flow is determined according to the third optical flow, wherein the sixth optical flow is the optical flow from the second video frame to the third video frame; The sixth optical flow is reverse-distorted using the fourth optical flow to obtain a second reverse-distorted optical flow and a second optical flow mask. The second optical flow mask is used to indicate whether the optical flow value in the sixth optical flow is reverse-distorted. The fourth optical flow is updated based on the second optical flow mask and the second reverse twisted optical flow to obtain the second target optical flow.

4. The method according to any one of claims 1 to 3, characterized in that, The first matching cost is the matching cost of feature matching from the first video frame to the second video frame, and the second matching cost is the matching cost of feature matching from the second video frame to the first video frame.

5. The method according to any one of claims 1 to 4, characterized in that, Based on the first video frame and the second video frame, the first matching cost and the second matching cost are obtained, including: Feature extraction is performed on the first video frame to obtain the second feature of the first video frame; Feature extraction is performed on the second video frame to obtain the second feature of the second video frame; The first matching cost and the second matching cost are determined based on the second feature of the first video frame and the second feature of the second video frame.

6. The method according to any one of claims 1 to 5, characterized in that, The first target optical flow and the second target optical flow are obtained by a coupled optical flow network; The coupled optical flow network is trained based on a first training sample set, which includes a first sample video frame, a second sample video frame, optical flow from the first sample video frame to the second sample video frame, optical flow from the second sample video frame to the first sample video frame, optical flow from the third sample video frame to the first sample video frame, and optical flow from the third sample video frame to the second sample video frame.

7. The method according to claim 6, characterized in that, The third video frame is obtained by a video interpolation network; The video frame interpolation network is trained based on a second training sample set, which includes the first sample video frame, the second sample video frame, the optical flow from the third sample video frame to the first sample video frame output by the coupled optical flow network, the optical flow from the third sample video frame to the second sample video frame output by the coupled optical flow network, and the third sample video frame.

8. A video frame interpolation device, characterized in that, The video frame interpolation device is used to insert a third video frame between a first video frame and a second video frame. The device includes: a matching cost calculation module, a first feature extraction module, an optical flow estimation module, and a video frame interpolation module. The matching cost calculation module is used to obtain a first matching cost and a second matching cost based on the first video frame and the second video frame. The first feature extraction module is used to extract features from the first video frame to obtain a first feature of the first video frame; and to extract features from the second video frame to obtain a first feature of the second video frame; The optical flow estimation module is used to obtain a first optical flow and a second optical flow based on the first matching cost and the first feature of the first video frame, wherein the first optical flow is the optical flow from the first video frame to the second video frame and the second optical flow is the optical flow from the third video frame to the first video frame; and to obtain a first target optical flow based on the first optical flow and the second optical flow, wherein the first target optical flow is the optical flow from the third video frame to the first video frame. The optical flow estimation module is further configured to obtain a third optical flow and a fourth optical flow based on the second matching cost and the first feature of the second video frame, wherein the third optical flow is the optical flow from the second video frame to the first video frame, and the fourth optical flow is the optical flow from the third video frame to the second video frame; and to obtain a second target optical flow based on the third optical flow and the fourth optical flow, wherein the second target optical flow is the optical flow from the third video frame to the second video frame; The video frame interpolation module is used to obtain the third video frame based on the first video frame, the second video frame, the first target optical flow, and the second target optical flow.

9. The apparatus according to claim 8, characterized in that, The optical flow estimation module is specifically used to determine a fifth optical flow based on the first optical flow according to the linear motion characteristics of the optical flow, wherein the fifth optical flow is the optical flow from the first video frame to the third video frame; and to reverse-distort the fifth optical flow using the second optical flow to obtain a first reverse-distorted optical flow and a first optical flow mask, wherein the first optical flow mask is used to indicate whether the optical flow value in the fifth optical flow has been reverse-distorted. And the second optical flow is updated based on the first optical flow mask and the first reverse twisted optical flow to obtain the first target optical flow.

10. The apparatus according to claim 8 or 9, characterized in that, The optical flow estimation module is specifically used to determine a sixth optical flow based on the linear motion characteristics of the optical flow and the third optical flow, wherein the sixth optical flow is the optical flow from the second video frame to the third video frame; and to reverse-distort the sixth optical flow using the fourth optical flow to obtain a second reverse-distorted optical flow and a second optical flow mask, wherein the second optical flow mask is used to indicate whether the optical flow value in the sixth optical flow has been reverse-distorted. The fourth optical flow is updated based on the second optical flow mask and the second reverse twisted optical flow to obtain the second target optical flow.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The first matching cost is the matching cost of feature matching from the first video frame to the second video frame, and the second matching cost is the matching cost of feature matching from the second video frame to the first video frame.

12. The apparatus according to any one of claims 8 to 11, characterized in that, The device also includes a second feature extraction module; The second feature extraction module is used to extract features from the first video frame to obtain a second feature of the first video frame; and to extract features from the second video frame to obtain a second feature of the second video frame. The matching cost calculation module is specifically used to determine the first matching cost and the second matching cost based on the second feature of the first video frame and the second feature of the second video frame.

13. The apparatus according to any one of claims 8 to 12, characterized in that, The optical flow estimation module is a coupled optical flow network; The coupled optical flow network is trained based on a first training sample set, which includes a first sample video frame, a second sample video frame, optical flow from the first sample video frame to the second sample video frame, optical flow from the second sample video frame to the first sample video frame, optical flow from the third sample video frame to the first sample video frame, and optical flow from the third sample video frame to the second sample video frame.

14. The apparatus according to claim 13, characterized in that, The video frame interpolation module is a video frame interpolation network; The video frame interpolation network is trained based on a second training sample set, which includes the first sample video frame, the second sample video frame, the optical flow from the third sample video frame to the first sample video frame output by the coupled optical flow network, the optical flow from the third sample video frame to the second sample video frame output by the coupled optical flow network, and the third sample video frame.

15. An electronic device, characterized in that, The method includes a memory and at least one processor connected to the memory, the memory being used to store instructions which, when read by the at least one processor, execute the method as described in any one of claims 1 to 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that, The computer program product includes instructions that, when the computer program product is run on a computer, perform the method described in any one of claims 1 to 7.

Images