Frame insertion processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a frame insertion processing method and device, electronic equipment and a readable storage medium, and belongs to the field of image processing. The method comprises the steps that a first frame, a second frame, a first motion vector and a second motion vector are obtained, the second frame is adjacent to the first frame and is behind the first frame, the first motion vector is a motion vector from a second moment when the second frame is collected to a first moment when the first frame is collected, and the second motion vector is a motion vector from the first moment to the second moment; determining at least three prediction frames corresponding to a third moment based on the first frame, the second frame, the first motion vector and the second motion vector, the third moment being a moment between the first moment and the second moment; based on the at least three prediction frames, at least two weights are determined, and the at least two weights are used for distributing fusion processing proportions of the at least three prediction frames; and based on the at least two weights, performing fusion processing on the at least three prediction frames to obtain an intermediate frame corresponding to the third moment.

Description

Technical Field

[0001] The present application belongs to the field of image processing, and specifically relates to a frame insertion processing method, device, electronic device and readable storage medium. Background Art

[0002] As electronic devices continue to expand in functionality, more and more users are using them to run games, virtual reality (VR), and augmented reality (AR) services for an immersive experience. These services often have stringent frame rate requirements, but due to limitations in the hardware performance of electronic devices, it's difficult to maintain a high frame rate when running these services.

[0003] Currently, interpolation can be used to increase frame rates and reduce image lag. Common interpolation techniques include motion-compensated interpolation, which analyzes the motion vector of an object from the subsequent frame to the previous frame, calculates the object's displacement between adjacent frames, and then uses specific rules to determine the fusion weight of the previous and next frames. Based on this fusion weight, a new frame is inserted in the middle.

[0004] However, because the same location can be rendered differently due to varying lighting conditions, simply analyzing the motion vector of an object from the subsequent frame to the previous one can lead to color discrepancies due to the different lighting conditions. Furthermore, the fixed weighting rules for the previous and next frames often struggle to adapt to complex and changing scenes, resulting in poor weight distribution accuracy. This results in poor interpolation performance. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a frame insertion processing method, device, electronic device and readable storage medium, which can improve the frame insertion effect during frame insertion.

[0006] In a first aspect, an embodiment of the present application provides an interpolation processing method, the method including: acquiring a first frame, a second frame, a first motion vector, and a second motion vector, the second frame being adjacent to the first frame and after the first frame, the first motion vector being a motion vector from the second moment of acquiring the second frame to the first moment of acquiring the first frame, and the second motion vector being a motion vector from the first moment to the second moment; based on the first frame, the second frame, the first motion vector, and the second motion vector, determining at least three predicted frames corresponding to a third moment, the third moment being a moment between the first moment and the second moment; based on the at least three predicted frames, determining at least two weights, the at least two weights being used to allocate fusion processing ratios of the at least three predicted frames; based on the at least two weights, performing fusion processing on the at least three predicted frames to obtain an intermediate frame corresponding to the third moment.

[0007] In a second aspect, an embodiment of the present application provides an interpolation processing device, which includes: an acquisition module, a determination module and a processing module; the acquisition module is used to acquire a first frame, a second frame, a first motion vector and a second motion vector, the second frame is adjacent to the first frame and is after the first frame, the first motion vector is a motion vector from the second moment of acquiring the second frame to the first moment of acquiring the first frame, and the second motion vector is a motion vector from the first moment to the second moment; the determination module is used to determine at least three predicted frames corresponding to a third moment based on the first frame, the second frame, the first motion vector and the second motion vector, the third moment being a moment between the first moment and the second moment; and based on the at least three predicted frames, determine at least two weights, the at least two weights being used to allocate fusion processing ratios of the at least three predicted frames; the processing module is used to perform fusion processing on the at least three predicted frames based on the at least two weights to obtain an intermediate frame corresponding to the third moment.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program / program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, a first frame, a second frame, a first motion vector, and a second motion vector can be obtained, where the second frame is adjacent to the first frame and follows the first frame, the first motion vector is a motion vector from the second moment when the second frame is acquired to the first moment when the first frame is acquired, and the second motion vector is a motion vector from the first moment to the second moment; based on the first frame, the second frame, the first motion vector, and the second motion vector, at least three predicted frames corresponding to a third moment are determined, where the third moment is a moment between the first moment and the second moment; based on the at least three predicted frames, at least two weights are determined, and the at least two weights are used to allocate fusion processing ratios of the at least three predicted frames; based on the at least two weights, fusion processing is performed on the at least three predicted frames to obtain an intermediate frame corresponding to the third moment. This solution, when performing interpolation, analyzes the motion vectors of objects from the subsequent frame to the previous frame, and from the previous frame to the subsequent frame. By introducing inverse motion vectors, bidirectional motion compensation is achieved. Using bidirectional motion vectors, the light and shadow changes between the previous and next frames can be captured, eliminating color differences caused by varying lighting conditions. Furthermore, weights are determined based on at least three predicted frames, rather than using fixed setting rules as in related technologies. This allows accurate weighting of the predicted frames themselves, even in complex and changing scenes. This improves interpolation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flowchart of a frame insertion processing method provided by some embodiments of the present application;

[0014] Figure 2 is a flowchart of a frame insertion processing method provided by some embodiments of the present application;

[0015] Figure 3 is a flowchart of a frame insertion processing method provided by some embodiments of the present application;

[0016] Figure 4 is a flowchart of a frame insertion processing method provided by some embodiments of the present application;

[0017] Figure 5 is a flowchart of a frame insertion processing method provided by some embodiments of the present application;

[0018] Figure 6 is a flowchart of a frame insertion processing method provided by some embodiments of the present application;

[0019] Figure 7 is a schematic diagram of a frame insertion processing device provided in an embodiment of the present application;

[0020] Figure 8 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0021] Figure 9 This is a hardware diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0023] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] The terms "at least one" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the objects included. For example, at least one of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "at least two" means two or more, and its meaning is similar to "at least one".

[0025] The following explains some concepts and terms involved in the interpolation processing method, device, electronic device and readable storage medium provided in the embodiments of the present application.

[0026] A motion vector (MV) is a parameter that describes the two-dimensional spatial displacement of a region (such as a pixel block or object) between adjacent frames in a video or image sequence. It is typically expressed as (Δx, Δy), where Δx and Δy represent the horizontal and vertical pixel displacements of the region, respectively. It is mapped from a reference frame to the current frame through methods such as block matching, feature tracking, or deep learning. It is used for tasks such as video compression (reducing redundant data), motion compensation (frame prediction and interpolation), and target tracking. It is a core tool for quantifying spatiotemporal motion.

[0027] A predicted frame is a frame generated by referring to a previously encoded frame (and using motion vectors and residual data) in video compression and inter-frame coding. Its core principle is to predict the current frame content from the reference frame through motion compensation technology, and only store the difference between the actual pixel and the predicted value and the motion vector, thereby greatly reducing redundant information and achieving efficient compression.

[0028] Optical flow describes the instantaneous motion pattern of each pixel in an image sequence between consecutive frames. It is expressed as a two-dimensional vector field, where each vector represents the motion direction and speed of the corresponding pixel between adjacent frames. Its calculation is based on the assumption of constant brightness and spatiotemporal continuity constraints. It is widely used in tasks such as target tracking, action recognition, autonomous driving, and video interpolation. Classic algorithms include Lucas-Kanade (sparse optical flow) and Horn-Schunck (dense optical flow). Deep learning models (such as FlowNet and RAFT) have further improved estimation accuracy in complex scenarios.

[0029] A hole is a data missing or invalid area caused by the inability of the reference frame pixels to completely cover certain areas of the target frame through motion vectors during image processing or video frame generation. It is usually manifested as the pixel value at that location not being correctly predicted or filled. It needs to be eliminated through interpolation, repair, or context-based completion algorithms to ensure the continuity and visual consistency of the frame content.

[0030] The following describes in detail the frame insertion processing method, device, electronic device, and readable storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] The interpolation processing method provided in the embodiment of the present application can be applied to scenarios where interpolation processing is performed when running game services, VR services or AR services.

[0032] Exemplarily, when a game service is running, the first frame, the second frame, and the first motion vector and the second motion vector of the game service can be obtained, the second frame is adjacent to the first frame and follows the first frame, the first motion vector is the motion vector from the second moment of collecting the second frame to the first moment of collecting the first frame, and the second motion vector is the motion vector from the first moment to the second moment; and based on the first frame, the second frame, the first motion vector and the second motion vector, the predicted frame 1, the predicted frame 2 and the predicted frame 3 corresponding to the third moment are determined, and the third moment is the moment between the first moment and the second moment; and based on the predicted frame 1, the predicted frame 2 and the predicted frame 3, the weight a and the weight b are determined, and the weight a and the weight b are used to allocate the fusion processing ratio of the predicted frame 1, the predicted frame 2 and the predicted frame 3; and based on the weight a and the weight b, the predicted frame 1, the predicted frame 2 and the predicted frame 3 are fused to obtain the intermediate frame corresponding to the third moment.

[0033] Exemplarily, when a VR service is running, the first frame, the second frame, and the first motion vector and the second motion vector of the VR service can be obtained, the second frame is adjacent to the first frame and follows the first frame, the first motion vector is the motion vector from the second moment of acquiring the second frame to the first moment of acquiring the first frame, and the second motion vector is the motion vector from the first moment to the second moment; and based on the first frame, the second frame, the first motion vector and the second motion vector, the predicted frame A, predicted frame B, predicted frame C and predicted frame D corresponding to the third moment are determined, and the third moment is the moment between the first moment and the second moment; and based on the predicted frame A, predicted frame B, predicted frame C and predicted frame D, the weight m, weight n and weight t are determined, and the weight m, weight n and weight t are used to allocate the fusion processing ratio of the predicted frame A, predicted frame B, predicted frame C and predicted frame D; and based on the weight m, weight n and weight t, fusion processing is performed on the predicted frame A, predicted frame B, predicted frame C and predicted frame D to obtain the intermediate frame corresponding to the third moment.

[0034] Thus, when performing interpolation, by analyzing the motion vectors of objects from the subsequent frame to the previous frame, and from the previous frame to the subsequent frame, and introducing the reverse motion vector, bidirectional motion compensation is achieved. Using the bidirectional motion vectors, the lighting changes between the previous and next frames can be captured, thereby eliminating color differences caused by different lighting conditions. Furthermore, when determining weights, the weights are based on at least three predicted frames, rather than using fixed setting rules as in related technologies. This allows accurate weighting of the predicted frames themselves, even in complex and changing scenes. This improves the interpolation effect.

[0035] It should be noted that the frame insertion processing method provided in the embodiments of the present application can be executed by a frame insertion processing device, an electronic device, or a functional module in an electronic device. In some embodiments of the present application, the frame insertion processing method provided in the embodiments of the present application is described by taking an electronic device executing the frame insertion processing method as an example.

[0036] Figure 1 FIG. 1 shows a flow chart of the frame insertion processing method provided by an embodiment of the present application. Figure 1 As shown, the interpolation processing method provided in the embodiment of the present application may include the following steps 101 to 104.

[0037] Step 101: The electronic device obtains a first frame, a second frame, a first motion vector, and a second motion vector.

[0038] The second frame is adjacent to the first frame and follows the first frame, the first motion vector is the motion vector from the second moment of capturing the second frame to the first moment of capturing the first frame, and the second motion vector is the motion vector from the first moment to the second moment.

[0039] Optionally, in an embodiment of the present application, the first frame and the second frame may include but are not limited to any of the following: video frames, frames in video call services, frames in game services, frames in VR services, and frames in AR services.

[0040] Optionally, in the embodiment of the present application, the second frame is adjacent to the first frame and is after the first frame, which can be understood as: the first frame and the second frame are adjacent frames, and the second frame is a frame collected after the first frame.

[0041] For example, in the rendering pipeline of an electronic game, two consecutive frames of original frame images at a certain continuous moment are captured, the earlier moment is called moment 0, and the original frame image at moment 0 is the above-mentioned first frame; and the later moment is called moment 1, and the original frame image at moment 1 is the above-mentioned second frame.

[0042] Optionally, in an embodiment of the present application, the second frame may be a frame currently being rendered in the rendering pipeline, and the first frame may be a frame that has been rendered before the second frame.

[0043] Optionally, in the embodiment of the present application, the first motion vector may also be referred to as the motion vector at the second moment.

[0044] Optionally, in the embodiment of the present application, the second motion vector may also be referred to as the reverse motion vector of the first motion vector.

[0045] It can be understood that the first motion vector can represent the two-dimensional spatial displacement information of the object from the second frame to the first frame; the second motion vector can represent the two-dimensional spatial displacement information of the object from the first frame to the second frame.

[0046] Optionally, in the embodiment of the present application, Figure 1 ,like Figure 2 As shown, the above step 101 can be specifically implemented through the following steps 101a to 101d, or can be implemented through the following steps 101a, 101b, 101c and 101e.

[0047] Step 101a: The electronic device obtains a first frame, a second frame, and a first motion vector from a rendering pipeline.

[0048] Optionally, in an embodiment of the present application, the first frame and the second frame may be original frame images obtained from a rendering pipeline.

[0049] Optionally, in an embodiment of the present application, the above-mentioned rendering pipeline may be a rendering pipeline corresponding to the services corresponding to the first frame and the second frame.

[0050] Optionally, in an embodiment of the present application, the electronic device may first obtain the first frame, the second frame and the geometric buffer (G-Buffer) from the above-mentioned rendering pipeline, and then obtain the first motion vector from the G-Buffer; wherein the G-Buffer is used to store geometric information during the rendering process.

[0051] Step 101b: The electronic device performs forward mapping processing on the first motion vector using the first motion vector to obtain a third motion vector.

[0052] It should be noted that the calculation of the reverse motion vector is to add a minus sign to the original motion vector. However, since the pixel positions will change, the image also needs to be forward mapped (forward warp) processing. Forward mapping processing is based on the existing motion vector, and the pixels of the previous frame are "moved" to the corresponding positions in the current frame according to the direction and distance indicated by the motion vector. Simply put, based on the motion information, the content of the previous frame is "moved" to the current frame to construct the new image content.

[0053] Step 101c: The electronic device performs an inversion process on the third motion vector to obtain a fourth motion vector.

[0054] For example, assuming that the first motion vector is MV 1→0 , then the fourth motion vector MV 0→1 It can be determined by the following formula (1):

[0055] MV 0→1 =-1*ForwardWarp(MV 1→0 ,MV 1→0 ); (1)

[0056] Among them, ForwardWarp(MV 1→0 ,MV 1→0 ) is the third motion vector.

[0057] Step 101d: The electronic device determines the fourth motion vector as the second motion vector.

[0058] Optionally, in the embodiment of the present application, after determining the fourth motion vector, the electronic device may directly determine the fourth motion vector as the second motion vector.

[0059] Step 101e: The electronic device performs invalid value processing on the hole in the fourth motion vector, and determines the processed fourth motion vector as the second motion vector.

[0060] The hole is determined based on the first motion vector.

[0061] It's important to note that during forward mapping, several pixels may point to the same pixel simultaneously, resulting in holes in the motion vector. Holes are pixels that cannot be mapped using motion vectors. Therefore, holes need to be assigned invalid values, effectively performing invalid value processing on them.

[0062] Optionally, in an embodiment of the present application, the electronic device may first determine a hole in the fourth motion vector based on the first motion vector, and then perform invalid value processing on the hole in the fourth motion vector.

[0063] For the specific method of determining the hole in the motion vector, reference may be made to the relevant description in the related art, and in order to avoid repetition, it will not be described here.

[0064] Optionally, in an embodiment of the present application, the invalid value processing may include: mapping the pixel values in the hole to outside the image.

[0065] For example, assuming that the first motion vector is MV 1→0 , then the electronic device can first perform invalid value processing on the hole in the fourth motion vector using the following formulas (2) and (3) to obtain the second motion vector MV 0→1 ′:

[0066] Hole=ForwardWarp(I,MV 1→0 ); (2)

[0067] MV 0→1 ′=MV 0→1 *Hole+MAX_HW*(I-Hole); (3)

[0068] Where I is an image with all pixel values 1, MV 1→0 is the first motion vector, Hole is the hole area after invalid value processing, MV 0→1 is the fourth motion vector, and MAX_HW is the maximum value of the image height H and width W at each position. Then, after performing invalid value processing on the hole in the fourth motion vector, the electronic device may determine the processed fourth motion vector as the second motion vector.

[0069] In an embodiment of the present application, since the electronic device can quickly obtain the second motion vector by directly performing forward mapping processing and inversion operation on the first motion vector, or can perform forward mapping processing and inversion operation on the first motion vector and then process the invalid values of the motion vector holes to more accurately obtain the second motion vector, the second motion vector can be determined in different ways, thereby improving the flexibility of determining the second motion vector.

[0070] Step 102: The electronic device determines at least three predicted frames corresponding to a third moment based on the first frame, the second frame, the first motion vector, and the second motion vector.

[0071] The third moment is a moment between the first moment and the second moment.

[0072] Optionally, in the embodiment of the present application, the third moment may be any moment between the first moment and the second moment.

[0073] For example, the third moment may be a moment in between the first moment and the second moment, or may be a moment determined according to interpolation processing requirements, etc.

[0074] Optionally, in an embodiment of the present application, the at least three predicted frames are used to predict an intermediate frame at the third moment.

[0075] Optionally, in the embodiment of the present application, Figure 1 ,like Figure 3 As shown, the above step 102 can be specifically implemented through the following steps 102a to 102d.

[0076] Step 102a: The electronic device performs forward mapping processing on the first frame according to the second motion vector and the third moment to obtain a first motion vector predicted frame.

[0077] It should be noted that the motion vector predicted frame in the embodiments of the present application is a predicted frame determined based on a motion vector. Since the motion vector in the rendering pipeline tracks the change between the "position of the previous frame" and "position of the current frame" of the pixels that make up the polygons on the screen, the predicted frame can be determined by the motion vector.

[0078] Optionally, in an embodiment of the present application, the electronic device may obtain a predicted frame at the third moment by warping.

[0079] For example, assuming that the object moves linearly, the motion vector at the intermediate moment can be calculated by multiplying the motion vector by the corresponding time, and then the images at the first moment and the second moment are forward mapped based on the obtained motion vector to construct the predicted frame at the third moment. Specifically, the electronic device can determine the predicted frame of the first motion vector by the following formula (4):

[0080]

[0081] Among them, I0 is the first frame, t is the third moment, MV 0→1 is the second motion vector.

[0082] Step 102b: The electronic device performs forward mapping processing on the second frame according to the first motion vector and the third moment to obtain a second motion vector predicted frame.

[0083] Exemplarily, the electronic device may determine the second motion vector prediction frame by the following formula (5):

[0084]

[0085] Among them, I1 is the second frame, t is the third moment, MV 1→0 is the first motion vector.

[0086] Step 102c: The electronic device determines a first optical flow from the first moment to the second moment based on the first frame and the second frame, and performs forward mapping processing on the first frame based on the first optical flow and the third moment to obtain a first optical flow prediction frame.

[0087] It should be noted that the optical flow prediction frame in the embodiment of the present application is a prediction frame determined based on the optical flow. Shadows and special effects are usually not specific polygons. They are graphic effects that interact with the scene, and their motion information may require additional optical flow to supplement. Optical flow estimation can be achieved through methods such as Lucas-Kanade or block matching. This application requires that the optical flow calculation has the ability to calculate in real time. For the specific method of optical flow estimation, please refer to the specific description in the relevant technology. In order to avoid repetition, it will not be repeated here.

[0088] For example, assuming that the object moves linearly, the optical flow at the intermediate moment can be calculated by multiplying the optical flow by the corresponding time. The images at the first moment and the second moment are forward mapped based on the generated optical flow, and then the predicted frame at the third moment is constructed. Specifically, the electronic device can determine the first optical flow prediction frame by the following formula (6):

[0089]

[0090] Among them, I0 is the first frame, t is the third moment, OF 0→1 is the first optical flow.

[0091] Step 102d: The electronic device determines the first motion vector prediction frame, the second motion vector prediction frame, and the first optical flow prediction frame as at least three prediction frames.

[0092] Optionally, in an embodiment of the present application, after determining the first motion vector prediction frame, the second motion vector prediction frame and the first optical flow prediction frame, the electronic device can directly determine the first motion vector prediction frame, the second motion vector prediction frame and the first optical flow prediction frame as the above-mentioned at least three prediction frames for use in determining subsequent intermediate frames.

[0093] In an embodiment of the present application, since the electronic device can determine the predicted frame for predicting the intermediate frame based on the bidirectional motion vector and the unidirectional optical flow, the determined predicted frame can accurately reflect the change of the pixel between the "position of the previous frame" and the "position of the current frame" to improve the accuracy of the predicted intermediate frame.

[0094] Optionally, in the embodiment of the present application, Figure 3 ,like Figure 4 As shown, the interpolation processing method provided in the embodiment of the present application may further include the following step 102e, and the above step 102d may be specifically implemented through the following step 102d1.

[0095] Step 102e: The electronic device determines a second optical flow from the second moment to the first moment based on the first frame and the second frame, and performs forward mapping processing on the second frame based on the second optical flow and the third moment to obtain a second optical flow prediction frame.

[0096] It should be noted that, in order to achieve better interpolation effect, the present application can also calculate bidirectional optical flow and determine the predicted frame based on the bidirectional optical flow.

[0097] Exemplarily, the electronic device may determine the second optical flow prediction frame by the following formula (7):

[0098]

[0099] Among them, I1 is the second frame, t is the third moment, OF 1→0 is the second optical flow.

[0100] Step 102d1: The electronic device determines the first motion vector prediction frame, the second motion vector prediction frame, the first optical flow prediction frame, and the second optical flow prediction frame as at least three prediction frames.

[0101] Optionally, in an embodiment of the present application, after determining the first motion vector prediction frame, the second motion vector prediction frame, the first optical flow prediction frame and the second optical flow prediction frame, the electronic device can determine the first motion vector prediction frame, the second motion vector prediction frame, the first optical flow prediction frame and the second optical flow prediction frame as the above-mentioned at least three prediction frames for use in determining subsequent intermediate frames.

[0102] In an embodiment of the present application, since the electronic device can determine the predicted frame for predicting the intermediate frame based on the bidirectional motion vector and the bidirectional optical flow, the determined predicted frame can further accurately reflect the change of the pixel between the "position of the previous frame" and the "position of the current frame", so as to improve the accuracy of predicting the intermediate frame and improve the effect of interpolation.

[0103] Step 103: The electronic device determines at least two weights based on at least three predicted frames.

[0104] The at least two weights are used to allocate a fusion processing ratio of the at least three predicted frames.

[0105] For example, assuming that the electronic device confirms that the above-mentioned at least two weights include weight a and weight b, after confirming the at least two weights, the fusion processing ratio of predicted frame 1 can be determined as weight a, the fusion processing ratio of predicted frame 2 can be determined as weight b, and the fusion processing ratio of predicted frame 3 can be determined as (1-weight a-weight b) during fusion processing.

[0106] Optionally, in an embodiment of the present application, the at least two weights mentioned above can be used to determine the fusion weight corresponding to each predicted frame, that is, the weight used in the fusion process.

[0107] Optionally, in the embodiment of the present application, the at least three prediction frames include two motion vector prediction frames and at least one optical flow prediction frame; illustratively, in combination with Figure 1 ,like Figure 5 As shown, the above step 103 can be specifically implemented through the following steps 103a to 103c.

[0108] Step 103a: The electronic device inputs at least three prediction frames into the neural network.

[0109] The above neural network is used to determine the fusion weight of the image.

[0110] Optionally, in the embodiment of the present application, the neural network is intended to fuse images, so the network size can be kept small. In terms of network structure, a Unet-like backbone network can be used, which includes several upsampling and downsampling processes, for example, 3, 4, 5, or 6 upsampling and downsampling times, which can achieve good results.

[0111] Optionally, in an embodiment of the present application, the electronic device may merge the above-mentioned at least three prediction frames into a tensor as input of the above-mentioned neural network.

[0112] For example, when the above-mentioned at least three prediction frames include the above-mentioned first motion vector prediction frame, the second motion vector prediction frame and the first optical flow prediction frame, the electronic device can merge the first motion vector prediction frame, the second motion vector prediction frame and the first optical flow prediction frame into a tensor with 9 channels as the input of the above-mentioned neural network.

[0113] For example, when the above-mentioned at least three prediction frames include the above-mentioned first motion vector prediction frame, the second motion vector prediction frame, the first optical flow prediction frame and the second optical flow prediction frame, the electronic device can merge the first motion vector prediction frame, the second motion vector prediction frame, the first optical flow prediction frame and the second optical flow prediction frame into a tensor with 12 channels as the input of the above-mentioned neural network.

[0114] Step 103b: The electronic device obtains difference information between at least one optical flow prediction frame and each motion vector prediction frame through a neural network.

[0115] In the embodiment of the present application, each of the above-mentioned motion vector prediction frames is each of the above-mentioned two motion vector prediction frames.

[0116] Optionally, in an embodiment of the present application, the neural network may include a Transformer Block, which can extract the above-mentioned difference information using an inter-frame attention mechanism. Specifically, this is achieved by setting any region in an optical flow prediction frame as a query in the attention mechanism, and simultaneously using all spatially adjacent regions of a motion vector prediction frame as keys and values. This effectively compensates for inaccurate optical flow calculations and improves interpolation quality.

[0117] Optionally, in the embodiment of the present application, the above difference information may include but is not limited to at least one of the following:

[0118] Pixel-level differences in color and brightness, contrast and noise, and basic metrics;

[0119] low-level feature differences between edges and textures and local responses;

[0120] mid-level feature differences in shape and components and spatial structure;

[0121] Spatial differences in position and scale, deformation and movement, etc.

[0122] Exemplarily, assuming that the at least one optical flow prediction frame includes the first optical flow prediction frame and the second optical flow prediction frame, and the two motion vector prediction frames include the first motion vector prediction frame and the second motion vector prediction frame, then the difference information between the at least one optical flow prediction frame and each motion vector prediction frame may include: the difference information between the first optical flow prediction frame and the first motion vector prediction frame, the difference information between the first optical flow prediction frame and the second motion vector prediction frame, the difference information between the second optical flow prediction frame and the first motion vector prediction frame, and the difference information between the second optical flow prediction frame and the second motion vector prediction frame.

[0123] Step 103c: The electronic device determines at least two weights according to the difference information.

[0124] Optionally, in an embodiment of the present application, after determining the difference information, the neural network may directly output the at least two weights according to the difference information.

[0125] It's important to note that due to the relatively small scale of this neural network, it can be run on electronic devices with limited computing power. For example, at a 1080p resolution, this neural network can achieve a minimum computational load of 20GFlops, enabling easy real-time computing on mobile devices.

[0126] In an embodiment of the present application, since the electronic device can input the above-mentioned at least three prediction frames into the neural network, and then obtain the difference between the optical flow prediction frame and the motion vector prediction frame through the neural network to determine at least two weights, the at least two weights can be determined only through the neural network, thereby simplifying the operational steps of determining the weights.

[0127] Step 104: The electronic device performs a fusion process on the at least three predicted frames based on at least two weights to obtain an intermediate frame corresponding to a third moment.

[0128] Optionally, in an embodiment of the present application, the electronic device may first determine the fusion processing ratio of the at least three predicted frames based on the at least two weights, and then perform fusion processing on the at least three predicted frames using the fusion processing ratio of the at least three predicted frames to obtain an intermediate frame corresponding to the third moment.

[0129] Optionally, in an embodiment of the present application, after determining the above-mentioned at least two weights, the electronic device can directly assign the at least two weights to different prediction frames, and then multiply each prediction frame by its corresponding weight, and then add the results corresponding to each prediction frame after multiplication to obtain the intermediate frame corresponding to the third moment.

[0130] For the specific method of the above fusion processing, please refer to the relevant description in the relevant technology. In order to avoid repetition, it will not be repeated here.

[0131] It can be understood that since the third moment is a moment between the first moment and the second moment, the intermediate frame corresponding to the third moment is the intermediate frame inserted between the first frame and the second frame.

[0132] Optionally, in the embodiment of the present application, Figure 1 ,like Figure 6 As shown, the above step 104 can be specifically implemented through the following steps 104a and 104b.

[0133] Step 104a: The electronic device determines a fusion weight corresponding to each predicted frame according to at least two weights.

[0134] Exemplarily, assuming that the above-mentioned at least two weights include weight 1 and weight 2, and the above-mentioned at least three prediction frames include the above-mentioned first motion vector prediction frame, the second motion vector prediction frame and the first optical flow prediction frame, then the electronic device can determine weight 1 as the fusion weight corresponding to the first motion vector prediction frame, and determine weight 2 as the fusion weight corresponding to the second motion vector prediction frame, and determine weight 1 as the fusion weight corresponding to the first optical flow prediction frame; or, the electronic device can determine weight 1 as the fusion weight corresponding to the first motion vector prediction frame, and determine weight 2 as the fusion weight corresponding to the second motion vector prediction frame, and determine weight 3 as the fusion weight corresponding to the first optical flow prediction frame, where weight 3 = 1-weight 1-weight 2.

[0135] Step 104b: The electronic device performs a fusion process on the at least three predicted frames according to the first information to obtain an intermediate frame corresponding to the third moment.

[0136] The first information includes any of the following:

[0137] a fusion weight corresponding to each of the at least three predicted frames;

[0138] The fusion weight corresponding to each of the at least three prediction frames and the residual determined according to the at least three prediction frames, the residual being used to correct the image obtained by the fusion process.

[0139] Optionally, in an embodiment of the present application, the electronic device may input the at least three predicted frames into the neural network, and then the neural network outputs the residual while outputting the at least two weights.

[0140] For example, taking the case where the first information includes the fusion weights and the residual corresponding to each of the above-mentioned prediction frames, and the above-mentioned at least three prediction frames include the above-mentioned first motion vector prediction frame, the second motion vector prediction frame, and the first optical flow prediction frame, the electronic device can perform fusion processing on the above-mentioned at least three prediction frames according to the first information using the following formula (8) to obtain the intermediate frame I corresponding to the third moment t :

[0141]

[0142] in, is the first motion vector predicted frame, is the second motion vector prediction frame, mask1 is the fusion weight corresponding to the first motion vector prediction frame, and mask2 is the fusion weight corresponding to the second motion vector prediction frame.

[0143] (1-mask1-mask2) is the fusion weight corresponding to the first optical flow prediction frame, and refine_delta is the residual determined according to the first motion vector prediction frame, the second motion vector prediction frame, and the first optical flow prediction frame.

[0144] In an embodiment of the present application, since the electronic device can perform fusion processing on at least three predicted frames based on the first information to obtain an intermediate frame corresponding to the third moment, and the first information includes any one of the following items: the fusion weight corresponding to each of the at least three predicted frames; the fusion weight corresponding to each of the at least three predicted frames and the residual determined based on the at least three predicted frames, and the residual is used to correct the image obtained by the fusion processing, the fusion processing can be performed on the at least three predicted frames through different first information, thereby improving the flexibility of obtaining the intermediate frame.

[0145] In the interpolation processing method provided in the embodiments of the present application, when performing interpolation processing, on the one hand, by analyzing the motion vector of the object from the subsequent frame to the previous frame, and the motion vector of the object from the previous frame to the subsequent frame, after introducing the reverse motion vector, bidirectional motion compensation is achieved. The use of the bidirectional motion vector can obtain the light and shadow changes between the previous and next frames, thereby eliminating color differences caused by different lighting conditions. On the other hand, when determining the weight, it is based on at least three predicted frames, rather than using a fixed setting rule as in related technologies. Therefore, even in complex and changing scenes, the weight can be accurately assigned based on the predicted frame itself. This can improve the interpolation effect.

[0146] The following is an exemplary description of the frame insertion processing method provided in the embodiment of the present application.

[0147] For example, the frame insertion processing method provided in the embodiment of the present application may specifically include the following process:

[0148] Step 1: First, get the original frame images and corresponding G-Buffer of two frames in the rendering pipeline. The original frame is the image obtained by the rendering pipeline at time 0 (first moment) and time 1 (second moment), which are respectively recorded as I0 (first frame) and I1 (second frame). G-Buffer stores the geometric information during the rendering process. We take the motion vector (Motion Vector) at time 1 and record it as MV 1→0 (First motion vector).

[0149] Step 2: Calculate the reverse motion vector MV 0→1(Second motion vector). The calculation of the reverse motion vector is to add a minus sign in front of the original motion vector, but because the pixel position will change, the image also needs to be forward mapped. As shown in the above formula (1), the forward mapping is based on the existing motion vector, and the pixels of the previous frame are "moved" to the corresponding position of the current frame according to the direction and distance indicated by the motion vector. In simple terms, it is to "move" the content of the previous frame to the current frame based on the motion information to construct the new image content. It should also be noted that when performing the forward mapping process, there will be a situation where several pixels point to the same pixel at the same time, which leads to the MV 0→1 There will be some holes. The holes here refer to the pixels that cannot pass the MV 1→0 Therefore, it is necessary to assign the hole values to invalid values. As shown in the above formula (2) and formula (3), in order to simplify the operation, these pixel values are uniformly mapped outside the image.

[0150] Step 3: Based on the original frame images I0 and I1, use the optical flow estimation method to estimate the optical flow OF between time 0 and time 1 1→0 (First optical flow). Since the motion vector in the rendering pipeline tracks the changes between the "position of the previous frame" and the "position of the current frame" of the pixels that make up the polygons on the screen. However, shadows and special effects are usually not specific polygons. They are graphic effects that interact with the scene, and their motion information may require additional optical flow to supplement. Optical flow estimation can be achieved through methods such as Lucas-Kanade or block matching. This application requires that the optical flow calculation has the ability to calculate in real time, so deep learning methods are not used. This application mainly describes the solution for calculating unidirectional optical flow. If bidirectional optical flow is calculated (the optical flow OF is also calculated), the optical flow OF 0→1 (Second optical flow) will achieve better interpolation effect, but it also doubles the time consumption of optical flow calculation, which puts pressure on resource consumption. It can be flexibly adjusted according to the situation in practice.

[0151] Step 4: Obtain the image at time t by distortion. This application assumes that the object moves linearly. The motion vector and optical flow can be calculated by multiplying the motion vector and optical flow by the corresponding time. According to the obtained motion vector and the generated optical flow, the images at time 0 and time 1 are forward mapped to construct the image at time t. As shown in the above formula (4), the first image at time t can be obtained from the motion vector. (First motion vector prediction frame), as shown in the above formula (5), the second image at time t can be obtained from the motion vector (Second motion vector prediction frame), as shown in the above formula (6), the third image at time t can be obtained by optical flow (first optical flow prediction frame), if the bidirectional optical flow scheme is adopted, the fourth image at time t needs to be calculated using the above formula (7) (First optical flow prediction frame).

[0152] Step 5: Neural network calculation. and Combined into a tensor with 9 channels (if it is a bidirectional optical flow solution, you need to add The network is assembled into a 12-channel tensor. The subsequent network output also needs to be fine-tuned accordingly, which will not be described in detail here. This is used as the input of the neural network. The neural network in this application is intended to fuse images, so the network size can be kept small. In terms of network structure, this application uses a Unet-like backbone, which includes several upsampling and downsampling processes. This application has tried 3, 4, 5, and 6 upsampling times, all of which can achieve good results. It should be noted that this application adopts a unique Transformer Block design. The core purpose of this design is to extract the difference between the optical flow t-time map and the motion vector t-time map using an inter-frame attention mechanism. The specific implementation method is: we set any area in the optical flow t-time map as the query in the attention mechanism, and at the same time use all areas of the motion vector t-time map that are spatially adjacent to it as the key and value. This method can effectively compensate for the problem of inaccurate optical flow calculation, thereby improving the quality of interpolated frames. At a resolution of 1080P, this application can achieve a minimum computational load of 20GFlops, which can be easily performed in real time on mobile devices. In this application, the output of the neural network is two masks for fusion, denoted as mask1 and mask2, and a residual for micro-correction of the network, denoted as refine_delta.

[0153] Step 6: Fuse the images to obtain the intermediate frame I t The image obtained in step 4 is fused and calculated. The intermediate frame I can be calculated using formula (8): t .

[0154] In this way, this application combines traditional optical flow calculation algorithms with neural network interpolation, achieving the dual goals of excellent interpolation effects and low resource consumption. In its implementation, this application uses a traditional motion compensation algorithm to calculate optical flow, effectively controlling resource consumption. However, the optical flow calculated by traditional motion compensation algorithms suffers from poor accuracy. To address this issue, this application introduces an inter-frame attention mechanism to correct the accuracy issues associated with optical flow calculation. Furthermore, this application leverages the advantages of neural network interpolation by balancing the weights of preceding and following frames. Specifically, in situations such as object position changes or character movement in a 3D scene, the calculated weights significantly favor the image at time t obtained by forward mapping the motion vector. However, when faced with moving light and shadow or special effects, the calculated weights significantly favor the image at time t obtained by forward mapping the optical flow, thereby achieving high-quality interpolation effects. Furthermore, this application incorporates inverse motion vector calculation to address color differences between preceding and following frames due to different lighting and shadows, further improving interpolation quality.

[0155] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0156] The frame insertion processing method provided in the embodiment of the present application can be executed by a frame insertion processing device. In the embodiment of the present application, the frame insertion processing device provided in the embodiment of the present application is described by taking the frame insertion processing method performed by the frame insertion processing device as an example.

[0157] like Figure 7 As shown, an embodiment of the present application provides a frame insertion processing device 70 , which may include: an acquisition module 71 , a determination module 72 and a processing module 73 .

[0158] Among them, the acquisition module 71 can be used to acquire the first frame, the second frame, the first motion vector and the second motion vector, the second frame is adjacent to the first frame and after the first frame, the first motion vector is the motion vector from the second moment of capturing the second frame to the first moment of capturing the first frame, and the second motion vector is the motion vector from the first moment to the second moment; the determination module 72 can be used to determine at least three predicted frames corresponding to the third moment based on the first frame, the second frame, the first motion vector and the second motion vector, the third moment being the moment between the first moment and the second moment; and based on the at least three predicted frames, determine at least two weights, and the at least two weights are used to allocate the fusion processing ratio of the at least three predicted frames; the processing module 73 can be used to perform fusion processing on the at least three predicted frames based on the above-mentioned at least two weights to obtain the intermediate frame corresponding to the third moment.

[0159] In one possible implementation, the at least three prediction frames include two motion vector prediction frames and at least one optical flow prediction frame; the determination module 72 can be specifically used to input the at least three prediction frames into a neural network, and the neural network is used to determine the fusion weight of the image; and obtain the difference information between the at least one optical flow prediction frame and each motion vector prediction frame through the neural network; and determine at least two weights based on the difference information.

[0160] In one possible implementation, the acquisition module 71 can be specifically used to obtain the first frame, the second frame and the first motion vector from the rendering pipeline; and use the first motion vector to perform forward mapping processing on the first motion vector to obtain a third motion vector; and perform inversion processing on the third motion vector to obtain a fourth motion vector; and determine the fourth motion vector as the second motion vector; or, perform invalid value processing on the holes in the fourth motion vector, and determine the processed fourth motion vector as the second motion vector, and the holes are determined based on the first motion vector.

[0161] In one possible implementation, the at least three prediction frames include a first prediction frame, a second prediction frame, and a third prediction frame; the determination module 72 can be specifically used to perform forward mapping processing on the first frame according to the second motion vector and the third moment to obtain a first motion vector prediction frame; and perform forward mapping processing on the second frame according to the first motion vector and the third moment to obtain a second motion vector prediction frame; and determine the first optical flow from the first moment to the second moment based on the first frame and the second frame, and perform forward mapping processing on the first frame according to the first optical flow and the third moment to obtain a first optical flow prediction frame; and determine the first motion vector prediction frame, the second motion vector prediction frame, and the first optical flow prediction frame as at least three prediction frames.

[0162] In one possible implementation, determination module 72 may further be configured to determine a second optical flow from the second moment to the first moment based on the first frame and the second frame, and to perform forward mapping on the second frame based on the second optical flow and the third moment to obtain a second optical flow prediction frame. Determination module 72 may specifically be configured to determine the first motion vector predicted frame, the second motion vector predicted frame, the first optical flow predicted frame, and the second optical flow predicted frame as the at least three predicted frames.

[0163] In one possible implementation, the processing module 73 can be specifically used to determine the fusion weight corresponding to each predicted frame based on the above-mentioned at least two weights; and perform fusion processing on the above-mentioned at least three predicted frames based on the first information to obtain an intermediate frame corresponding to the third moment; wherein the first information includes any one of the following items: the fusion weight corresponding to each predicted frame; the fusion weight corresponding to each predicted frame and the residual determined based on the above-mentioned at least three predicted frames, and the residual is used to correct the image obtained by the fusion processing.

[0164] In the interpolation processing device provided in the embodiment of the present application, when performing interpolation processing, on the one hand, the interpolation processing device can analyze the motion vector of the object from the subsequent frame to the previous frame, and the motion vector of the object from the previous frame to the subsequent frame. After introducing the reverse motion vector, it can achieve bidirectional motion compensation. The use of the bidirectional motion vector can obtain the light and shadow changes between the previous and next frames, thereby eliminating the color difference caused by different lighting conditions. On the other hand, when determining the weight, the interpolation processing device is based on at least three predicted frames, rather than using a fixed setting rule as in the related art. Therefore, even in complex and changing scenes, the weight can be accurately assigned based on the predicted frame itself. This can improve the interpolation effect.

[0165] The interpolation processing device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0166] The frame insertion processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0167] The interpolation processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0168] like Figure 8As shown, an embodiment of the present application further provides an electronic device 100, including a processor 101 and a memory 102, wherein the memory 102 stores a program or instruction that can be run on the processor 101, and when the program or instruction is executed by the processor 101, the various steps of the above-mentioned interpolation processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0169] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0170] Figure 9 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0171] like Figure 9 As shown, the electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0172] Those skilled in the art will understand that the electronic device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0173] Among them, the processor 1010 can be used to obtain a first frame, a second frame, a first motion vector and a second motion vector, the second frame is adjacent to the first frame and after the first frame, the first motion vector is a motion vector from the second moment of capturing the second frame to the first moment of capturing the first frame, and the second motion vector is a motion vector from the first moment to the second moment; and based on the first frame, the second frame, the first motion vector and the second motion vector, determine at least three predicted frames corresponding to the third moment, the third moment being the moment between the first moment and the second moment; and based on the at least three predicted frames, determine at least two weights, the at least two weights being used to allocate the fusion processing ratio of the at least three predicted frames; and based on the above-mentioned at least two weights, perform fusion processing on the above-mentioned at least three predicted frames to obtain an intermediate frame corresponding to the third moment.

[0174] In one possible implementation, the at least three prediction frames include two motion vector prediction frames and at least one optical flow prediction frame; the processor 1010 can be specifically used to input the at least three prediction frames into a neural network, and the neural network is used to determine the fusion weight of the image; and obtain the difference information between the at least one optical flow prediction frame and each motion vector prediction frame through the neural network; and determine at least two weights based on the difference information.

[0175] In one possible implementation, the processor 1010 can be specifically used to obtain a first frame, a second frame and a first motion vector from a rendering pipeline; and use the first motion vector to perform forward mapping processing on the first motion vector to obtain a third motion vector; and perform inversion processing on the third motion vector to obtain a fourth motion vector; and determine the fourth motion vector as the second motion vector; or, perform invalid value processing on the holes in the fourth motion vector, and determine the processed fourth motion vector as the second motion vector, and the holes are determined based on the first motion vector.

[0176] In one possible implementation, the at least three prediction frames include a first prediction frame, a second prediction frame, and a third prediction frame; the processor 1010 can be specifically used to perform forward mapping processing on the first frame according to the second motion vector and the third moment to obtain a first motion vector prediction frame; and perform forward mapping processing on the second frame according to the first motion vector and the third moment to obtain a second motion vector prediction frame; and determine the first optical flow from the first moment to the second moment based on the first frame and the second frame, and perform forward mapping processing on the first frame according to the first optical flow and the third moment to obtain a first optical flow prediction frame; and determine the first motion vector prediction frame, the second motion vector prediction frame, and the first optical flow prediction frame as at least three prediction frames.

[0177] In one possible implementation, processor 1010 may be further configured to determine a second optical flow from the second moment to the first moment based on the first frame and the second frame, and perform forward mapping on the second frame based on the second optical flow and the third moment to obtain a second optical flow prediction frame. Processor 1010 may specifically be configured to determine the first motion vector predicted frame, the second motion vector predicted frame, the first optical flow predicted frame, and the second optical flow predicted frame as the at least three predicted frames.

[0178] In one possible implementation, the processor 1010 can be specifically used to determine the fusion weight corresponding to each predicted frame based on the above-mentioned at least two weights; and perform fusion processing on the above-mentioned at least three predicted frames based on the first information to obtain an intermediate frame corresponding to the third moment; wherein the first information includes any one of the following items: the fusion weight corresponding to each predicted frame; the fusion weight corresponding to each predicted frame and the residual determined based on the above-mentioned at least three predicted frames, and the residual is used to correct the image obtained by the fusion processing.

[0179] In the electronic device provided in the embodiment of the present application, when performing frame interpolation, on the one hand, the electronic device can analyze the motion vector of the object from the subsequent frame to the previous frame, and the motion vector of the object from the previous frame to the subsequent frame. After introducing the reverse motion vector, bidirectional motion compensation is achieved. The use of the bidirectional motion vector can obtain the light and shadow changes between the previous and next frames, thereby eliminating the color difference caused by different lighting conditions. On the other hand, when determining the weight, the electronic device is based on at least three predicted frames, rather than using a fixed setting rule as in the related art. Therefore, even in complex and changing scenes, the weight can be accurately assigned by the predicted frame itself. This can improve the interpolation effect during interpolation.

[0180] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0181] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0182] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0183] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned interpolation processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0184] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0185] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned interpolation processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0186] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0187] An embodiment of the present application provides a computer program / program product, which is stored in a storage medium. The program / program product is executed by at least one processor to implement the various processes of the above-mentioned interpolation processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0188] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0190] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A frame insertion processing method, characterized in that: The method comprises: Acquire a first frame, a second frame, a first motion vector, and a second motion vector, where the second frame is adjacent to and subsequent to the first frame, the first motion vector is a motion vector from a second moment when the second frame is acquired to a first moment when the first frame is acquired, and the second motion vector is a motion vector from the first moment to the second moment; Determining at least three predicted frames corresponding to a third moment based on the first frame, the second frame, the first motion vector, and the second motion vector, where the third moment is a moment between the first moment and the second moment; Determining at least two weights based on the at least three predicted frames, wherein the at least two weights are used to allocate fusion processing ratios of the at least three predicted frames; Based on the at least two weights, a fusion process is performed on the at least three predicted frames to obtain an intermediate frame corresponding to the third moment.

2. The method according to claim 1, characterized in that The at least three prediction frames include two motion vector prediction frames and at least one optical flow prediction frame; The determining of at least two weights based on the at least three predicted frames includes: Inputting the at least three predicted frames into a neural network, wherein the neural network is used to determine a fusion weight of the image; Obtaining difference information between the at least one optical flow prediction frame and each motion vector prediction frame through the neural network; The at least two weights are determined according to the difference information.

3. The method according to claim 1, characterized in that The acquiring of the first frame, the second frame, the first motion vector and the second motion vector includes: Obtain the first frame, the second frame, and the first motion vector from a rendering pipeline; performing a forward mapping process on the first motion vector using the first motion vector to obtain a third motion vector; performing an inversion process on the third motion vector to obtain a fourth motion vector; determining the fourth motion vector as the second motion vector; or, Null value processing is performed on a hole in the fourth motion vector, and the processed fourth motion vector is determined as the second motion vector, wherein the hole is determined based on the first motion vector.

4. The method according to claim 1, wherein The determining, based on the first frame, the second frame, the first motion vector, and the second motion vector, at least three predicted frames corresponding to a third moment includes: Performing forward mapping processing on the first frame according to the second motion vector and the third moment to obtain a first motion vector predicted frame; Performing forward mapping processing on the second frame according to the first motion vector and the third moment to obtain a second motion vector predicted frame; Determining a first optical flow from the first moment to the second moment based on the first frame and the second frame, and performing forward mapping processing on the first frame based on the first optical flow and the third moment to obtain a first optical flow prediction frame; The first motion vector prediction frame, the second motion vector prediction frame, and the first optical flow prediction frame are determined as the at least three prediction frames.

5. The method according to claim 4, characterized in that The method further comprises: Determining a second optical flow from the second moment to the first moment based on the first frame and the second frame, and performing forward mapping processing on the second frame based on the second optical flow and the third moment to obtain a second optical flow prediction frame; The step of determining the first motion vector predicted frame, the second motion vector predicted frame, and the first optical flow predicted frame as the at least three predicted frames includes: The first motion vector prediction frame, the second motion vector prediction frame, the first optical flow prediction frame, and the second optical flow prediction frame are determined as the at least three prediction frames.

6. The method according to claim 1, characterized in that The performing a fusion process on the at least three predicted frames based on the at least two weights to obtain an intermediate frame corresponding to the third moment includes: Determining a fusion weight corresponding to each predicted frame according to the at least two weights; performing a fusion process on the at least three predicted frames according to the first information to obtain an intermediate frame corresponding to the third moment; The first information includes any one of the following: The fusion weight corresponding to each predicted frame; The fusion weight corresponding to each predicted frame and the residual determined according to the at least three predicted frames, the residual being used to correct the image obtained by the fusion process.

7. A frame insertion processing device, characterized in that: The device includes: an acquisition module, a determination module and a processing module; The acquisition module is configured to acquire a first frame, a second frame, a first motion vector, and a second motion vector, wherein the second frame is adjacent to and subsequent to the first frame, the first motion vector is a motion vector from a second moment when the second frame is acquired to a first moment when the first frame is acquired, and the second motion vector is a motion vector from the first moment to the second moment; The determining module is configured to determine, based on the first frame, the second frame, the first motion vector, and the second motion vector, at least three predicted frames corresponding to a third moment, where the third moment is a moment between the first moment and the second moment; and determine, based on the at least three predicted frames, at least two weights used to allocate fusion processing ratios for the at least three predicted frames; The processing module is configured to perform fusion processing on the at least three predicted frames based on the at least two weights to obtain an intermediate frame corresponding to the third moment.

8. The device according to claim 7, characterized in that The at least three prediction frames include two motion vector prediction frames and at least one optical flow prediction frame; The determination module is specifically used to input the at least three prediction frames into a neural network, and the neural network is used to determine the fusion weight of the image; and obtain the difference information between the at least one optical flow prediction frame and each motion vector prediction frame through the neural network; and determine the at least two weights based on the difference information.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the frame insertion processing method according to any one of claims 1 to 6 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the frame insertion processing method according to any one of claims 1 to 6 are implemented.

Images