Image generation method and apparatus based on video frame interpolation, and computer device

Abstract

The present application relates to an image generation method and apparatus based on video frame interpolation, and a computer device. The method comprises: acquiring two video frame images, and on the basis of the displacement of pixel points in the two video frame images, determining first optical flow information between the two video frame images (202); on the basis of the two video frame images and the first optical flow information, predicting an intermediate frame image between the two video frame images to obtain a predicted intermediate frame image (204); for each of the two video frame images, on the basis of the video frame image and the displacement of pixel points in the predicted intermediate frame image, determining second optical flow information between the video frame image and the predicted intermediate frame image (206); for each first pixel point in the predicted intermediate frame image, acquiring, from the second optical flow information, a position offset corresponding to the first pixel point, and on the basis of the position offset, selecting an image block corresponding to the first pixel point from the video frame image (208); on the basis of an image block corresponding to each first pixel point in the predicted intermediate frame image, updating a pixel value at each first pixel point in the predicted intermediate frame image to obtain an updated intermediate frame image corresponding to the video frame image (210); and fusing the updated intermediate frame images corresponding to the two video frame images respectively to obtain a fused intermediate frame image (212).

Description

Video interpolation image generation method, device and computer equipment

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023115105221, filed on November 14, 2023, entitled “Video interpolation image generation method, device, computer equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method and apparatus for generating a video interpolation image, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0004] With the development of computer technology, video interpolation technology has emerged. Video interpolation technology is a technology that generates intermediate frame images between existing video frame images. It is used to increase the frame rate of the video, make the video playback smoother, or be used for video repair and editing.

[0005] In traditional technology, video interpolation is mainly based on the optical flow method, which estimates the optical flow information of the intermediate frame image and the previous and next frame images through the optical flow information between the previous and next frame images, and then uses image warping to further calculate the intermediate frame image based on the optical flow information of the intermediate frame image and the previous and next frame images.

[0006] However, the accuracy of optical flow estimation in traditional methods is affected by many factors, such as illumination changes, occlusion, motion blur, etc., which may lead to inaccurate estimated optical flow information and thus poor image quality of the generated intermediate frame images.

[0007] Summary of the Invention

[0008] Embodiments of the present application provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating a video interpolation image.

[0009] A method for generating a video interpolation image, executed by a computer device, comprising:

[0010] Acquire two video frame images, and determine first optical flow information between the two video frame images based on the displacement of pixels in the two video frame images;

[0011] Predicting an intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information to obtain a predicted intermediate frame image;

[0012] For each of the two video frame images, determining second optical flow information between the targeted video frame image and the predicted intermediate frame image based on displacements of pixels in the targeted video frame image and the predicted intermediate frame image;

[0013] For each first pixel in the predicted intermediate frame image, obtain a position offset corresponding to the first pixel from the second optical flow information, and select an image block corresponding to the first pixel from the targeted video frame image based on the position offset;

[0014] Based on the image blocks corresponding to the first pixels in the predicted intermediate frame image, pixel values ​​at the first pixels in the predicted intermediate frame image are updated to obtain updated intermediate frame images corresponding to the targeted video frame image; and

[0015] The updated intermediate frame images corresponding to the two video frame images are fused to obtain a fused intermediate frame image for insertion between the two video frame images.

[0016] A video interpolation image generation device, the device comprising:

[0017] A first optical flow calculation module is used to obtain two video frame images and determine first optical flow information between the two video frame images based on the displacement of pixels in the two video frame images;

[0018] a prediction module, configured to predict an intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information, to obtain a predicted intermediate frame image;

[0019] a second optical flow calculation module, configured to determine, for each of the two video frame images, second optical flow information between the targeted video frame image and the predicted intermediate frame image based on displacements of pixels in the targeted video frame image and the predicted intermediate frame image;

[0020] an image block selection module, configured to obtain, for each first pixel in the predicted intermediate frame image, a position offset corresponding to the first pixel from the second optical flow information, and select, from the targeted video frame image, an image block corresponding to the first pixel based on the position offset;

[0021] an updating module, configured to update the pixel values ​​of the first pixels in the predicted intermediate frame image based on the image blocks corresponding to the first pixels in the predicted intermediate frame image, respectively, to obtain an updated intermediate frame image corresponding to the target video frame image; and

[0022] The fusion module is used to fuse the updated intermediate frame images corresponding to the two video frame images to obtain a fused intermediate frame image for insertion between the two video frame images.

[0023] A computer device includes a memory and one or more processors, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the one or more processors execute the steps of the above-mentioned video interpolation image generation method.

[0024] One or more non-volatile computer-readable storage media storing computer-readable instructions, on which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned video interpolation image generation method.

[0025] A computer program product or computer program includes computer-readable instructions stored in a computer-readable storage medium. One or more processors of a computer device read the computer-readable instructions from the computer-readable storage medium. The one or more processors execute the computer-readable instructions, causing the computer device to perform the steps of the above-mentioned video interpolation image generation method.

[0026] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0028] FIG1 is a diagram showing an application environment of a method for generating a video interpolation image in one embodiment;

[0029] FIG2 is a schematic flow chart of a method for generating a video interpolation image in one embodiment;

[0030] FIG3 is a schematic diagram of optical flow information in one embodiment;

[0031] FIG4 is a schematic diagram of an image block selected in one embodiment;

[0032] FIG5 is a schematic diagram of an image block selected in another embodiment;

[0033] FIG6 is a schematic diagram of an image block selected in yet another embodiment;

[0034] FIG7 is a schematic diagram of a first pixel position and a fourth pixel point in one embodiment;

[0035] FIG8 is a schematic diagram of a first pixel position and a fourth pixel point in another embodiment;

[0036] FIG9 is a schematic diagram of frame interpolation based on optical flow modeling based on a linear motion model in one embodiment;

[0037] FIG10 is a schematic diagram of an intermediate frame of optical flow prediction using a nonlinear motion model according to an embodiment;

[0038] FIG11 is a schematic diagram of an image warping operation according to one embodiment;

[0039] FIG12 is a schematic diagram of a transformer (a deep learning model) model in one embodiment;

[0040] FIG13 is a schematic diagram of implicit alignment based on a transformer structure in one embodiment;

[0041] FIG14 is a schematic diagram illustrating an application of a method for generating a video interpolation image in one embodiment;

[0042] FIG15 is a structural block diagram of a video interpolation image generating apparatus according to an embodiment;

[0043] FIG16 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] The video interpolation image generation method provided in the embodiments of the present application can be applied in the application environment shown in FIG1 . Terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed in the cloud or on another server. The terminal 102 obtains two video frame images from the server 104, determines first optical flow information between the two video frame images based on the displacement of pixels in the two video frame images, predicts an intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information, and obtains a predicted intermediate frame image. For each video frame image in the two video frame images, second optical flow information between the target video frame image and the predicted intermediate frame image is determined based on the displacement of pixels in the target video frame image and the predicted intermediate frame image. For each first pixel in the predicted intermediate frame image, a position offset corresponding to the target first pixel is obtained from the second optical flow information. Based on the position offset, an image block corresponding to the target first pixel is selected from the target video frame image. Based on the image block corresponding to each first pixel in the predicted intermediate frame image, the pixel value at each first pixel in the predicted intermediate frame image is updated to obtain an updated intermediate frame image corresponding to the target video frame image. The updated intermediate frame images corresponding to the two video frame images are fused to obtain a fused intermediate frame image for insertion between the two video frame images.

[0046] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0047] In one embodiment, as shown in FIG2 , a method for generating a video interpolation image is provided. The method can be executed by a terminal or a server alone, or by a terminal and a server in collaboration. In the embodiment of the present application, the method is described using the terminal as an example, and includes the following steps:

[0048] Step 202: Acquire two video frame images, and determine first optical flow information between the two video frame images based on the displacement of pixels in the two video frame images.

[0049] Here, a video frame image refers to a frame image obtained from a video, that is, an image of a certain frame obtained from a video. Two video frame images refer to two frames of images obtained from a video, including a first video frame image and a second video frame image, wherein the first video frame image is the preceding frame of the second video frame image, that is, the first video frame image and the second video frame image are the two frames before and after in the video, and the first video frame image is before the second video frame image in the timing of the video. It should be noted that the first video frame image and the second video frame image can be two consecutive frames before and after, or two discontinuous frames before and after.

[0050] Optical flow is the instantaneous speed of pixel motion of a moving object on the observation plane. Some also define optical flow as the instantaneous rate of change of the grayscale / brightness of a specific pixel within a two-dimensional image. When the time interval is very small (such as between consecutive video frames), it is equivalent to the displacement of the projection of a spatial point within the imaging plane. Generally speaking, optical flow is caused by relative motion caused by the movement of the foreground object in the scene, the movement of the camera, or a combination of both. In layman's terms, when the human eye observes a moving object, the scene forms a series of continuously changing images on the retina. This series of continuously changing information continuously "flows" through the retina (i.e., the image plane) like a "stream" of light, hence the name optical flow. It can also be understood as the flow of pixel intensity within an image. Optical flow represents changes in the image. Because it contains information about the object's motion, it can be used by the observer to determine the object's motion. The first optical flow information is used to describe the movement of pixels in the two video frame images, that is, the instantaneous speed of the movement of pixels on the two video frame images. When the two video frame images are two consecutive frames, the first optical flow information can specifically be the displacement of pixels in the two video frame images.

[0051] Specifically, the terminal acquires two video frames and determines the first optical flow information between the two video frames based on the displacement of pixels in the two video frames. In a specific application, the terminal may first acquire a video and then acquire two video frames from the video. The two acquired video frames may be consecutive or discontinuous.

[0052] In a specific application, the first optical flow information includes forward optical flow information and backward optical flow information, wherein the forward optical flow information refers to the optical flow information from the video frame image at an earlier time to the video frame image at a later time, and the backward optical flow information refers to the optical flow information from the video frame image at a later time to the video frame image at an earlier time. For example, assuming that the two video frames include a first video frame image and a second video frame image, and the first video frame image is the forward frame of the second video frame image, then the forward optical flow information refers to the optical flow information from the first video frame image to the second video frame image, and the backward optical flow information refers to the optical flow information from the second video frame image to the first video frame image.

[0053] In a specific application, when determining the first optical flow information between two video frame images, if the forward optical flow information is determined, for each pixel point in the video frame image at an earlier moment, the terminal will determine the position change (i.e., displacement) of the targeted pixel point from the video frame image at an earlier moment to the video frame image at a later moment based on the position of the targeted pixel point in the video frame image at an earlier moment and the position of the targeted pixel point in the video frame image at a later moment, and obtain the optical flow information of the targeted pixel point, and obtain the forward optical flow information based on the optical flow information of each pixel point.

[0054] In a specific application, if the backward optical flow information is determined, for each pixel in the video frame image at a later time, the terminal will determine the position change (i.e., displacement) of the targeted pixel from the video frame image at a later time to the video frame image at an earlier time based on the position of the targeted pixel in the video frame image at a later time and the position of the targeted pixel in the video frame image at an earlier time, and obtain the optical flow information of the targeted pixel, and obtain the backward optical flow information based on the optical flow information of each pixel.

[0055] In a specific application, as shown in FIG3 , it is assumed that the two video frame images include a first video frame image and a second video frame image, and the first video frame image is a forward frame of the second video frame image. Taking the pixel point A in the first video frame image as an example, its position in the first video frame image is (x1, y1), and its position in the second video frame image after movement is (x2, y2). When calculating the forward optical flow information, the optical flow information of the pixel point A can be (u x ,u y )=(x2,y2)-(x1,y1), vector (u x ,u y ) is the optical flow information generated by pixel A, which includes the movement in the x-direction and the y-direction, that is, the displacement of pixel A. The value of the optical flow is a sub-pixel floating-point value, as shown in Figure 3. The arrow represents the optical flow vector of pixel A between the first video frame image and the second video frame image.

[0056] Step 204 : Based on the two video frame images and the first optical flow information, predict an intermediate frame image between the two video frame images to obtain a predicted intermediate frame image.

[0057] The intermediate frame image refers to the image at the intermediate frame moment inserted between two video frame images, and the predicted intermediate frame image refers to the intermediate frame image determined by prediction.

[0058] Specifically, based on the forward optical flow information and backward optical flow information in the first optical flow information, and the intermediate frame moment corresponding to the intermediate frame image, the terminal can estimate the intermediate frame optical flow information corresponding to each of the two video frame images through optical flow estimation, and then predict the intermediate frame image between the two video frame images based on the two video frame images and the intermediate frame optical flow information corresponding to the two video frame images to obtain the predicted intermediate frame image.

[0059] In specific applications, the intermediate frame moment refers to the moment in the video corresponding to the intermediate frame image. It can be understood that since the intermediate frame image is a frame image between two video frame images, the intermediate frame moment is the moment between the two moments corresponding to the two video frame images. For example, when the two video frame images include a first video frame image and a second video frame image, and the first video frame image is the forward frame of the second video frame image, the intermediate frame moment refers to the moment between the first moment in the video corresponding to the first video frame image and the second moment in the video corresponding to the second video frame image. For example, when the first moment is 1 minute and the second moment is 1 minute 30 seconds, the intermediate frame moment can be any moment between 1 minute and 1 minute 30 seconds, such as 1 minute 10 seconds, 1 minute 15 seconds, 1 minute 20 seconds, etc.

[0060] In specific applications, the terminal can also use a pre-trained intermediate frame image prediction model to predict the intermediate frame image between two video frame images. By inputting the two video frame images and the first optical flow information into the pre-trained intermediate frame image prediction model, the intermediate frame image prediction model can output the predicted intermediate frame image. The pre-trained intermediate frame image prediction model mainly predicts the intermediate frame image between two video frame images through multiple convolution processes. The specific model structure can be selected according to the actual application scenario and is not limited to this in this embodiment.

[0061] Step 206 : For each of the two video frame images, determine second optical flow information between the targeted video frame image and the predicted intermediate frame image based on the displacement of pixels in the targeted video frame image and the predicted intermediate frame image.

[0062] Among them, the second optical flow information is used to describe the movement of pixel points in the targeted video frame image and the predicted intermediate frame image, that is, the instantaneous speed of the movement of pixel points on the targeted video frame image and the predicted intermediate frame image. The second optical flow information can specifically be the displacement of pixel points in the targeted video frame image and the predicted intermediate frame image.

[0063] Specifically, for each of the two video frames, the terminal determines second optical flow information between the targeted video frame and the predicted intermediate frame based on the displacement of pixels in the targeted video frame and the predicted intermediate frame. In specific applications, the second optical flow information can be optical flow information from the targeted video frame to the predicted intermediate frame, or from the predicted intermediate frame to the targeted video frame.

[0064] In a specific application, if the second optical flow information is the optical flow information from the targeted video frame image to the predicted intermediate frame image, for each pixel in the targeted video frame image, the terminal will determine the targeted pixel based on the position of the targeted pixel in the targeted video frame image and the position in the predicted intermediate frame image, and obtain the optical flow information of the targeted pixel based on the position change (i.e., displacement) from the targeted video frame image to the predicted intermediate frame image, and obtain the second optical flow information based on the optical flow information of each pixel.

[0065] In a specific application, if the second optical flow information is the optical flow information from the predicted intermediate frame image to the targeted video frame image, for each first pixel point in the predicted intermediate frame image, the terminal will determine the position change (i.e., displacement) of the targeted first pixel point from the predicted intermediate frame image to the targeted video frame image based on the position of the targeted first pixel point in the predicted intermediate frame image and the position in the targeted video frame image, and obtain the optical flow information of the targeted first pixel point, and obtain the second optical flow information based on the optical flow information of each first pixel point.

[0066] In step 208, for each first pixel point in the predicted intermediate frame image, a position offset corresponding to the first pixel point is obtained from the second optical flow information, and an image block corresponding to the first pixel point is selected from the video frame image based on the position offset.

[0067] The first pixel refers to a pixel on the predicted intermediate frame image. The image block refers to the portion of the image corresponding to the first pixel on the target video frame image, that is, the portion of the target video frame image corresponding to the first pixel. It can be understood that the image block includes at least one pixel, that is, the image block is formed by at least one pixel on the target video frame image.

[0068] Specifically, for each first pixel in the predicted intermediate frame image, the terminal can obtain the position offset corresponding to the targeted first pixel from the second optical flow information, and then select the image block corresponding to the targeted first pixel from the targeted video frame image based on the position offset. The position offset refers to the position offset of the targeted first pixel between the targeted video frame image and the predicted intermediate frame image, that is, the displacement of the targeted first pixel between the targeted video frame image and the predicted intermediate frame image. It should be noted that the second optical flow information includes the position offset corresponding to each first pixel in the predicted intermediate frame image, and the terminal only needs to directly obtain it.

[0069] In specific applications, the size of the image block can be configured according to the actual application scenario. For example, the image block can be specifically formed by N pixels, where N is a positive integer. Then, based on the position offset, the terminal will select N pixels corresponding to the first pixel in the target video frame image as the image block corresponding to the first pixel.

[0070] Step 210 : Based on the image blocks corresponding to the first pixels in the predicted intermediate frame image, the pixel values ​​at the first pixels in the predicted intermediate frame image are updated to obtain updated intermediate frame images corresponding to the corresponding video frame images.

[0071] Specifically, for each first pixel in the predicted intermediate frame image, the terminal updates the pixel value at the first pixel based on the image block corresponding to the first pixel, obtaining an updated pixel for the first pixel. Based on the updated pixel value at each first pixel in the predicted intermediate frame image, an updated intermediate frame image corresponding to the video frame image is obtained. It should be understood that in this embodiment, the updated intermediate frame image corresponding to the video frame image refers to the intermediate frame image obtained by updating the pixel value at each first pixel in the video frame image using the image block corresponding to each first pixel.

[0072] In a specific application, the image block corresponding to the first pixel includes multiple second pixels, and the terminal can update the pixel value at the first pixel based on the pixel values ​​at the multiple second pixels. In a specific application, the terminal can obtain pixel features corresponding to the multiple second pixels based on the pixel values ​​at the multiple second pixels, and then use the pixel features corresponding to the multiple second pixels to update the pixel features corresponding to the first pixel, so that the updated pixel value at the first pixel can be obtained through the updated pixel features.

[0073] In a specific application, since the position of the first pixel point in the predicted intermediate frame image and the position of the image block corresponding to the first pixel point in the video frame image are related, when updating the pixel value at the first pixel point, the terminal will obtain the position information of the image block corresponding to the first pixel point from the video frame image, and then update the pixel value at the first pixel point based on the image block corresponding to the first pixel point and the position information.

[0074] In a specific application, when the image block corresponding to the first pixel point includes multiple second pixel points, the position information includes the pixel positions of each of the multiple second pixel points. The terminal can update the pixel value at the first pixel point based on the pixel values ​​at the multiple second pixel points and the pixel positions of each of the multiple second pixel points.

[0075] Step 212: Fusing the updated intermediate frame images corresponding to the two video frame images to obtain a fused intermediate frame image for insertion between the two video frame images.

[0076] Specifically, the terminal fuses the updated intermediate frame images corresponding to the two video frames to obtain a fused intermediate frame image for insertion between the two video frames. It can be understood that by fusing the updated intermediate frame images corresponding to the two video frames, the features of the updated intermediate frames can be complementary, resulting in a fused intermediate frame image with richer features and higher image quality, thereby improving the image quality of the generated intermediate frame images.

[0077] In a specific application, when performing fusion, the terminal will respectively extract features of the updated intermediate frame images corresponding to the two video frame images, obtain the updated intermediate frame image features corresponding to the two video frame images, and then fuse the updated intermediate frame image features corresponding to the two video frame images to obtain a fused intermediate frame image for insertion between the two video frame images.

[0078] The above-mentioned video interpolation image generation method obtains two video frame images, determines the first optical flow information between the two video frame images according to the displacement of the pixel points in the two video frame images, and can realize the prediction of the intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information to obtain the predicted intermediate frame image. By for each video frame image in the two video frame images, according to the displacement of the pixel points in the targeted video frame image and the predicted intermediate frame image, the second optical flow information between the targeted video frame image and the predicted intermediate frame image is determined. For each first pixel point in the predicted intermediate frame image, the corresponding position offset of the targeted first pixel point is obtained from the second optical flow information, and according to the position offset, the corresponding position offset of the targeted first pixel point is obtained from the targeted pixel point. In the video frame image, the image block corresponding to the first pixel point is selected, and the pixel value at each first pixel point in the predicted intermediate frame image is updated respectively based on the image block corresponding to each first pixel point in the predicted intermediate frame image. The morphological features and dynamic features between adjacent video frame images can be learned through implicit image and feature alignment, so as to better model the dynamic changes and evolution process of adjacent video frame images, and obtain the updated intermediate frame image corresponding to the video frame image with high image quality. Furthermore, the updated intermediate frame images corresponding to the two video frame images can be fused to obtain a fused intermediate frame image for insertion between the two video frame images, which can improve the image quality of the generated intermediate frame image.

[0079] In one embodiment, based on the image blocks corresponding to the first pixels in the predicted intermediate frame images, respectively updating the pixel values ​​at the first pixels in the predicted intermediate frame images to obtain the updated intermediate frame images corresponding to the video frame images includes:

[0080] For each first pixel point in the predicted intermediate frame image, the position information of the image block corresponding to the first pixel point is obtained from the targeted video frame image, and based on the image block corresponding to the first pixel point and the position information, the pixel value at the targeted first pixel point is updated to obtain the updated pixel value at the targeted first pixel point, and according to the updated pixel value at each first pixel point in the predicted intermediate frame image, the updated intermediate frame image corresponding to the targeted video frame image is obtained.

[0081] The position information is used to describe the position of the image block in the target video frame image, and can be obtained by the pixel position of the second pixel point in the image block in the target video frame image. For example, when the image block is a single second pixel point, the position information can be the pixel position of the single second pixel point in the target video frame image. When the image block includes multiple second pixels, the position information can be the pixel position of each of the multiple second pixels.

[0082] Specifically, for each first pixel in the predicted intermediate frame image, since the position of the first pixel in the predicted intermediate frame image and the position of the image block corresponding to the first pixel in the video frame image are related, when updating the pixel value at the first pixel, the terminal obtains the position information of the image block corresponding to the first pixel from the video frame image, and then updates the pixel value at the first pixel based on the image block and position information corresponding to the first pixel to obtain the updated pixel value at the first pixel. After obtaining the updated pixel value at each first pixel in the predicted intermediate frame image, the updated intermediate frame image corresponding to the video frame image can be obtained based on the updated pixel value at each first pixel in the predicted intermediate frame image, that is, the intermediate frame image obtained after updating the pixel value at each first pixel in the predicted intermediate frame image.

[0083] In a specific application, the terminal can first perform feature extraction based on the image block and position information corresponding to the first pixel point, obtain the image block feature corresponding to the image block, and generate the pixel point feature corresponding to the first pixel point, and then use the image block feature to update the pixel point feature corresponding to the first pixel point, and use the updated pixel point feature to update the pixel value at the first pixel point to obtain the updated pixel value at the first pixel point.

[0084] Among them, using the image block features to update the pixel features corresponding to the first pixel point can be understood as implicit feature alignment of the image block features and the pixel features corresponding to the first pixel point, and representing the pixel features corresponding to the first pixel point by the image block features.

[0085] In a specific application, when the image block is a single second pixel, the position information may be the pixel position of the single second pixel in the target video frame image. The terminal then performs feature extraction based on the pixel value at the single second pixel and the pixel position of the single second pixel to obtain the image block features. When the image block includes multiple second pixels, it is necessary to combine the pixel values ​​at the multiple second pixels and the pixel positions of the multiple second pixels to determine the image block features.

[0086] In this embodiment, for each first pixel point in the predicted intermediate frame image, by obtaining the position information of the image block corresponding to the first pixel point from the targeted video frame image, the image block corresponding to the first pixel point and the position information can be combined to accurately update the pixel value at the targeted first pixel point, and then the updated intermediate frame image corresponding to the targeted video frame image can be determined based on the updated pixel value at each first pixel point in the predicted intermediate frame image, so as to obtain an updated intermediate frame image with high image quality, so as to generate a fused intermediate frame image with high image quality based on the updated intermediate frame image.

[0087] In one embodiment, the image block corresponding to the first pixel point includes multiple second pixel points, and the position information includes the pixel position of each of the multiple second pixel points. Based on the image block corresponding to the first pixel point and the position information, the pixel value at the first pixel point is updated to obtain the updated pixel value at the first pixel point, including: for each second pixel point among the multiple second pixel points, feature extraction is performed based on the pixel value at the second pixel point and the pixel position of the second pixel point to obtain the pixel feature corresponding to the second pixel point; based on the pixel feature corresponding to each of the multiple second pixel points, the pixel value at the first pixel point is updated to obtain the updated pixel value at the first pixel point.

[0088] The pixel feature corresponding to the second pixel refers to a feature used to characterize the second pixel. For example, the pixel feature corresponding to the second pixel may be a feature vector used to characterize the second pixel.

[0089] Specifically, the image block corresponding to the first pixel point includes multiple second pixel points, and the position information includes the pixel position of each of the multiple second pixel points. Then, when the pixel value at the first pixel point is updated based on the image block corresponding to the first pixel point and the position information, for each of the multiple second pixel points, the terminal will perform feature extraction based on the pixel value at the second pixel point and the pixel position of the second pixel point to obtain the pixel feature corresponding to the second pixel point, and generate the pixel feature corresponding to the first pixel point. Then, based on the pixel feature corresponding to each of the multiple second pixel points, the pixel feature corresponding to the first pixel point is updated, and the pixel value at the first pixel point is updated through the updated pixel feature to obtain the updated pixel value at the first pixel point.

[0090] In a specific application, the terminal can use a pre-trained pixel feature extraction network to extract features of the pixel value at the second pixel and the pixel position at the second pixel, and input the pixel value and the pixel position into the pre-trained pixel feature extraction network to obtain the pixel features corresponding to the second pixel. The pre-trained pixel feature extraction network can be configured according to the actual application scenario. In addition, the terminal can also obtain the pixel features corresponding to the second pixel by first extracting features of the pixel value at the second pixel and the pixel position at the second pixel, respectively, and then fusing the two extracted features.

[0091] In a specific application, the pixel features corresponding to the first pixel point are updated based on the pixel features corresponding to each of the multiple second pixel points. It can also be understood as implicit feature alignment of the pixel features corresponding to each of the multiple second pixel points and the pixel features corresponding to the first pixel point, and the pixel features corresponding to the first pixel point are represented by referring to the pixel features corresponding to the first pixel point.

[0092] In this embodiment, for each of the multiple second pixel points, feature extraction is performed using the pixel value at the targeted second pixel point and the pixel position of the targeted second pixel point, so that a pixel feature that can accurately characterize the targeted second pixel point can be obtained, and then implicit feature alignment can be performed using the pixel features corresponding to each of the multiple second pixel points, and the pixel value at the targeted first pixel point can be updated to obtain an updated pixel value at the targeted first pixel point.

[0093] In one embodiment, feature extraction is performed based on the pixel value at the second pixel point and the pixel position of the second pixel point to obtain the pixel feature corresponding to the second pixel point, including: encoding the pixel value at the second pixel point to obtain the pixel value code corresponding to the second pixel point, and encoding the pixel position of the second pixel point to obtain the position code corresponding to the second pixel point, and fusing the pixel value code and the position code corresponding to the second pixel point to obtain the pixel feature corresponding to the second pixel point.

[0094] Specifically, in order to obtain the pixel point feature corresponding to the second pixel point, the terminal will encode the pixel value at the second pixel point to obtain the pixel value code corresponding to the second pixel point, and encode the pixel point position of the second pixel point to obtain the position code corresponding to the second pixel point. Finally, the pixel value code and position code corresponding to the second pixel point are merged to obtain the pixel point feature corresponding to the second pixel point.

[0095] In a specific application, the terminal can encode pixel values ​​using a pre-trained first encoding network to obtain pixel value codes, and encode pixel positions using a pre-trained second encoding network to obtain position codes. The pre-trained first encoding network and the pre-trained second encoding network can be trained according to actual application scenarios. In this embodiment, the training of the first encoding network and the second encoding network is not limited.

[0096] In a specific application, the terminal can fuse the pixel value code and position code corresponding to the second pixel point by splicing the pixel value code and position code corresponding to the second pixel point to obtain the pixel point feature corresponding to the second pixel point, that is, the pixel point feature corresponding to the second pixel point can be in the form of pixel value code + position code.

[0097] In this embodiment, by encoding the pixel value at the second pixel point, the pixel value code corresponding to the second pixel point can be obtained, and by encoding the pixel position of the second pixel point, the position code corresponding to the second pixel point can be obtained. Then, by fusing the pixel value code and the position code, the pixel point features corresponding to the second pixel point can be obtained.

[0098] In one embodiment, based on the pixel features corresponding to each of the multiple second pixel points, the pixel value at the first pixel point is updated to obtain the updated pixel value at the first pixel point, including: performing feature extraction based on the pixel value at the first pixel point and the corresponding position offset of the first pixel point to obtain the pixel features corresponding to the first pixel point; based on the attention mechanism, performing feature interaction processing on the pixel features corresponding to each of the multiple second pixel points and the pixel features corresponding to the first pixel point to obtain the updated pixel value at the first pixel point.

[0099] Among them, the attention mechanism is a resource allocation scheme that allocates computing resources to more important tasks when computing power is limited, while solving the problem of information overload. In neural network learning, generally speaking, the more parameters a model has, the stronger its expressive power is, and the more information the model stores, but this will bring about the problem of information overload. Then, by introducing the attention mechanism, focusing on the information that is more critical to the current task among the numerous input information, reducing the attention to other information, and even filtering out irrelevant information, the problem of information overload can be solved and the efficiency and accuracy of task processing can be improved. In this embodiment, the focus is on the information that is more critical to the pixel feature corresponding to the first pixel point among the pixel features corresponding to each of the multiple second pixels, so as to accurately update the pixel value at the first pixel point.

[0100] Specifically, when updating the pixel value at the first pixel point, the terminal will first perform feature extraction based on the pixel value at the first pixel point and the corresponding position offset of the first pixel point to obtain the pixel point features corresponding to the first pixel point, and then based on the attention mechanism, perform feature interaction processing on the pixel point features corresponding to each of the multiple second pixel points and the pixel point features corresponding to the first pixel point to obtain the updated pixel value at the first pixel point.

[0101] In a specific application, the terminal can use a pre-trained pixel feature extraction network to extract features of the pixel value at the first pixel and the position offset corresponding to the first pixel, and by inputting the pixel value and the position offset into the pre-trained pixel feature extraction network, the pixel feature corresponding to the first pixel can be obtained. The pre-trained pixel feature extraction network can be configured according to the actual application scenario. In addition, the terminal can also obtain the pixel feature corresponding to the first pixel by first extracting features of the pixel value at the first pixel and the position offset corresponding to the first pixel, and then fusing the two extracted features.

[0102] In a specific application, based on the attention mechanism, feature interaction processing is performed on the pixel features corresponding to multiple second pixel points and the pixel features corresponding to the first pixel point, so that the updated pixel features corresponding to the first pixel point can be obtained, and then the updated pixel value at the first pixel point can be obtained by updating the pixel features.

[0103] In a specific application, when the attention mechanism is a multi-head attention mechanism, for each attention mechanism in the multi-head attention mechanism, the terminal will perform feature interaction processing on the pixel features corresponding to multiple second pixels and the pixel features corresponding to the first pixel based on the attention mechanism to obtain the pixel features corresponding to the attention mechanism. By fusing the pixel features corresponding to each attention mechanism, the updated pixel features corresponding to the first pixel can be obtained.

[0104] Among them, the implementation of the attention mechanism mainly depends on three vectors: query vector, key vector and value vector. These three vectors are linear transformations of the input data. In this embodiment, the input data includes pixel features corresponding to multiple second pixel points and pixel features corresponding to the first pixel point.

[0105] The query vector, key vector, and value vector function and interact as follows: Query vector: represents the current input element and is used to query other related elements. Key vector: represents all input elements and is used for querying. Value vector: represents all input elements. When an element is found, its corresponding value vector is used to calculate the output.

[0106] In a specific application, based on the transformation matrix of the attention mechanism, the pixel features corresponding to each of the multiple second pixels are linearly transformed to obtain the key vectors and value vectors corresponding to each of the multiple second pixels. At the same time, based on the transformation matrix of the attention mechanism, the pixel features corresponding to the first pixel are linearly transformed to obtain the query vector for the first pixel. Then, the pixel features corresponding to the first pixel can be updated through the interaction of the query vector, key vector, and value vector. Among them, the transformation matrix of the attention mechanism includes a query vector transformation matrix, a key vector transformation matrix, and a value vector transformation matrix, which correspond to the query vector, key vector, and value vector, respectively.

[0107] In this embodiment, by using the pixel value at the first pixel point and the corresponding position offset of the first pixel point to perform feature extraction, it is possible to obtain pixel features that can accurately characterize the pixel point corresponding to the first pixel point, and then by performing feature interaction processing on the pixel point features corresponding to multiple second pixel points and the pixel point features corresponding to the first pixel point based on the attention mechanism, the pixel value at the first pixel point can be updated to obtain the updated pixel value at the first pixel point.

[0108] In one embodiment, feature extraction is performed based on the pixel value at the first pixel point and the position offset corresponding to the first pixel point to obtain the pixel point feature corresponding to the first pixel point, including: encoding the pixel value at the first pixel point to obtain the pixel value code corresponding to the first pixel point, and encoding the position offset corresponding to the first pixel point to obtain the position code corresponding to the first pixel point, and fusing the pixel value code and position code corresponding to the first pixel point to obtain the pixel point feature corresponding to the first pixel point.

[0109] Specifically, in order to obtain the pixel point feature corresponding to the first pixel point, the terminal will encode the pixel value at the first pixel point to obtain the pixel value code corresponding to the first pixel point, and encode the corresponding position offset of the first pixel point to obtain the position code corresponding to the first pixel point. Finally, the pixel value code and position code corresponding to the first pixel point are merged to obtain the pixel point feature corresponding to the first pixel point.

[0110] In a specific application, the terminal can encode pixel values ​​using a pre-trained first encoding network to obtain pixel value codes, and encode position offsets using a pre-trained second encoding network to obtain position codes. The pre-trained first encoding network and the pre-trained second encoding network can be trained according to actual application scenarios. In this embodiment, the training of the first encoding network and the second encoding network is not limited.

[0111] In a specific application, the terminal can fuse the pixel value code and position code corresponding to the first pixel point by splicing the pixel value code and position code corresponding to the first pixel point to obtain the pixel point feature corresponding to the first pixel point, that is, the pixel point feature corresponding to the first pixel point can be in the form of pixel value code + position code.

[0112] In this embodiment, by encoding the pixel value at the first pixel point, the pixel value code corresponding to the first pixel point can be obtained, and by encoding the corresponding position offset of the first pixel point, the position code corresponding to the first pixel point can be obtained. Then, by fusing the pixel value code and position code corresponding to the first pixel point, the pixel point feature corresponding to the first pixel point can be obtained.

[0113] In one embodiment, based on the attention mechanism, feature interaction processing is performed on the pixel features corresponding to each of the multiple second pixels and the pixel features corresponding to the first pixel to obtain the updated pixel value at the first pixel, including: based on the transformation matrix of the attention mechanism, linearly transforming the pixel features corresponding to each of the multiple second pixels to obtain the key vectors and value vectors corresponding to each of the multiple second pixels, and linearly transforming the pixel features corresponding to the first pixel to obtain the query vector for the first pixel, calculating the attention weights according to the key vectors corresponding to each of the multiple second pixels and the query vector for the first pixel to obtain the attention weights corresponding to each of the multiple second pixels, using the attention weights corresponding to each of the multiple second pixels as weighting coefficients of the value vectors corresponding to each of the multiple second pixels, weighting the value vectors corresponding to each of the multiple second pixels to obtain the updated pixel features for the first pixel, and obtaining the updated pixel value at the first pixel based on the updated pixel features for the first pixel.

[0114] Among them, the transformation matrix of the attention mechanism includes the query vector transformation matrix, the key vector transformation matrix and the value vector transformation matrix, which correspond to the query vector, key vector and value vector respectively.

[0115] Specifically, the terminal will perform linear transformations on the pixel features corresponding to each of the multiple second pixels based on the key vector transformation matrix in the transformation matrix of the attention mechanism to obtain key vectors corresponding to each of the multiple second pixels, and will perform linear transformations on the pixel features corresponding to each of the multiple second pixels based on the value vector transformation matrix in the transformation matrix of the attention mechanism to obtain value vectors corresponding to each of the multiple second pixels. At the same time, the terminal will perform linear transformations on the pixel features corresponding to the first pixel based on the query vector transformation matrix in the transformation matrix of the attention mechanism to obtain a query vector for the first pixel.

[0116] Specifically, on the basis of obtaining the query vector, key vector and value vector, the terminal can update the pixel feature corresponding to the first pixel through the interaction of the query vector, key vector and vector, and obtain the updated pixel feature for the first pixel. Specifically, the terminal will calculate the attention weight based on the key vector corresponding to each of the multiple second pixels and the query vector for the first pixel, obtain the attention weight corresponding to each of the multiple second pixels, and weight the value vector corresponding to each of the multiple second pixels based on the attention weight corresponding to each of the multiple second pixels to obtain the updated pixel feature for the first pixel. After obtaining the updated pixel feature for the first pixel, the terminal can obtain the updated pixel value at the first pixel by means of feature mapping.

[0117] In a specific application, the terminal can first use a similarity measurement method (such as dot product, cosine similarity, etc.) to calculate the similarity between the query vector for the first pixel point and the key vectors corresponding to each of the multiple second pixel points to obtain an attention score matrix, and then transform each element in the attention score matrix through a weight function (such as a softmax function) to generate a normalized attention weight, that is, the attention weight corresponding to each of the multiple second pixel points.

[0118] In a specific application, for each of the multiple second pixel points, the terminal will use the attention weight corresponding to the second pixel point as the weighting coefficient of the value vector corresponding to the second pixel point, and weight the value vectors corresponding to each of the multiple second pixel points to obtain the updated pixel point features for the first pixel point.

[0119] In a specific application, before linearly transforming the pixel features corresponding to each of the multiple second pixel points and the pixel features corresponding to the first pixel point through the transformation matrix of the attention mechanism, the terminal can perform layer normalization on the pixel features corresponding to each of the multiple second pixel points and the pixel features corresponding to the first pixel point, so that the distribution of the pixel features corresponding to each of the multiple second pixel points and the pixel features corresponding to the first pixel point is relatively stable, which facilitates better feature interaction processing based on the attention mechanism.

[0120] In this embodiment, based on the transformation matrix of the attention mechanism, the key vectors and value vectors corresponding to each of the multiple second pixel points and the query vector for the first pixel point can be determined by linear transformation. By calculating the attention weights according to the key vectors corresponding to each of the multiple second pixel points and the query vector for the first pixel point, the attention weights corresponding to each of the multiple second pixel points can be obtained. Therefore, the value vectors corresponding to each of the multiple second pixel points can be weighted based on the attention weights corresponding to each of the multiple second pixel points to obtain the updated pixel point features for the first pixel point, and then the updated pixel value for the first pixel point can be determined based on the updated pixel point features for the first pixel point.

[0121] In one embodiment, the updated intermediate frame images corresponding to each of the two video frame images are fused to obtain a fused intermediate frame image for insertion between the two video frame images, including: extracting features of the updated intermediate frame images corresponding to each of the two video frame images to obtain updated intermediate frame image features corresponding to each of the two video frame images, fusing the updated intermediate frame image features corresponding to each of the two video frame images to obtain fused features, and performing feature mapping based on the fused features to obtain a fused intermediate frame image for insertion between the two video frame images.

[0122] Specifically, the terminal will respectively extract features of the updated intermediate frame images corresponding to the two video frame images to obtain the updated intermediate frame image features corresponding to the two video frame images, and then fuse the updated intermediate frame image features corresponding to the two video frame images to obtain fused features. Finally, by performing feature mapping based on the fused features, the fused intermediate frame image for insertion between the two video frame images can be obtained.

[0123] In a specific application, the image fusion process in this embodiment can be performed using a pre-trained image fusion network. By simultaneously inputting the updated intermediate frame images corresponding to the two video frame images into the pre-trained image fusion network, the image fusion network can output a fused intermediate frame image. The pre-trained image fusion network can be trained according to actual application scenarios. It is understood that the image fusion network includes a feature extraction layer, a feature fusion layer, and a feature mapping layer, which are used to respectively perform feature extraction, feature fusion, and obtain a fused intermediate frame image for insertion between the two video frame images based on the fused features.

[0124] In this embodiment, by fusing the updated intermediate frame images corresponding to the two video frame images, the features of the updated intermediate frame images can be made complementary, thereby obtaining a fused intermediate frame image with rich features and higher image quality, thereby improving the image quality of the generated intermediate frame image.

[0125] In one embodiment, feature mapping is performed based on the fusion features to obtain a fused intermediate frame image for insertion between two video frame images, including: feature mapping the pixel point features of each of the multiple pixel points in the fusion features, respectively, to obtain the pixel values ​​at each third pixel point in the fused intermediate frame image, and according to the pixel values ​​at each third pixel point, to obtain the fused intermediate frame image for insertion between the two video frame images.

[0126] Specifically, the multiple pixel points in the fusion feature actually indicate the third pixel points in the fused intermediate frame image. Therefore, the terminal can obtain the pixel values ​​of each third pixel point in the fused intermediate frame image by performing feature mapping on the pixel point features of each of the multiple pixel points in the fusion feature. Then, based on the pixel values ​​at each third pixel point, the terminal can obtain the fused intermediate frame image for insertion between the two video frame images.

[0127] In a specific application, the pixel features of each of the multiple pixels can be specifically encoded as the pixel values ​​of each of the multiple pixels, and the feature mapping method can be specifically decoded, that is, the terminal can obtain the pixel value of each third pixel in the fused intermediate frame image by decoding the pixel features of each of the multiple pixels in the fused feature.

[0128] In this embodiment, since the fusion feature actually refers to the feature of the fused intermediate frame image, by performing feature mapping on the pixel point features of each of the multiple pixel points in the fusion feature, the pixel value at each third pixel point in the fused intermediate frame image can be determined, and then, based on the determination of the pixel value at each third pixel point, the fused intermediate frame image used to be inserted between the two video frame images can be accurately obtained.

[0129] In one embodiment, selecting an image block corresponding to a targeted first pixel point from a targeted video frame image according to a position offset includes: determining a first pixel position corresponding to the targeted first pixel point from the targeted video frame image according to the position offset, and selecting an image block corresponding to the targeted first pixel point from the targeted video frame image based on the first pixel position.

[0130] The position offset refers to the position offset of the first pixel point between the video frame image and the predicted intermediate frame image, that is, the displacement of the first pixel point between the video frame image and the predicted intermediate frame image.

[0131] The first pixel position refers to the pixel position of the first pixel in the target video frame before the position shift occurs. That is, for the first pixel in the predicted intermediate frame, it is the pixel position that the first pixel should be located at in the target video frame before the position shift occurs.

[0132] Specifically, for each first pixel point in the predicted intermediate frame image, the terminal can directly obtain the position offset corresponding to the targeted first pixel point from the second optical flow information, and then determine the first pixel position corresponding to the targeted first pixel point from the targeted video frame image based on the position offset, so that the image block corresponding to the targeted first pixel point can be selected from the targeted video frame image based on the first pixel position and the size of the image block.

[0133] In a specific application, the pixel position of the first pixel point can be represented by pixel coordinates, and then the first pixel position corresponding to the first pixel point can be located from the video frame image based on the pixel coordinates and the position offset. In a specific application, taking the pixel coordinates as (x, y) and the position offset as (dx, dy) as an example, the first pixel position can be expressed as (x+dx, y+dy). It should be noted that if the first pixel position corresponding to the first pixel point cannot be located from the video frame image based on the pixel coordinates and the position offset, the terminal will directly set the image block corresponding to the first pixel point according to the size of the image block, and the pixel value at each second pixel point in the set image block is 0.

[0134] In this embodiment, for each first pixel point in the predicted intermediate frame image, by obtaining the corresponding position offset of the targeted first pixel point from the second optical flow information, the position offset can be used to determine the first pixel position corresponding to the targeted first pixel point from the targeted video frame image, and then based on the first pixel position, the image block corresponding to the targeted first pixel point can be selected from the targeted video frame image.

[0135] In one embodiment, based on the first pixel position, selecting an image block corresponding to the first pixel point from the targeted video frame image includes: determining a fourth pixel point from the targeted video frame image based on the first pixel position, and selecting an image block corresponding to the first pixel point from the targeted video frame image based on the fourth pixel point and a preset image block size.

[0136] Specifically, based on determining the first pixel position, the fourth pixel point can be determined from the targeted video frame image based on the first pixel position, and then based on the size of the image block, an image block corresponding to the targeted first pixel point can be selected from the targeted video frame image, and the image block includes the fourth pixel point.

[0137] In specific applications, the size of the image block can be configured according to the actual application scenario. When selecting the image block corresponding to the first pixel point from the targeted video frame image based on the size of the image block, the selection rule can also be configured according to the actual application scenario. For example, the selection rule can be specifically for selecting from the fourth pixel point as the starting point to the left, or for selecting from the fourth pixel point as the center point. For example, taking the size of the image block as 4 pixels and selecting from the fourth pixel point as the starting point to the left as an example, the resulting image block can be shown in Figure 4 (the selected pixels are represented by bold small squares in Figure 4). For another example, taking the size of the image block as 9 pixels and selecting from the fourth pixel point as the center point as an example, the resulting image block can be shown in Figure 5 (the selected pixels are represented by bold small squares in Figure 5).

[0138] It should be noted that, in this embodiment, when selecting an image block, if the selection range is at the edge of the targeted video frame image, it may be impossible to select completely. In this case, the terminal will pad pixels in the image edge area that cannot be selected, and the pixel values ​​at the padded pixels are 0. For example, taking the image block size as 9 pixels and selecting with the fourth pixel as the center point as an example, as shown in FIG6 , when selecting with the fourth pixel as the center point, the selection range is at the edge of the targeted video frame image, and only 6 pixels can be selected from the targeted video frame image (indicated by bold small squares in FIG6 ). In this case, the terminal will pad pixels in the image edge area that cannot be selected (indicated by small squares with dotted borders in FIG6 ), and the pixel values ​​at the padded pixels are 0, so that the size of the image block remains unchanged.

[0139] In this embodiment, the first pixel position can be used to accurately determine the fourth pixel point from the targeted video frame image, and then the fourth pixel point and the size of the image block can be used to accurately select the image block.

[0140] In one embodiment, determining the fourth pixel point from the targeted video frame image based on the first pixel position includes: when the pixel coordinates at the first pixel position are coordinates in floating-point form, taking the pixel point in the targeted video frame image that is closest to the first pixel position as the fourth pixel point.

[0141] In one embodiment, determining the fourth pixel point from the targeted video frame image according to the first pixel position includes: when the pixel coordinates at the first pixel position are not floating-point coordinates, using the pixel point at the first pixel position as the fourth pixel point.

[0142] Specifically, since the position offset may be a floating point number, the pixel coordinates at the first pixel position located based on the pixel coordinates and the position offset may also be floating point coordinates. When the pixel coordinates at the first pixel position are floating point coordinates, the terminal will use the pixel closest to the first pixel position in the targeted video frame image as the fourth pixel. If the position offset is not a floating point number, the pixel coordinates at the first pixel position located based on the pixel coordinates and the position offset are also not floating point coordinates. In this case, the terminal can directly use the pixel at the first pixel position as the fourth pixel.

[0143] For example, as shown in Figure 7, in the video frame image, each small square represents a pixel point, and the center of the small square represents the pixel coordinates of the pixel point. If the first pixel position located is not the center of the small square, it means that the pixel coordinates at the first pixel position are in floating-point form, and the terminal will use the pixel point closest to the first pixel position as the fourth pixel point. For further example, as shown in Figure 8, if the first pixel position located is the center of the small square, it means that the pixel coordinates at the first pixel position are not in floating-point form, and the terminal will use the pixel point at the first pixel position as the fourth pixel point.

[0144] In this embodiment, by utilizing the coordinate form of the pixel coordinates at the first pixel position and adopting different methods to determine the fourth pixel point, the fourth pixel point can be accurately determined, and then the determined fourth pixel point can be used to achieve accurate selection of the image block.

[0145] In one embodiment, based on two video frame images and the first optical flow information, an intermediate frame image between the two video frame images is predicted to obtain the predicted intermediate frame image, including: based on the forward optical flow information and the backward optical flow information in the first optical flow information, and the intermediate frame moment, the intermediate frame optical flow information corresponding to each of the two video frame images is estimated; based on the two video frame images and the intermediate frame optical flow information corresponding to each of the two video frame images, the intermediate frame image between the two video frame images is predicted to obtain the predicted intermediate frame image.

[0146] The intermediate frame moment refers to the moment in the video corresponding to the intermediate frame image. It can be understood that since the intermediate frame image is the frame image between two video frames, the intermediate frame moment is the moment between the two moments corresponding to the two video frames. The intermediate frame optical flow information refers to the optical flow information between the intermediate frame image and the video frame image determined by optical flow estimation.

[0147] The forward optical flow information refers to the optical flow information from the earlier video frame image to the later video frame image, and the backward optical flow information refers to the optical flow information from the later video frame image to the earlier video frame image. For example, assuming that the two video frames include a first video frame image and a second video frame image, and the first video frame image is the forward frame of the second video frame image, then the forward optical flow information refers to the optical flow information from the first video frame image to the second video frame image, and the backward optical flow information refers to the optical flow information from the second video frame image to the first video frame image.

[0148] Specifically, the terminal will estimate the intermediate frame optical flow information corresponding to each of the two video frame images based on the forward optical flow information and the backward optical flow information in the first optical flow information, as well as the intermediate frame moment. After obtaining the intermediate frame optical flow information corresponding to each of the two video frame images, the terminal can predict the intermediate frame image between the two video frame images based on the two video frame images and the intermediate frame optical flow information corresponding to the two video frame images to obtain the predicted intermediate frame image.

[0149] In a specific application, it is assumed that the two video frame images are the first video frame image I0 and the second video frame image I1, and the first video frame image I0 is the forward frame of the second video frame image I1, and the intermediate frame time is t∈[0,1] (assuming that the first video frame image I0 corresponds to time 0 and the second video frame image I1 corresponds to time 1). In order to obtain the predicted intermediate frame image I t , it is necessary to calculate the intermediate frame optical flow information F t→0 and F t→1 , where F t→0 represents the intermediate frame optical flow information from the intermediate frame time t to time 0, F t→1 Represents the intermediate frame optical flow information from the intermediate frame time t to the time 1. Based on the intermediate frame optical flow information, the intermediate frame image between the two video frame images can be predicted by reverse image distortion to obtain the predicted intermediate frame image.

[0150] In specific applications, the motion between frames can be modeled based on a linear motion model. The schematic diagram of modeling optical flow based on the linear motion model for interpolation can be shown in Figure 9. If one direction is considered, the following formula can be used for estimation:

[0151] Ft→1 (p) = (1 - t) F 0→1 (p) or F t→1 (p) = -(1-t)F 1→0 (p);

[0152] Among them, (1-t) and t are weight coefficients, corresponding to the middle frame time. It can be understood that the closer t is to 0, the higher the weight in the calculation. 0→1 (p) refers to the forward optical flow information, that is, the optical flow information from the first video frame image I0 to the second video frame image I1, F 1→0 (p) refers to the backward optical flow information, that is, the optical flow information from the second video frame image I1 to the first video frame image I0, F t→1 (p) refers to the optical flow information of the intermediate frame corresponding to the second video frame image. p refers to the pixel position of the pixel point, that is, the pixel coordinate.

[0153] Furthermore, combining the bidirectional relationship, the formula for estimating the optical flow information of the intermediate frame can be further derived as follows:

[0154] F t→0 =-(1-t)tF 0→1 +t 2 F 1→0 ;

[0155] F t→1 =(1-t 2 )F 0→1 -t(1-t)F 1→0 ;

[0156] Among them, F t→0 is the intermediate frame optical flow information corresponding to the first video frame image, F t→1 is the optical flow information of the intermediate frame corresponding to the second video frame image, (1-t) and t are weight coefficients, corresponding to the intermediate frame time. It can be understood that the closer t is to 0, the higher the weight in the calculation. 0→1 (p) refers to the forward optical flow information, F 1→0 (p) refers to the backward optical flow information.

[0157] Finally, the intermediate frame image I is predicted t It can be calculated as follows:

[0158] I t =(1-t)*g(I0,F t→1 )+t*g(I1,F t→1 );

[0159] Among them, (1-t) and t are weight coefficients, corresponding to the middle frame time. It can be understood that the closer t is to 0, the higher the weight in the calculation. g(I0,F t→1) represents the image distortion operation based on the first video frame image I0 and the corresponding intermediate frame optical flow information, g(I1,F t→1 ) represents an image warping operation based on the second video frame image I1 and its corresponding intermediate frame optical flow information.

[0160] In specific applications, the motion between frames can be modeled based on a nonlinear motion model. Compared with the linear motion modeling mentioned above, this method mainly considers the second-order nonlinear relationship and considers more complex motion. In order to more realistically depict the motion of the object, assuming that the motion of the object is uniformly accelerated motion, the schematic diagram of the optical flow prediction of the intermediate frame of the nonlinear motion model can be shown in Figure 10. For uniformly accelerated motion modeling, optical flow estimation can also be performed based on the forward optical flow information and the backward optical flow information to calculate the intermediate frame optical flow information corresponding to the frame image 0 in Figure 10, that is, F 0→t As an example, we can get the following quadratic integral formula:

[0161] Among them, t is the weight coefficient, corresponding to the middle frame time. It can be understood that the closer t is to 0, the higher the weight in the calculation. k is a parameter, corresponding to the middle frame time, which can be configured according to the actual application scenario. For example, k can be a value between 0 and 1. v0 is the speed of the first pixel in the first video frame image, and a τ is the acceleration of the first pixel in the first video frame image.

[0162] F after uniform acceleration 0→t It can be:

[0163] Among them, F 0→1 is the optical flow information from frame image 0 to frame image 1 in Figure 10, F 0→(-1) is the optical flow information from frame image 0 to frame image -1 in Figure 10, and t is the weight coefficient, corresponding to the intermediate frame time. It can be understood that the closer t is to 0, the higher the weight it occupies in the calculation.

[0164] It should be noted that the weight coefficient t in the above formula can be obtained by the intermediate frame time and the two times corresponding to the two video frame images. Specifically, assuming that the earlier of the two times corresponding to the two video frame images is t1 and the later is t2, and the intermediate frame time is t3, the weight coefficient t can be obtained by the formula weight coefficient t = (t3-t2) / (t1-t2).

[0165] In this embodiment, optical flow estimation is performed based on the forward optical flow information and backward optical flow information in the first optical flow information, as well as the intermediate frame moments, to obtain the intermediate frame optical flow information corresponding to each of the two video frame images. Thus, the two video frame images and the intermediate frame optical flow information corresponding to each of the two video frame images can be used to predict the intermediate frame image between the two video frame images to obtain the predicted intermediate frame image.

[0166] In one embodiment, based on two video frame images and the intermediate frame optical flow information corresponding to each of the two video frame images, an intermediate frame image between the two video frame images is predicted to obtain the predicted intermediate frame image, which includes: for each of the two video frame images, based on the targeted video frame image and the intermediate frame optical flow information corresponding to the targeted video frame image, the intermediate frame image between the two video frame images is predicted to obtain a predicted image corresponding to the targeted video frame image, and the predicted images corresponding to each of the two video frame images are fused to obtain a predicted intermediate frame image.

[0167] Specifically, for each video frame image in the two video frame images, the terminal will perform image distortion operation on the targeted video frame image and the intermediate frame optical flow information corresponding to the targeted video frame image, predict the intermediate frame image between the two video frame images, obtain the predicted image corresponding to the targeted video frame image, and then fuse the predicted images corresponding to the two video frame images to obtain the predicted intermediate frame image.

[0168] In a specific application, as shown in FIG11, the image warping operation includes a forward image warping operation and a backward image warping operation. The forward image warping operation is to predict the image based on the earlier of the two video frames (the first video frame as shown in FIG11) and the corresponding intermediate frame optical flow information. The backward image warping operation is to predict the image based on the later of the two video frames (the second video frame as shown in FIG11) and the corresponding intermediate frame optical flow information.

[0169] Among them, the image distortion operation is essentially an image interpolation scheme, which is generally bilinear interpolation or nearest neighbor interpolation. Bilinear interpolation or nearest neighbor interpolation are essentially an approximate display modeling, that is, the motion changes of adjacent frame images are modeled as linear motion for processing. Therefore, this approximate display modeling can often only achieve very general results. For complex motion scenes, or scenes such as lighting changes, this approximate modeling method will fail. Therefore, in this application, after obtaining the predicted intermediate frame image, the predicted intermediate frame image will be further processed to generate a fused intermediate frame image with high image quality.

[0170] In this embodiment, for each of the two video frame images, based on the targeted video frame image and the intermediate frame optical flow information corresponding to the targeted video frame image, the intermediate frame image between the two video frame images can be predicted to obtain the predicted image corresponding to the targeted video frame image, and then the predicted intermediate frame image can be obtained by fusing the predicted images corresponding to the two video frame images.

[0171] The inventors believe that traditional methods are mainly based on the optical flow method to explicitly calculate intermediate frame images for video interpolation. However, the optical flow-based method mainly describes the more obvious movement of rigid target images and has two strong assumptions. The first assumption is that the pixel value of the target remains basically unchanged between different video frame images, and the second assumption is that adjacent pixels have basically similar movements (that is, the optical flow is basically similar). In addition, the accuracy of optical flow estimation is affected by many factors, such as lighting changes, occlusion, motion blur, etc., which may lead to inaccurate estimated optical flow, thereby affecting the quality of video interpolation. Furthermore, optical flow estimation may encounter difficulties when processing fast or complex motion. For example, when the movement of an object between two frames exceeds one pixel, the optical flow estimation may not be able to accurately capture this motion. Finally, interpolation technology based on optical flow estimation may introduce some visual artifacts, such as blurring and jittering. This is because optical flow estimation is an interpolation technology, it may not be able to fully capture all subtle changes between two frames.

[0172] Based on this, this application proposes a method for generating video interpolation images, which can be understood as a latent space alignment scheme based on the transformer architecture. For video interpolation scenarios with complex and nonlinear motion and complex lighting conditions, latent space alignment can effectively learn the morphological and dynamic characteristics between adjacent video frames, thereby better modeling the evolution process of adjacent video frames and providing better dynamic material interpolation effects.

[0173] In one embodiment, the video interpolation image generation method in the present application adopts an implicit modeling method, that is, directly modeling the motion changes in the video frame through a neural network. Based on the acquisition of two video frame images, in the present application, the intermediate frame image between the two video frame images is first predicted to obtain a predicted intermediate frame image. For each of the two video frame images, the second optical flow information between the targeted video frame image and the predicted intermediate frame image is determined. Then, based on the targeted video frame image, the predicted intermediate frame image, and the second optical flow information between the targeted video frame image and the predicted intermediate frame image, an updated intermediate frame image corresponding to the targeted video frame image is generated. By fusing the updated intermediate frame images corresponding to the two video frame images, a fused intermediate frame image for insertion between the two video frame images can be obtained.

[0174] It should be noted that this application does not adopt the traditional bilinear interpolation or nearest neighbor modeling method, but implicitly learns the process of generating such intermediate frames through the transformer structure. Specifically, in this application, the process of obtaining the updated intermediate frame image corresponding to the targeted video frame image can be described by the following formula: Implicit_warp = function(I_ref, I_target, flow), where Implicit_warp refers to updating the intermediate frame image, function() represents the function of the neural network, i.e., generating the updated intermediate frame image, I_ref refers to the targeted video frame image, I_target refers to the predicted intermediate frame image, and flow refers to the second optical flow information between the targeted video frame image and the predicted intermediate frame image.

[0175] In a specific application, in this application, the process of generating an updated intermediate frame image corresponding to the targeted video frame image based on the targeted video frame image, the predicted intermediate frame image, and the second optical flow information between the targeted video frame image and the predicted intermediate frame image can be implemented by a pre-trained neural network. The pre-trained neural network can perform the steps of generating the updated intermediate frame image in the above embodiment. The pre-trained neural network can be trained according to the actual application scenario, and this embodiment does not limit this here.

[0176] The Transformer is a deep learning model, as shown in Figure 12. It primarily comprises a multi-head attention mechanism, residual connections and normalization, and feed-forward layers. It is primarily used for processing sequential data, such as natural language processing tasks and processing image patch sequences. Its core is the self-attention mechanism, which can capture long-range dependencies within sequences. In the Transformer, the self-attention mechanism relies primarily on three vectors: the query vector (Query, Q), the key vector (Key, K), and the value vector (Value, V). These three vectors are linear transformations of the input data. As shown in Figure 12, the self-attention mechanism first calculates the dot product of Q and K to produce an attention score matrix representing each element's relevance to the others. These scores are then converted into weights using a softmax function, so that the sum of each element's weight is 1, indicating the distribution of its attention to the others. Finally, these weights are multiplied by the corresponding V vector and summed to produce the output. Thus, each output element is a weighted sum of the input elements, with the weight representing the contribution of the input element to the output element.

[0177] The following describes how the transformer structure is used for implicit alignment. As shown in Figure 13, the left side of Figure 13 is traditional linear interpolation, which mainly processes the four pixels around the interpolation point and performs bilinear interpolation on the pixel values ​​at the four pixels. The optical flow information mainly guides the search for the above four pixels. The right side of Figure 13 is the implicit alignment method based on the transformer structure proposed in this application. In the solution of this application, the optical flow is used to assist in locating the approximate alignment area, that is, for each first pixel in the predicted intermediate frame image, the position offset corresponding to the first pixel (as shown in Figure 13 as I(x, y)) is obtained from the second optical flow information. Based on the position offset, the first pixel position corresponding to the first pixel is determined from the video frame image (as shown in Figure 13 as I'(x+dx, y+dy)). Based on the determination of the first pixel position, the image block corresponding to the first pixel can be selected from the video frame image based on the first pixel position. Specifically, the terminal can first select an image window block, the size of which can be 2 pixels or 3 pixels, and then select an image block based on the image window block. When the size of the image window block is 2 pixels, as shown in Figure 13, the image block includes 4 pixels. When the size of the image window block is 3 pixels, the image block includes 9 pixels.

[0178] Furthermore, when the transformer structure processes an image block (M as shown in Figure 13), it is generally necessary to add a position code (PE M ) information is used to learn the position information of the image block in the global image. In this application, that is, for each first pixel point in the predicted intermediate frame image, it is necessary to obtain the position information of the image block corresponding to the first pixel point from the targeted video frame image, and then based on the image block corresponding to the first pixel point and the position information, update the pixel value at the targeted first pixel point to obtain the updated pixel value at the targeted first pixel point, and according to the updated pixel value at each first pixel point in the predicted intermediate frame image, obtain the updated intermediate frame image corresponding to the targeted video frame image.

[0179] In one embodiment, taking the image block including 4 second pixels as an example, as shown in FIG13, for each of the 4 second pixels, the terminal performs feature extraction based on the pixel value at the second pixel and the pixel position of the second pixel to obtain the pixel feature corresponding to the second pixel, and based on the pixel features corresponding to the plurality of second pixels (as shown in FIG13 for PE M + M), the pixel value at the first pixel point is updated to obtain the updated pixel value at the first pixel point (as shown in FIG. 13 ). ).

[0180] In one embodiment, the terminal encodes the pixel value at the second pixel point to obtain the pixel value code corresponding to the second pixel point, and encodes the pixel position of the second pixel point to obtain the position code corresponding to the second pixel point, and fuses the pixel value code and position code corresponding to the second pixel point to obtain the pixel feature corresponding to the second pixel point.

[0181] In one embodiment, when the pixel value at the first pixel point is updated, the terminal extracts features based on the pixel value at the first pixel point and the position offset corresponding to the first pixel point (as shown in FIG13 as (Δx, Δy)), obtains the pixel point features corresponding to the first pixel point, and then linearly transforms the pixel point features corresponding to each of the multiple second pixel points based on the transformation matrix of the attention mechanism, and obtains the key vectors corresponding to each of the multiple second pixel points (as shown in FIG13 as P K ) and value vector (as shown in Figure 13 P V ), and linearly transform the pixel features corresponding to the first pixel to obtain the query vector (P Q ), interact with the key vector, value vector and query vector to obtain the updated pixel value at the first pixel point.

[0182] Among them, the key vector, value vector and query vector are interacted to obtain the updated pixel value at the first pixel point by calculating the attention weight according to the key vectors corresponding to each of the multiple second pixel points and the query vector corresponding to the first pixel point, obtaining the attention weight corresponding to each of the multiple second pixel points, and weighting the value vectors corresponding to each of the multiple second pixel points based on the attention weights corresponding to each of the multiple second pixel points to obtain the updated pixel point features for the first pixel point, and obtaining the updated pixel value at the first pixel point based on the updated pixel point features for the first pixel point.

[0183] In one embodiment, the terminal encodes the pixel value at the first pixel point to obtain the pixel value code corresponding to the first pixel point, and encodes the corresponding position offset of the first pixel point to obtain the position code corresponding to the first pixel point, and fuses the pixel value code and position code corresponding to the first pixel point to obtain the pixel point feature corresponding to the first pixel point.

[0184] In one embodiment, the video interpolation image generation method of the present application is mainly used in video super-resolution, and video super-resolution is widely used in online and offline content playback, live broadcast and on-demand scenarios, old film restoration, RTC (Real-Time Communications) super-resolution and other fields. It can be used for researchers to conduct further research and exploration in the above-mentioned fields, and it can also be used for consumers to integrate into the local end to perform video interpolation, thereby improving the frame rate of the video content and increasing the smoothness of the video. Dynamic content occupies a considerable proportion in the video, and the traditional interpolation method often has poor interpolation effect on dynamic content. Therefore, the video interpolation image generation method proposed in this application can effectively improve the application effect in this field for interpolation of dynamic scenes.

[0185] In one embodiment, as shown in Figure 14, a video interpolation image generation method provided in this application can be applied to dynamic scene interpolation. As an underlying technology, dynamic scene interpolation can be used together with other underlying technologies such as video super-resolution (including denoising, sharpening, face detection, etc. as shown in Figure 14) to construct the underlying code. Then, various application scenarios can be abstracted and practical algorithms can be summarized, such as on-demand / live RTC super-resolution, on-demand / live SDR (Standard Dynamic Range) enhancement, on-demand / live HDR (High Dynamic Range Imaging) enhancement, UGC (User Generated Content) image quality enhancement and other fields, which can be applied to various business needs (including video software, video cloud, online conference, video account, instant messaging software, advertising, etc. as shown in Figure 14).

[0186] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0187] Based on the same inventive concept, embodiments of the present application also provide a video interpolation image generation device for implementing the aforementioned video interpolation image generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the video interpolation image generation device provided below can be found in the aforementioned limitations of the video interpolation image generation method and will not be further elaborated here.

[0188] In one embodiment, as shown in FIG15 , a video interpolation image generation apparatus is provided, comprising: a first optical flow calculation module 1502 , a prediction module 1504 , a second optical flow calculation module 1506 , an image block selection module 1508 , an update module 1510 , and a fusion module 1512 , wherein:

[0189] A first optical flow calculation module 1502 is configured to obtain two video frame images and determine first optical flow information between the two video frame images based on the displacement of pixels in the two video frame images;

[0190] A prediction module 1504 is configured to predict an intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information to obtain a predicted intermediate frame image;

[0191] A second optical flow calculation module 1506 is configured to determine, for each of the two video frame images, second optical flow information between the video frame image and the predicted intermediate frame image based on the displacement of pixels in the video frame image and the predicted intermediate frame image;

[0192] An image block selection module 1508 is configured to obtain, for each first pixel in the predicted intermediate frame image, a position offset corresponding to the first pixel from the second optical flow information, and select an image block corresponding to the first pixel from the video frame image based on the position offset;

[0193] An updating module 1510 is configured to update the pixel value of each first pixel point in the predicted intermediate frame image based on the image block corresponding to each first pixel point in the predicted intermediate frame image, to obtain an updated intermediate frame image corresponding to the video frame image; and

[0194] The fusion module 1512 is configured to fuse the updated intermediate frame images corresponding to the two video frame images to obtain a fused intermediate frame image for insertion between the two video frame images.

[0195] The above-mentioned video interpolation image generation device obtains two video frame images and determines the first optical flow information between the two video frame images according to the displacement of the pixel points in the two video frame images. It can realize the prediction of the intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information to obtain the predicted intermediate frame image. By for each video frame image in the two video frame images, according to the displacement of the pixel points in the video frame image and the predicted intermediate frame image, the second optical flow information between the targeted video frame image and the predicted intermediate frame image is determined. For each first pixel point in the predicted intermediate frame image, the corresponding position offset of the targeted first pixel point is obtained from the second optical flow information. According to the position offset, the corresponding position offset of the targeted first pixel point is obtained from the targeted pixel point. In the video frame image, the image block corresponding to the first pixel point is selected, and the pixel value at each first pixel point in the predicted intermediate frame image is updated respectively based on the image block corresponding to each first pixel point in the predicted intermediate frame image. The morphological features and dynamic features between adjacent video frame images can be learned through implicit image and feature alignment, so as to better model the dynamic changes and evolution process of adjacent video frame images, and obtain the updated intermediate frame image corresponding to the video frame image with high image quality. Furthermore, the updated intermediate frame images corresponding to the two video frame images can be fused to obtain a fused intermediate frame image for insertion between the two video frame images, which can improve the image quality of the generated intermediate frame image.

[0196] In one embodiment, the update module is also used to obtain the position information of the image block corresponding to each first pixel point in the predicted intermediate frame image from the targeted video frame image, update the pixel value at the targeted first pixel point based on the image block corresponding to the targeted first pixel point and the position information, and obtain the updated pixel value at the targeted first pixel point; according to the updated pixel value at each first pixel point in the predicted intermediate frame image, obtain the updated intermediate frame image corresponding to the targeted video frame image.

[0197] In one embodiment, the image block corresponding to the first pixel point includes multiple second pixel points, and the position information includes the pixel position of each of the multiple second pixel points. The update module is also used to perform feature extraction for each of the multiple second pixel points based on the pixel value at the second pixel point and the pixel position of the second pixel point to obtain the pixel feature corresponding to the second pixel point, and update the pixel value at the first pixel point based on the pixel feature corresponding to each of the multiple second pixel points to obtain the updated pixel value at the first pixel point.

[0198] In one embodiment, the update module is also used to encode the pixel value at the second pixel point to obtain the pixel value code corresponding to the second pixel point, and to encode the pixel position of the second pixel point to obtain the position code corresponding to the second pixel point, and to fuse the pixel value code and position code corresponding to the second pixel point to obtain the pixel feature corresponding to the second pixel point.

[0199] In one embodiment, the update module is also used to perform feature extraction based on the pixel value at the first pixel point and the corresponding position offset of the first pixel point to obtain the pixel point features corresponding to the first pixel point, and based on the attention mechanism, perform feature interaction processing on the pixel point features corresponding to each of the multiple second pixel points and the pixel point features corresponding to the first pixel point to obtain the updated pixel value at the first pixel point.

[0200] In one embodiment, the update module is also used to encode the pixel value at the first pixel point to obtain the pixel value code corresponding to the first pixel point, and to encode the corresponding position offset of the first pixel point to obtain the position code corresponding to the first pixel point, and to fuse the pixel value code and position code corresponding to the first pixel point to obtain the pixel point feature corresponding to the first pixel point.

[0201] In one embodiment, the update module is also used to perform linear transformation on the pixel features corresponding to each of the multiple second pixel points based on the transformation matrix of the attention mechanism, obtain the key vector and value vector corresponding to each of the multiple second pixel points, and perform linear transformation on the pixel features corresponding to the first pixel point to obtain the query vector for the first pixel point, calculate the attention weights according to the key vectors corresponding to each of the multiple second pixel points and the query vector for the first pixel point, obtain the attention weights corresponding to each of the multiple second pixel points, use the attention weights corresponding to each of the multiple second pixel points as the weighting coefficients of the value vectors corresponding to each of the multiple second pixel points, weight the value vectors corresponding to each of the multiple second pixel points, obtain the updated pixel features for the first pixel point, and obtain the updated pixel value at the first pixel point based on the updated pixel features for the first pixel point.

[0202] In one embodiment, the fusion module is also used to extract features of the updated intermediate frame images corresponding to each of the two video frame images, obtain the updated intermediate frame image features corresponding to each of the two video frame images, fuse the updated intermediate frame image features corresponding to each of the two video frame images, obtain fused features, perform feature mapping based on the fused features, and obtain a fused intermediate frame image for insertion between the two video frame images.

[0203] In one embodiment, the fusion module is also used to perform feature mapping on the pixel features of each of the multiple pixel points in the fusion feature, obtain the pixel values ​​at each third pixel point in the fused intermediate frame image, and obtain the fused intermediate frame image for insertion between the two video frame images based on the pixel values ​​at each third pixel point.

[0204] In one embodiment, the image block selection module is further used to determine a first pixel position corresponding to the targeted first pixel point from the targeted video frame image based on the position offset, and based on the first pixel position, select an image block corresponding to the targeted first pixel point from the targeted video frame image.

[0205] In one embodiment, the image block selection module is further used to determine a fourth pixel point from the targeted video frame image based on the first pixel position, and select an image block corresponding to the targeted first pixel point from the targeted video frame image based on the fourth pixel point and the size of the preset image block.

[0206] In one embodiment, the image block selection module is further configured to, when the pixel coordinates at the first pixel position are floating-point coordinates, use the pixel point closest to the first pixel position in the targeted video frame image as the fourth pixel point.

[0207] In one embodiment, the image block selection module is further configured to use the pixel point at the first pixel position as the fourth pixel point when the pixel coordinates at the first pixel position are not in floating-point form.

[0208] In one embodiment, the prediction module is also used to estimate the intermediate frame optical flow information corresponding to each of the two video frame images based on the forward optical flow information and the backward optical flow information in the first optical flow information, as well as the intermediate frame moment, and predict the intermediate frame image between the two video frame images based on the two video frame images and the intermediate frame optical flow information corresponding to the two video frame images to obtain a predicted intermediate frame image.

[0209] In one embodiment, the prediction module is also used to predict the intermediate frame image between the two video frame images based on the targeted video frame image and the intermediate frame optical flow information corresponding to the targeted video frame image for each of the two video frame images, obtain the predicted image corresponding to the targeted video frame image, and fuse the predicted images corresponding to the two video frame images to obtain the predicted intermediate frame image.

[0210] Each module in the aforementioned video interpolation image generation device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0211] In one embodiment, a computer device is provided. The computer device can be a terminal or a server. Taking the terminal as an example, its internal structure diagram can be shown in Figure 16. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication method can be implemented via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a method for generating video interpolation images. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0212] Those skilled in the art will understand that the structure shown in Figure 16 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0213] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0214] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0215] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0216] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0217] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for generating a video interpolation image, characterized in that: Executed by a computer device, the method includes: Acquire two video frame images, and determine first optical flow information between the two video frame images according to the displacement of pixels in the two video frame images; Based on the two video frame images and the first optical flow information, predict an intermediate frame image between the two video frame images to obtain a predicted intermediate frame image; For each of the two video frame images, determining second optical flow information between the targeted video frame image and the predicted intermediate frame image according to displacements of pixels in the targeted video frame image and the predicted intermediate frame image; For each first pixel point in the predicted intermediate frame image, obtain a position offset corresponding to the first pixel point from the second optical flow information, and select an image block corresponding to the first pixel point from the targeted video frame image according to the position offset; Based on the image blocks corresponding to the first pixels in the predicted intermediate frame image, respectively updating the pixel values ​​at the first pixels in the predicted intermediate frame image to obtain the updated intermediate frame image corresponding to the targeted video frame image; and The updated intermediate frame images corresponding to the two video frame images are fused to obtain a fused intermediate frame image for insertion between the two video frame images.

2. The method according to claim 1, characterized in that The updating of the pixel values ​​at the first pixels in the predicted intermediate frame image based on the image blocks corresponding to the first pixels in the predicted intermediate frame image to obtain the updated intermediate frame image corresponding to the video frame image comprises: For each first pixel point in the predicted intermediate frame image, obtaining position information of an image block corresponding to the first pixel point from the targeted video frame image; Based on the image block corresponding to the targeted first pixel and the position information, updating the pixel value at the targeted first pixel to obtain an updated pixel value at the targeted first pixel; and According to the updated pixel value at each first pixel point in the predicted intermediate frame image, an updated intermediate frame image corresponding to the targeted video frame image is obtained.

3. The method according to claim 2, characterized in that The image block corresponding to the first pixel includes a plurality of second pixels, and the position information includes the pixel position of each of the plurality of second pixels; The updating of the pixel value at the targeted first pixel based on the image block corresponding to the targeted first pixel and the position information to obtain the updated pixel value at the targeted first pixel includes: For each second pixel point among the plurality of second pixel points, feature extraction is performed based on a pixel value at the second pixel point and a pixel point position of the second pixel point to obtain a pixel point feature corresponding to the second pixel point; and Based on the pixel point features corresponding to each of the plurality of second pixel points, the pixel value at the targeted first pixel point is updated to obtain the updated pixel value at the targeted first pixel point.

4. The method according to claim 3, characterized in that The extracting a feature based on the pixel value at the second pixel and the pixel position of the second pixel to obtain the pixel feature corresponding to the second pixel comprises: Encoding the pixel value at the second pixel point to obtain the pixel value code corresponding to the second pixel point, and encoding the pixel point position of the second pixel point to obtain the position code corresponding to the second pixel point; and The pixel value code and the position code corresponding to the second pixel point are fused to obtain the pixel point feature corresponding to the second pixel point.

5. The method according to claim 3, characterized in that: The updating of the pixel value at the first pixel based on the pixel features corresponding to each of the plurality of second pixel points to obtain the updated pixel value at the first pixel point includes: Perform feature extraction based on the pixel value at the targeted first pixel and the position offset corresponding to the targeted first pixel to obtain a pixel feature corresponding to the targeted first pixel; and Based on the attention mechanism, feature interaction processing is performed on the pixel features corresponding to each of the multiple second pixel points and the pixel features corresponding to the first pixel point to obtain an updated pixel value at the first pixel point.

6. The method according to claim 5, characterized in that The extracting a feature based on the pixel value at the first pixel and the position offset corresponding to the first pixel to obtain the pixel feature corresponding to the first pixel comprises: Encoding the pixel value at the targeted first pixel point to obtain a pixel value code corresponding to the targeted first pixel point, and encoding the position offset corresponding to the targeted first pixel point to obtain a position code corresponding to the targeted first pixel point; and The pixel value code and the position code corresponding to the first pixel point are fused to obtain the pixel point feature corresponding to the first pixel point.

7. The method according to claim 5, characterized in that Based on the attention mechanism, performing feature interaction processing on the pixel features corresponding to each of the plurality of second pixel points and the pixel features corresponding to the first pixel point to obtain the updated pixel value at the first pixel point includes: Based on the transformation matrix of the attention mechanism, linearly transform the pixel point features corresponding to each of the plurality of second pixel points respectively to obtain the key vector and the value vector corresponding to each of the plurality of second pixel points, and linearly transform the pixel point features corresponding to the first pixel point to obtain the query vector for the first pixel point; Calculate the attention weights according to the key vectors corresponding to the plurality of second pixels and the query vector for the first pixel to obtain the attention weights corresponding to the plurality of second pixels; Using the attention weights corresponding to the plurality of second pixels as weighting coefficients of the value vectors corresponding to the plurality of second pixels, weighting the value vectors corresponding to the plurality of second pixels, to obtain the updated pixel feature for the first pixel; and Based on the updated pixel feature of the first pixel, an updated pixel value at the first pixel is obtained.

8. The method according to any one of claims 1 to 7, characterized in that: The step of fusing the updated intermediate frame images corresponding to the two video frame images to obtain a fused intermediate frame image for insertion between the two video frame images comprises: Extracting features of the updated intermediate frame images respectively corresponding to the two video frame images to obtain features of the updated intermediate frame images respectively corresponding to the two video frame images; Fusing the updated intermediate frame image features corresponding to the two video frame images to obtain fused features; and Feature mapping is performed based on the fusion features to obtain a fused intermediate frame image for insertion between the two video frame images.

9. The method according to claim 8, characterized in that The performing feature mapping based on the fusion feature to obtain a fused intermediate frame image for inserting between the two video frame images comprises: Performing feature mapping on the pixel features of each of the plurality of pixels in the fused features, respectively, to obtain pixel values ​​at each of the third pixels in the fused intermediate frame image; and According to the pixel value at each of the third pixel points, a fused intermediate frame image for inserting between the two video frame images is obtained.

10. The method according to any one of claims 1 to 9, characterized in that: The selecting, according to the position offset, from the targeted video frame image, an image block corresponding to the targeted first pixel point comprises: Determining, from the targeted video frame image, a first pixel position corresponding to the targeted first pixel point according to the position offset; and Based on the first pixel position, an image block corresponding to the targeted first pixel point is selected from the targeted video frame image.

11. The method according to claim 10, characterized in that The selecting, based on the first pixel position, an image block corresponding to the targeted first pixel point from the targeted video frame image comprises: Determining a fourth pixel point from the targeted video frame image according to the first pixel position; and According to the fourth pixel point and a size of a preset image block, an image block corresponding to the targeted first pixel point is selected from the targeted video frame image.

12. The method according to claim 11, characterized in that The determining, according to the first pixel position, a fourth pixel point from the targeted video frame image comprises: When the pixel coordinates at the first pixel position are coordinates in floating point form, the pixel point closest to the first pixel position in the targeted video frame image is used as the fourth pixel point.

13. The method according to claim 11, characterized in that The determining, according to the first pixel position, a fourth pixel point from the targeted video frame image comprises: When the pixel coordinates at the first pixel position are not in floating point form, the pixel point at the first pixel position is used as the fourth pixel point.

14. The method according to any one of claims 1 to 13, characterized in that: The predicting, based on the two video frame images and the first optical flow information, an intermediate frame image between the two video frame images to obtain the predicted intermediate frame image comprises: Based on the forward optical flow information and the backward optical flow information in the first optical flow information, and the intermediate frame time, estimate the intermediate frame optical flow information corresponding to each of the two video frame images; and Based on the two video frame images and the intermediate frame optical flow information corresponding to the two video frame images respectively, an intermediate frame image between the two video frame images is predicted to obtain a predicted intermediate frame image.

15. The method according to claim 14, characterized in that The step of predicting an intermediate frame image between the two video frame images based on the two video frame images and the intermediate frame optical flow information corresponding to the two video frame images respectively, to obtain the predicted intermediate frame image comprises: For each of the two video frame images, based on the targeted video frame image and the intermediate frame optical flow information corresponding to the targeted video frame image, predict an intermediate frame image between the two video frame images to obtain a predicted image corresponding to the targeted video frame image; and The predicted images corresponding to the two video frame images are fused to obtain a predicted intermediate frame image.

16. A video interpolation image generation device, characterized in that: The device comprises: A first optical flow calculation module is used to obtain two video frame images, and determine first optical flow information between the two video frame images according to the displacement of pixels in the two video frame images; A prediction module, configured to predict an intermediate frame image between the two video frame images based on the two video frame images and the first optical flow information, to obtain a predicted intermediate frame image; A second optical flow calculation module is used to determine, for each of the two video frame images, second optical flow information between the targeted video frame image and the predicted intermediate frame image according to the displacement of pixels in the targeted video frame image and the predicted intermediate frame image; An image block selection module is used to obtain, for each first pixel point in the predicted intermediate frame image, a position offset corresponding to the first pixel point from the second optical flow information, and select an image block corresponding to the first pixel point from the targeted video frame image according to the position offset; an updating module, configured to update the pixel values ​​at each first pixel point in the predicted intermediate frame image based on the image blocks corresponding to each first pixel point in the predicted intermediate frame image, respectively, to obtain an updated intermediate frame image corresponding to the targeted video frame image; and The fusion module is used to fuse the updated intermediate frame images corresponding to the two video frame images to obtain a fused intermediate frame image for insertion between the two video frame images.

17. The device according to claim 16, characterized in that The update module is also used to obtain, for each first pixel point in the predicted intermediate frame image, the position information of the image block corresponding to the first pixel point from the targeted video frame image, update the pixel value at the targeted first pixel point based on the image block corresponding to the first pixel point and the position information, and obtain the updated pixel value at the targeted first pixel point; and according to the updated pixel value at each first pixel point in the predicted intermediate frame image, obtain the updated intermediate frame image corresponding to the targeted video frame image.

18. The device according to claim 17, characterized in that The image block corresponding to the targeted first pixel point includes multiple second pixels, and the position information includes the pixel position of each of the multiple second pixels; the update module is also used to perform feature extraction on each of the multiple second pixels based on the pixel value at the targeted second pixel point and the pixel position of the targeted second pixel point to obtain the pixel feature corresponding to the targeted second pixel point, and update the pixel value at the targeted first pixel point based on the pixel feature corresponding to each of the multiple second pixels to obtain the updated pixel value of the targeted first pixel point.

19. The device according to claim 18, characterized in that The update module is also used to encode the pixel value at the second pixel point to obtain the pixel value code corresponding to the second pixel point, and to encode the pixel point position of the second pixel point to obtain the position code corresponding to the second pixel point, and to fuse the pixel value code and position code corresponding to the second pixel point to obtain the pixel point feature corresponding to the second pixel point.

20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

Images