Video frame interpolation processing method, device, computer equipment and readable storage medium

Abstract

The present application provides a video frame interpolation processing method, device, computer equipment and computer-readable storage medium, which belongs to the field of computer vision processing. The method includes: determining the first frame missing position and the first frame interpolation frequency based on the motion information and detail information of the video to be processed through the strategy generation layer of the target frame generation model; extracting features based on the frame feature sequence of the video to be processed and the first frame missing position to obtain the first associated frame feature through the feature extraction layer of the target frame generation model; characterizing the description information of the video to be processed to obtain the first description feature of the video to be processed; generating a target compensation frame for the first frame missing position based on the first associated frame feature, the first frame interpolation frequency and the first description feature through the image generation layer of the target frame generation model; performing frame interpolation processing on the video to be processed based on the target compensation frame to obtain the video after frame interpolation. The present application can improve the accuracy of frame interpolation prediction for different types of video content.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video frame interpolation processing method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] In order to improve video quality, video interpolation technology is usually used to interpolate frames in the video. Currently, video interpolation technology is mainly divided into two categories: interpolation method and optical flow method. Among them, the interpolation method increases the frame rate by inserting new frames between adjacent frames, and usually uses linear or nonlinear estimation methods to predict intermediate frames. However, the video content of different videos varies greatly. For example, some videos have relatively fast movements while others are slow. When the same prediction model is used to predict interpolation frames for different types of video content, the quality of interpolation prediction varies greatly. It can be seen that the accuracy of the interpolation prediction model for different types of video content varies greatly, and the model adaptability is poor. How to improve the accuracy of the prediction model for interpolation frames for different types of video content has become one of the focuses of video interpolation technology. Summary of the Invention

[0003] The present application provides a video frame interpolation processing method, apparatus, computer equipment, and computer-readable storage medium, which belong to the field of computer vision technology and can improve the accuracy of frame interpolation prediction for different types of video content.

[0004] In a first aspect, the present application provides a video frame interpolation processing method, the method comprising:

[0005] Determining, by a strategy generation layer of a target frame generation model, a first frame missing position of the video to be processed and a first frame filling frequency of the first frame missing position based on motion information and detail information of the video to be processed;

[0006] Performing feature extraction based on the frame feature sequence of the video to be processed and the first frame missing position through the feature extraction layer of the target frame generation model to obtain a first associated frame feature of the first frame missing position;

[0007] Performing characterization processing on the description information of the video to be processed to obtain a first description feature of the video to be processed;

[0008] Generate a target compensation frame for the missing position of the first frame based on the first associated frame feature, the first frame complement frequency, and the first description feature through the image generation layer of the target frame generation model;

[0009] The video to be processed is subjected to frame interpolation processing based on the target compensation frame to obtain a frame-interpolated video.

[0010] In a second aspect, the present application provides a video frame interpolation processing device, the video frame interpolation processing device comprising:

[0011] a generating unit configured to determine, by a strategy generating layer of a target frame generating model, a first frame missing position of the video to be processed and a first frame filling frequency of the first frame missing position based on motion information and detail information of the video to be processed;

[0012] an extraction unit, configured to perform feature extraction based on a frame feature sequence of the video to be processed and the first frame missing position through a feature extraction layer of the target frame generation model, to obtain a first associated frame feature of the first frame missing position;

[0013] a characterization unit, configured to perform characterization processing on the description information of the video to be processed to obtain a first description feature of the video to be processed;

[0014] A processing unit, configured to generate, by means of an image generation layer of the target frame generation model, a target compensation frame for a missing position of the first frame based on the first associated frame feature, the first frame complementation frequency, and the first description feature;

[0015] The processing unit is further configured to perform frame interpolation processing on the video to be processed based on the target compensation frame to obtain a frame-interpolated video.

[0016] In a third aspect, the present application further provides a computer device comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the video frame interpolation processing method when executing the computer program.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the video frame interpolation processing method.

[0018] In this application, firstly, based on the motion information and detail information of the video to be processed, the first frame missing position and the first frame filling frequency of the first frame missing position are determined, and the target compensation frame is generated using the first associated frame feature of the first frame missing position. This model can generate the target compensation frame by capturing the video's motion information and detail information, enabling the model to generate target compensation frames for different types of video content, thus avoiding the problem of large differences in frame filling prediction accuracy for different types of video content. Secondly, by first determining the frame missing position and frame filling frequency using motion information and detail information, it is more convenient to formulate different frame filling strategies for different types of video content, thereby improving the accuracy of frame filling prediction for different types of video content. Thirdly, by using the first descriptive features obtained through characterization processing using the descriptive information, the target compensation frame is generated. Because this descriptive information can fully reflect information such as the image quality and image content of the video to be processed, the first descriptive features are integrated to generate the target compensation frame. This allows the model to more specifically predict frame filling for different types of video content (such as different image quality, different object motion, and different motion speed), generating target compensation frames that match the corresponding video content type, thereby improving the prediction accuracy of the compensation frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a flow chart of a video frame interpolation processing method provided in an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of a principle framework of a target frame generation model provided in an embodiment of the present application;

[0022] Figure 3 This is a flow chart of the preset frame generation model training provided in the embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a framework of a usage process of the preset frame generation model provided in an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a principle framework of a preset frame generation model provided in an embodiment of the present application;

[0025] Figure 6 This is a schematic structural diagram of an embodiment of a video frame interpolation processing device provided in an embodiment of the present application;

[0026] Figure 7 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0029] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0030] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that one of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, well-known processes will not be elaborated in detail to avoid obscuring the description of the embodiments of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0031] Embodiments of the present application provide a video frame interpolation processing method, apparatus, computer equipment, and computer-readable storage medium.

[0032] The executor of the video frame interpolation processing method of the embodiment of the present application may be the video frame interpolation processing device provided in the embodiment of the present application, or the computer device provided in the embodiment of the present application, wherein the video frame interpolation processing device may be implemented in hardware or software.

[0033] The following embodiments of the present application are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0034] See also Figure 1 , Figure 1 101 is a flow chart of a video frame interpolation processing method provided in an embodiment of the present application. The video frame interpolation processing method includes steps 101 to 105, wherein:

[0035] 101. Determine, through a strategy generation layer of a target frame generation model, a first frame missing position of the video to be processed and a first frame-filling frequency of the first frame missing position based on motion information and detail information of the video to be processed.

[0036] The video to be processed is a video that needs to be frame-interpolated.

[0037] The first frame missing position is determined based on the motion information and detail information of the video to be processed, so as to determine the frame missing position of the video to be processed.

[0038] The first frame-filling frequency is determined based on the motion information and detail information of the video to be processed, to fill in the frame at the missing position of the first frame.

[0039] In order to better understand this embodiment, the target frame generation model in this embodiment is first introduced below. Figure 2 As shown, Figure 2 It is a schematic diagram of the principle framework of the target frame generation model provided in the embodiment of the present application. The target frame generation model may include a strategy generation layer, a feature extraction layer, a frame restoration layer, and an image generation layer. In some embodiments, the target frame generation model may further include a preprocessing layer. In some embodiments, the target frame generation model may further include a description characterization layer. The working principles of each layer of the target frame generation model are similar to the working principles of each module corresponding to the preset frame generation model (the strategy generation layer, the feature extraction layer, the frame restoration layer, and the image generation layer correspond to the strategy generation module, the feature extraction module, the frame restoration module, and the image generation module, respectively). For details, please refer to the relevant instructions below and will not be repeated here.

[0040] The motion information includes at least one of the object's motion speed, the number of moving objects, and the motion complexity.

[0041] The detail information includes at least one of scene detail richness, scene category, etc.

[0042] like Figure 2As shown, taking the example of motion information including object speed, number of moving objects, and motion complexity, and detail information including scene detail richness, first, the pre-processing layer of the target frame generation model can be used to extract key frames from the video to be processed to obtain a first key frame set of the video to be processed; and preliminary feature extraction processing such as color histogram and edge detection can be performed on each frame in the first key frame set to obtain frame features of each frame in the first key frame set. The frame features of each frame in the first key frame set can form a frame feature sequence of the video to be processed. Then, the strategy generation layer of the target frame generation model can be used to detect moving objects on each frame in the first key frame set based on the frame feature sequence of the video to be processed to obtain the motion speed, number of moving objects, and motion complexity of each frame in the first key frame set; and detail detection can be performed on each frame in the first key frame set to obtain the scene detail richness of each frame in the first key frame set; and based on the object motion speed, number of moving objects, motion complexity, and scene detail richness of each frame in the first key frame set, the first frame missing position and the first frame supplementation frequency can be determined.

[0043] 102. Perform feature extraction based on the frame feature sequence of the video to be processed and the first frame missing position through the feature extraction layer of the target frame generation model to obtain a first associated frame feature of the first frame missing position.

[0044] In some embodiments, step 102 may specifically include: performing multi-scale division on the frame feature sequence of the video to be processed based on the first frame missing position to obtain multiple first frame feature subsequences of different time scales, wherein the first frame feature subsequence of each time scale includes forward frame features and backward frame features of the first frame missing position; performing feature encoding on the first frame feature subsequence of each time scale to obtain encoding features of the first frame feature subsequence of each time scale; and fusing the encoding features of the multiple first frame feature subsequences of different time scales to obtain the first associated frame features. In order to enable the model to extract spatiotemporal features of the video at different resolutions, thereby improving the understanding and prediction capabilities of complex motion, the division sublayer may perform multi-time-scale division on the frame feature sequence of the input video to be processed based on the first frame missing position to obtain multiple first frame feature subsequences of different time scales; then, each encoding sublayer performs feature encoding on the first frame feature subsequence corresponding to a time scale to obtain encoding features of the first frame feature subsequence of each time scale; finally, the fusion sublayer fuses the encoding features of the multiple first frame feature subsequences of different time scales to obtain the first associated frame features of the first frame missing position.

[0045] Thus, by performing multi-scale partitioning on the frame feature sequence of the processed video, multiple first-frame feature subsequences at different time scales are obtained. Each first-frame feature subsequence at each time scale includes both forward and backward frame features of the first-frame missing location. This allows the model to extract spatiotemporal features at different video frame missing locations at different resolutions, thereby improving the first-association feature's ability to extract complex motion and, consequently, the accuracy of compensated frame prediction. Furthermore, since each first-frame feature subsequence at each time scale includes both forward and backward frame features of the first-frame missing location, multi-scale features are generated based on the first-frame missing location generated by the strategy generation layer. This allows for the generation of different multi-scale encoding features for different supplementary frame locations, improving feature expression and adaptability. Furthermore, since different videos generate different supplementary strategies (i.e., first-frame supplementary frequency and first-frame missing location), differentiated multi-scale encoding features can be captured for each video, improving the accuracy of supplementary frame generation to a certain extent, ensuring that the generated supplementary frames are more compatible with the video content type and enhancing the adaptability of supplementary frame generation.

[0046] In some embodiments, feature encoding is performed on adjacent frames at the location of the first frame loss to obtain local features at the location of the first frame loss. This local feature is then fused with the encoded features of multiple subsequences of first frame features at different time scales to obtain the first associated frame features. This allows the fusion of local details from the left and right adjacent frames at the location of the first frame loss, improving the first associated features' ability to extract complex motion and, consequently, the accuracy of the compensated frame prediction.

[0047] 103. Perform characterization processing on the description information of the video to be processed to obtain a first description feature of the video to be processed.

[0048] The description information of the video to be processed may include at least one of the resolution, sharpness, contrast, brightness, and scene category of the video to be processed. Since this description information can fully reflect the image quality and image content of the video to be processed, the first description feature is integrated to generate a target compensation frame for the missing position of the first frame. This allows the model to more specifically predict the target compensation frame for different types of video content (such as different image quality, different object motion, and different motion speeds) when performing frame complement prediction, thereby generating a target compensation frame that matches the corresponding type of video content, thereby improving the prediction accuracy of the compensation frame.

[0049] 104. Generate a target compensation frame for the missing position of the first frame based on the first associated frame feature, the first frame supplementation frequency, and the first description feature through the image generation layer of the target frame generation model.

[0050] The target compensation frame is a compensation frame at the missing position of the first frame obtained through prediction.

[0051] 105. Perform frame interpolation processing on the video to be processed based on the target compensation frame to obtain a frame-interpolated video.

[0052] Exemplarily, target compensation frames are added to the video to be processed according to the missing position of the first frame. When all target compensation frames have completed the correction process, a frame-filled video is obtained.

[0053] like Figure 3 and Figure 4 As shown, Figure 3 This is a flow chart of the preset frame generation model training provided in the embodiment of the present application. Figure 4 301 to 305.

[0054] 301. Determine, by a strategy generation module of a preset frame generation model, a second frame missing position of the sample video and a second frame supplementation frequency of the second frame missing position based on motion information and detail information of the sample video.

[0055] The frame missing position is used to indicate which frames in the video need to be supplemented before or after.

[0056] The second frame missing position is determined based on the motion information and detail information of the sample video to determine the frame missing position of the sample video.

[0057] The frame filling frequency is used to indicate the number of frames that need to be filled in at the position where the frame is missing.

[0058] The second frame-filling frequency is determined based on the motion information and detail information of the sample video to fill in the second frame missing position.

[0059] In order to better understand this embodiment, the preset frame generation model in this embodiment is first introduced below. Figure 5 As shown, Figure 5 This is a schematic diagram of the principle structure of the preset frame generation model provided in the embodiments of the present application. The preset frame generation model may include a strategy generation module, a feature extraction module, a frame restoration module, and an image generation module. In some embodiments, the preset frame generation model may also include a preprocessing module. In some embodiments, the preset frame generation model may also include a description and characterization module. The working principles of each module are as follows:

[0060] 1. Preprocessing module: used to segment the video, extract key frames, perform preliminary feature extraction processing on the key frames such as color histogram and edge detection, and obtain the frame feature sequence of the video.

[0061] 2. Strategy generation module: used to determine the frame missing position and the frame filling frequency of the frame missing position in the video based on the motion information of the video (such as the object motion speed, the number of moving objects, and the motion complexity) and detail information (such as the richness of scene details and scene category).

[0062] 3. Feature Extraction Module: This module is used to extract features based on the video's frame feature sequence and the location of frame loss, obtaining frame features associated with the frame loss location. In some embodiments, to enable the model to learn the spatiotemporal features of videos at different resolutions, thereby improving its understanding and prediction of complex motion, the feature extraction module may further include a partitioning submodule, multiple encoding submodules, and a fusion submodule. Each encoding submodule is used to perform feature encoding on a subsequence of frame features at a specific time scale. Among them, the division submodule can perform multi-time scale division on the frame feature sequence of the input video according to the frame missing position to obtain multiple frame feature subsequences of different time scales; then, multiple encoding submodules respectively perform feature encoding on the multiple frame feature subsequences of different time scales to obtain the encoding features of the frame feature subsequence of each time scale (such as encoding submodule 1 performs feature encoding on the frame feature subsequence of time scale a to obtain encoding feature 1 of the frame feature subsequence of time scale a, encoding submodule 2 performs feature encoding on the frame feature subsequence of time scale b to obtain encoding feature 2 of the frame feature subsequence of time scale b, and so on, encoding submodule i performs feature encoding on the frame feature subsequence of time scale x to obtain encoding feature i of the frame feature subsequence of time scale x); finally, the fusion submodule fuses the encoding features of the multiple frame feature subsequences of different time scales to obtain the associated frame features of the frame missing position.

[0063] In some embodiments, a multi-scale LSTM (Long Short-Term Memory) network can be used to construct multiple encoding submodules of the feature extraction module (wherein each LSTM unit serves as an encoding submodule) so that the feature extraction module can learn the spatiotemporal features of the video at different resolutions, thereby improving the understanding and prediction capabilities of complex motions. In some embodiments, in order to speed up training and improve the generalization ability of the model so that the model is more robust when facing new data, a residual connection structure and a jump structure can be added to each LSTM unit, thereby enabling the model to better retain image details and texture information, thereby generating clearer and more natural intermediate frames.

[0064] 4. Frame restoration module: used to generate forward restoration frames and backward restoration frames of the frame missing position based on the associated frame features and the frame filling frequency.

[0065] 5. Description and representation module: used to characterize the description information of the video and obtain the description features of the video.

[0066] 6. Image generation module: used to generate compensation frames for frame missing positions based on associated frame features and frame complementation frequency. In some embodiments, the image generation module can generate compensation frames for frame missing positions based on associated frame features and frame complementation frequency. In some embodiments, the image generation module can also generate compensation frames for frame missing positions based on the description features of the video, associated frame features and frame complementation frequency. In some embodiments, the image generation module can be constructed using a generator of a generative adversarial network. During the training process, an adversarial training method is adopted. The generator is responsible for generating new intermediate frames based on the information of adjacent key frames, while the discriminator is responsible for judging whether the generated frames are real. Through adversarial training, the generator can learn how to generate more natural and realistic intermediate frames.

[0067] In some embodiments, an adaptive weight update mechanism can be introduced into the pre-set frame generation model, enabling the model to dynamically adjust weights based on the content of the input video to better adapt to different scenes and motion types. This allows the model to better preserve image details and texture information, thereby generating clearer and more natural intermediate frames.

[0068] like Figure 5 As shown, first, the pre-processing module of the preset frame generation model can be used to extract key frames from the sample video to obtain the second key frame set of the sample video; and preliminary feature extraction processing such as color histogram and edge detection can be performed on each frame in the second key frame set to obtain the frame features of each frame in the second key frame set. The frame features of each frame in the second key frame set can form a frame feature sequence of the sample video. Then, the strategy generation module of the preset frame generation model can be used to detect moving objects on each frame in the second key frame set based on the frame feature sequence of the sample video to obtain the motion speed, number of moving objects and motion complexity of each frame in the second key frame set; detail detection can be performed on each frame in the second key frame set to obtain the scene detail richness of each frame in the second key frame set; based on the object motion speed, number of moving objects, motion complexity and scene detail richness of each frame in the second key frame set, the second frame missing position and the second frame supplementation frequency can be determined.

[0069] 302. Perform feature extraction based on the frame feature sequence of the sample video and the second frame missing position through the feature extraction module of the preset frame generation model to obtain a second associated frame feature of the frame missing position.

[0070] In some embodiments, in order to enable the model to learn the spatiotemporal features of the video at different resolutions, thereby improving the understanding and prediction capabilities of complex motion, the feature extraction module may further include a division submodule, multiple encoding submodules, and a fusion submodule. The division submodule can perform multi-time-scale division on the frame feature sequence of the input sample video according to the second frame missing position, and obtain multiple second frame feature subsequences of different time scales; then, each encoding submodule performs feature encoding on the second frame feature subsequence corresponding to a time scale, and obtains the encoding features of the second frame feature subsequence of each time scale; finally, the fusion submodule fuses the encoding features of the multiple second frame feature subsequences of different time scales to obtain the second associated frame features of the second frame missing position.

[0071] 303. Generate a forward restoration frame and a backward restoration frame for the missing position of the second frame based on the second associated frame feature and the second frame supplementation frequency through the frame restoration module of the preset frame generation model.

[0072] The forward restoration frame is a video frame obtained through prediction, which is adjacent to the missing position of the second frame and before the second missing position.

[0073] The backward restoration frame is a video frame obtained through prediction, which is adjacent to the missing position of the second frame and after the second missing position.

[0074] 304. Obtain a first loss between the forward restored frame and the forward actual frame at the second frame missing position, and a second loss between the backward restored frame and the backward actual frame at the second frame missing position.

[0075] The forward actual frame is a video frame that is actually adjacent to the second frame missing position and before the second frame missing position.

[0076] The backward actual frame is a video frame that is actually adjacent to the second frame missing position and is after the second frame missing position.

[0077] On the one hand, by constructing the first loss between the forward restored frame and the forward actual frame at the missing position of the second frame (such as Figure 5 As shown in L1 in the figure), the strategy generation module and feature extraction module of the preset frame generation model are used to train the strategy generation module and the feature extraction module, so that the strategy generation module can better learn to capture the motion information and detail information of the video and the relationship between the frame missing position, and the feature extraction module can capture the forward frame features of the frame missing position, thereby improving the strategy generation module's ability to capture the frame missing position and the feature extraction module's ability to capture the forward features of the compensation frame, thereby improving the accuracy of the video compensation frame generation. On the other hand, by constructing a second loss between the backward restored frame and the backward actual frame of the second frame missing position (as shown in the figure), the strategy generation module can better learn to capture the motion information and detail information of the video and the relationship between the frame missing position and the frame missing position, and the feature extraction module can capture the forward frame features of the compensation frame, thereby improving the accuracy of the video compensation frame generation. Figure 5L2 in the figure) is used to train the strategy generation module and feature extraction module of the preset frame generation model, so that the strategy generation module can better learn to capture the motion information and detail information of the video and associate them with the frame missing position, and enable the feature extraction module to capture the backward frame features of the frame missing position, thereby improving the strategy generation module's ability to capture the frame missing position and the feature extraction module's ability to capture the backward features of the compensation frame, thereby improving the accuracy of the compensation frame generation of the video.

[0078] 305. Based on the first loss and the second loss, adjust the model parameters of the preset frame generation model to obtain a trained frame generation model.

[0079] There are many ways to implement the training of the preset frame generation model in step 305, illustratively including:

[0080] (1) Using the first loss and the second loss. Exemplarily, at this time, step 305 may specifically include: adjusting the model parameters of the feature extraction module of the preset frame generation model using the first loss and the second loss, and adjusting the model parameters of the strategy generation module of the preset frame generation model using the first loss and the second loss, until a preset stop training condition is met, thereby obtaining a trained frame generation model.

[0081] (2) Using the first loss, the second loss and the third loss. For example, at this time, step 305 may specifically include: obtaining the third loss (such as Figure 5 L3 in the figure); using the first loss and the second loss to adjust the model parameters of the feature extraction module of the preset frame generation model, and using the first loss, the second loss and the third loss to adjust the model parameters of the strategy generation module of the preset frame generation model until the preset stop training condition is met, thereby obtaining the trained frame generation model.

[0082] Therefore, by constructing a third loss between the second frame missing position and the marked missing position of the sample video, it is used to train the strategy generation module of the preset frame generation model, so that the strategy generation module can better learn to capture the motion information and detail information of the video and associate it with the frame missing position, thereby improving the strategy generation module's ability to capture the frame missing position, and thus improving the accuracy of the video's compensation frame generation.

[0083] (3) Using the first loss, the second loss, and the fourth loss. Exemplarily, at this time, step 305 may specifically include: generating a sample compensation frame of the missing position of the second frame based on the second associated frame feature and the second frame supplementation frequency through the image generation module of the preset frame generation model; obtaining the fourth loss of the image generation module (such as Figure 5L4); using the first loss and the second loss to adjust the model parameters of the feature extraction module of the preset frame generation model, using the first loss and the second loss to adjust the model parameters of the strategy generation module of the preset frame generation model, and using the fourth loss to adjust the model parameters of the image generation module of the preset frame generation model, until the preset stop training condition is met, the trained frame generation model is obtained.

[0084] Furthermore, in some embodiments, the image generation module may also generate a sample compensation frame at the second frame missing position according to the second description feature, the second associated frame feature and the second frame supplementation frequency of the sample video.

[0085] Therefore, by utilizing the sample compensation frame, the fourth loss of the image generation module is constructed, which is used to train the image generation module of the preset frame generation model, thereby improving the missing frame fitting ability of the image generation module and thereby improving the accuracy of the compensation frame generation of the video.

[0086] (4) Using the first loss, the second loss, the third loss, and the fourth loss. Exemplarily, at this time, step 305 may specifically include: obtaining the third loss between the second frame missing position and the marked missing position of the sample video; generating a sample compensation frame of the second frame missing position based on the second associated frame feature and the second supplementary frame frequency through the image generation module of the preset frame generation model; adjusting the model parameters of the feature extraction module of the preset frame generation model using the first loss and the second loss, adjusting the model parameters of the strategy generation module of the preset frame generation model using the first loss, the second loss, and the third loss, and adjusting the model parameters of the image generation module of the preset frame generation model using the fourth loss, until the preset stop training condition is met, thereby obtaining a trained frame generation model.

[0087] Thus, a trained frame generation model can be obtained by training. At this time, in some embodiments, the preprocessing module of the trained frame generation model can be extracted as the preprocessing layer of the target frame generation model, the strategy generation module can be used as the strategy generation layer of the target frame generation model, the feature extraction module can be used as the feature extraction layer of the target frame generation model, the description and representation module can be used as the description and representation layer of the target frame generation model, and the image generation module can be used as the image generation layer of the target frame generation model to build the target frame generation model. In some embodiments, the trained frame generation model can also be directly used as the target frame generation model in steps 101 to 105, wherein the preprocessing module of the trained frame generation model is used as the preprocessing layer of the target frame generation model, the strategy generation module is used as the strategy generation layer of the target frame generation model, the feature extraction module is used as the feature extraction layer of the target frame generation model, the description and representation module is used as the description and representation layer of the target frame generation model, and the image generation module is used as the image generation layer of the target frame generation model.

[0088] As can be seen from the above, firstly, by determining the first frame missing location and the first frame complement frequency of the first frame missing location based on the motion and detail information of the video to be processed, and then generating the target compensation frame using the first associated frame feature of the first frame missing location, the model can generate the target compensation frame by capturing the video's motion and detail information. This allows the model to generate target compensation frames for different types of video content, thus avoiding the problem of large differences in frame complement prediction accuracy for different types of video content. Secondly, by first determining the frame missing location and frame complement frequency based on motion and detail information, it is more convenient to formulate different frame complement strategies for different types of video content, thereby improving the accuracy of frame complement prediction for different types of video content. Thirdly, by using the first descriptive features obtained through characterization processing of the descriptive information to generate the target compensation frame, because this descriptive information can fully reflect the image quality and image content of the video to be processed, the integration of the first descriptive features to generate the target compensation frame allows the model to more specifically predict frame complements for different types of video content (such as different image quality, different object motion, and different motion speeds), generating target compensation frames that match the corresponding video content, thereby improving the prediction accuracy of the compensation frame.

[0089] In addition, in order to better implement the video frame interpolation processing method in the embodiment of the present application, based on the video frame interpolation processing method, the embodiment of the present application also provides a video frame interpolation processing device, such as Figure 6 FIG. 1 is a schematic diagram of a structure of an embodiment of a video frame interpolation processing device provided in an embodiment of the present application. The video frame interpolation processing device 600 includes:

[0090] A generating unit 601 is configured to determine, through a strategy generating layer of a target frame generation model, a first frame missing position of the video to be processed and a first frame filling frequency of the first frame missing position based on motion information and detail information of the video to be processed;

[0091] An extraction unit 602 is configured to perform feature extraction based on the frame feature sequence of the video to be processed and the first frame missing position through a feature extraction layer of the target frame generation model to obtain a first associated frame feature of the first frame missing position;

[0092] The characterization unit 603 is configured to perform characterization processing on the description information of the video to be processed to obtain a first description feature of the video to be processed;

[0093] A processing unit 604 is configured to generate a target compensation frame for the missing position of the first frame based on the first associated frame feature, the first frame complementation frequency, and the first description feature through the image generation layer of the target frame generation model;

[0094] The processing unit 604 is further configured to perform frame interpolation processing on the video to be processed based on the target compensation frame to obtain a frame-interpolated video.

[0095] In some embodiments, the motion information includes object motion speed, the number of moving objects, and motion complexity, and the detail information includes scene detail richness; the generating unit 601 is specifically configured to:

[0096] Extracting key frames from the video to be processed to obtain a first key frame set of the video to be processed;

[0097] Performing moving object detection on each frame in the first key frame set to obtain a motion speed, a number of moving objects, and a motion complexity of each frame in the first key frame set;

[0098] performing detail detection on each frame in the first key frame set to obtain scene detail richness of each frame in the first key frame set;

[0099] The first frame missing position and the first frame supplementation frequency are determined based on the object motion speed, the number of moving objects, the motion complexity and the richness of scene details of each frame in the first key frame set.

[0100] In some embodiments, the extraction unit 602 is specifically configured to:

[0101] Performing multi-scale division on the frame feature sequence of the video to be processed based on the first frame missing position to obtain a plurality of first frame feature subsequences of different time scales, wherein the first frame feature subsequence of each time scale includes forward frame features and backward frame features of the first frame missing position;

[0102] Perform feature encoding on the first frame feature subsequence of each time scale respectively to obtain the encoding features of the first frame feature subsequence of each time scale;

[0103] The encoding features of a plurality of first frame feature subsequences at different time scales are fused to obtain the first associated frame features.

[0104] In some embodiments, the extraction unit 602 is specifically configured to:

[0105] Performing feature encoding on adjacent frames at a location where the first frame is missing to obtain local features of the location where the first frame is missing;

[0106] The first associated frame feature is obtained by performing a fusion process based on the local feature and the encoding features of multiple first frame feature subsequences at different time scales.

[0107] In some embodiments, the video frame interpolation processing apparatus further includes a training unit (not shown in the figure), which is specifically configured to:

[0108] Determining, by a strategy generation module of a preset frame generation model, a second frame missing position of the sample video and a second frame filling frequency of the second frame missing position based on motion information and detail information of the sample video;

[0109] Performing feature extraction based on the frame feature sequence of the sample video and the second frame missing position by a feature extraction module of the preset frame generation model to obtain a second associated frame feature of the frame missing position;

[0110] Generate a forward restoration frame and a backward restoration frame of a missing position of the second frame based on the second associated frame feature and the second supplementary frame frequency by a frame restoration module of the preset frame generation model;

[0111] Acquire a first loss between the forward restored frame and the forward actual frame at the second frame missing position, and a second loss between the backward restored frame and the backward actual frame at the second frame missing position;

[0112] Based on the first loss and the second loss, the model parameters of the preset frame generation model are adjusted to obtain a trained frame generation model, and the target frame generation model is obtained based on the trained frame generation model.

[0113] In some embodiments, the training unit is specifically configured to:

[0114] Obtaining a third loss between the missing position of the second frame and the marked missing position of the sample video;

[0115] The adjusting the model parameters of the preset frame generation model based on the first loss and the second loss to obtain a trained frame generation model includes:

[0116] Based on the first loss, the second loss and the third loss, the model parameters of the preset frame generation model are adjusted to obtain a trained frame generation model.

[0117] In some embodiments, the training unit is specifically configured to:

[0118] Generate, by means of an image generation module of the preset frame generation model, a sample compensation frame for a missing position of the second frame based on the second associated frame feature and the second frame complementation frequency;

[0119] obtaining a fourth loss of the image generation module based on the sample compensation frame;

[0120] The adjusting the model parameters of the preset frame generation model based on the first loss and the second loss to obtain a trained frame generation model includes:

[0121] Based on the first loss, the second loss and the fourth loss, the model parameters of the preset frame generation model are adjusted to obtain a trained frame generation model.

[0122] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the above video frame interpolation processing method embodiment, which will not be repeated here.

[0123] See also Figure 7 , Figure 7 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0124] like Figure 7 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0125] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the video frame interpolation processing methods.

[0126] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0127] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any video frame interpolation processing method.

[0128] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0129] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0131] Through the strategy generation layer of the target frame generation model, based on the motion information and detail information of the video to be processed, the first frame missing position of the video to be processed and the first frame filling frequency of the first frame missing position are determined; through the feature extraction layer of the target frame generation model, feature extraction is performed based on the frame feature sequence of the video to be processed and the first frame missing position to obtain the first associated frame feature of the first frame missing position; the description information of the video to be processed is characterized to obtain the first description feature of the video to be processed; through the image generation layer of the target frame generation model, based on the first associated frame feature, the first frame filling frequency and the first description feature, a target compensation frame for the first frame missing position is generated; based on the target compensation frame, the video to be processed is frame filled to obtain a video after frame filling.

[0132] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned video frame interpolation processing method may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0133] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores multiple computer programs. The computer programs can be loaded by a processor to execute any video frame interpolation processing method provided in the embodiment of the present application.

[0134] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0135] In the above-described embodiments of the video frame interpolation processing device and computer-readable storage medium, the descriptions of each embodiment have different emphases. For portions not described in detail in one embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for ease and brevity of description, the specific operating processes and beneficial effects of the above-described video frame interpolation processing device, computer-readable storage medium, and corresponding units can be referred to in the description of the video frame interpolation processing method in the above-described embodiments, and will not be further elaborated here.

[0136] The above is a detailed introduction to a video frame interpolation processing method, device, computer equipment, and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application. The non-Company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

Claims

1. A video frame interpolation processing method, characterized in that: The method comprises: Determining, by a strategy generation layer of a target frame generation model, a first frame missing position of the video to be processed and a first frame filling frequency of the first frame missing position based on motion information and detail information of the video to be processed; The motion information includes object motion speed, number of moving objects and motion complexity, and the detail information includes scene detail richness; determining the first frame missing position of the video to be processed and the first frame filling frequency of the first frame missing position based on the motion information and detail information of the video to be processed includes: performing key frame extraction on the video to be processed to obtain a first key frame set of the video to be processed; performing moving object detection on each frame in the first key frame set to obtain the motion speed, number of moving objects and motion complexity of each frame in the first key frame set; performing detail detection on each frame in the first key frame set to obtain the scene detail richness of each frame in the first key frame set; determining the first frame missing position and the first frame filling frequency based on the object motion speed, number of moving objects, motion complexity and scene detail richness of each frame in the first key frame set; Performing feature extraction based on the frame feature sequence of the video to be processed and the first frame missing position through the feature extraction layer of the target frame generation model to obtain a first associated frame feature of the first frame missing position; Performing characterization processing on the description information of the video to be processed to obtain a first description feature of the video to be processed; Generate a target compensation frame for the missing position of the first frame based on the first associated frame feature, the first frame complement frequency, and the first description feature through the image generation layer of the target frame generation model; The video to be processed is subjected to frame interpolation processing based on the target compensation frame to obtain a frame-interpolated video.

2. The video frame interpolation processing method according to claim 1, wherein: The extracting features based on the frame feature sequence of the video to be processed and the first frame missing position to obtain a first associated frame feature of the first frame missing position includes: Performing multi-scale division on the frame feature sequence of the video to be processed based on the first frame missing position to obtain a plurality of first frame feature subsequences of different time scales, wherein the first frame feature subsequence of each time scale includes forward frame features and backward frame features of the first frame missing position; Perform feature encoding on the first frame feature subsequence of each time scale respectively to obtain the encoding features of the first frame feature subsequence of each time scale; The encoding features of a plurality of first frame feature subsequences at different time scales are fused to obtain the first associated frame features.

3. The video frame interpolation processing method according to claim 2, wherein: The fusing the encoding features of the first frame feature subsequences at different time scales to obtain the first associated frame features includes: Performing feature encoding on adjacent frames at a location where the first frame is missing to obtain local features of the location where the first frame is missing; The first associated frame feature is obtained by performing a fusion process based on the local feature and the encoding features of multiple first frame feature subsequences at different time scales.

4. The video frame interpolation processing method according to claim 1, wherein: The method further comprises: Determining, by a strategy generation module of a preset frame generation model, a second frame missing position of the sample video and a second frame filling frequency of the second frame missing position based on motion information and detail information of the sample video; Performing feature extraction based on the frame feature sequence of the sample video and the second frame missing position by a feature extraction module of the preset frame generation model to obtain a second associated frame feature of the frame missing position; Generate a forward restoration frame and a backward restoration frame of a missing position of the second frame based on the second associated frame feature and the second supplementary frame frequency by a frame restoration module of the preset frame generation model; Acquire a first loss between the forward restored frame and the forward actual frame at the second frame missing position, and a second loss between the backward restored frame and the backward actual frame at the second frame missing position; Based on the first loss and the second loss, the model parameters of the preset frame generation model are adjusted to obtain a trained frame generation model, and the target frame generation model is obtained based on the trained frame generation model.

5. The video frame interpolation processing method according to claim 4, characterized in that: The method further comprises: Obtaining a third loss between the missing position of the second frame and the marked missing position of the sample video; The adjusting the model parameters of the preset frame generation model based on the first loss and the second loss to obtain a trained frame generation model includes: Based on the first loss, the second loss and the third loss, the model parameters of the preset frame generation model are adjusted to obtain a trained frame generation model.

6. The video frame interpolation processing method according to claim 4, characterized in that: The method further comprises: Generate, by means of an image generation module of the preset frame generation model, a sample compensation frame for a missing position of the second frame based on the second associated frame feature and the second frame complementation frequency; obtaining a fourth loss of the image generation module based on the sample compensation frame; The adjusting the model parameters of the preset frame generation model based on the first loss and the second loss to obtain a trained frame generation model includes: Based on the first loss, the second loss and the fourth loss, the model parameters of the preset frame generation model are adjusted to obtain a trained frame generation model.

7. A video frame interpolation processing device, characterized in that: The video frame interpolation processing device comprises: a generating unit configured to determine, by a strategy generating layer of a target frame generating model, a first frame missing position of the video to be processed and a first frame filling frequency of the first frame missing position based on motion information and detail information of the video to be processed; The motion information includes object motion speed, number of moving objects, and motion complexity, and the detail information includes scene detail richness; the generation unit is specifically configured to: perform key frame extraction on the video to be processed to obtain a first key frame set of the video to be processed; perform moving object detection on each frame in the first key frame set to obtain motion speed, number of moving objects, and motion complexity of each frame in the first key frame set; perform detail detection on each frame in the first key frame set to obtain scene detail richness of each frame in the first key frame set; and determine the first frame missing position and the first frame supplementation frequency based on the object motion speed, number of moving objects, motion complexity, and scene detail richness of each frame in the first key frame set; an extraction unit, configured to perform feature extraction based on a frame feature sequence of the video to be processed and the first frame missing position through a feature extraction layer of the target frame generation model, to obtain a first associated frame feature of the first frame missing position; a characterization unit, configured to perform characterization processing on the description information of the video to be processed to obtain a first description feature of the video to be processed; A processing unit, configured to generate, by means of an image generation layer of the target frame generation model, a target compensation frame for a missing position of the first frame based on the first associated frame feature, the first frame complementation frequency, and the first description feature; The processing unit is further configured to perform frame interpolation processing on the video to be processed based on the target compensation frame to obtain a frame-interpolated video.

8. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the video frame interpolation processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the video frame interpolation processing method according to any one of claims 1 to 6.

Images