Video frame processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a video frame processing method, device, electronic device and storage medium, the video frame processing method comprising: obtaining a second video frame sequence of multiple scales corresponding to a first video frame sequence, the first video frame sequence comprising multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale comprises a second video frame of the scale corresponding to each first video frame; for a first scale and a second scale of the multiple scales, obtaining motion information of the second video frame of the first scale relative to a target video frame of the first scale, and obtaining a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale, the second scale being larger than the first scale.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video frame processing method, device, electronic device and storage medium for video frame insertion. Background Art

[0002] Video Frame Interpretation (VFI) aims to use a set of consecutive video frames as input and synthesize images of intermediate frames by interpolation. VFI plays an important role in many applications, such as slow motion production, frame rate up-conversion, and video restoration. VFI is a challenging task, especially for cases involving objects with large motion. Large motion refers to the situation where the distance moved by pixels between two adjacent frames of a video is large. Not only because it is an under-constrained problem, but also because complex deformations and occlusions will severely affect the performance of many existing methods that rely on accurate correspondence between frames.

[0003] The VFI method of the related art can use a simple autoencoder as the backbone network to extract features from two frames of input and perform motion prediction, but the backbone network is difficult to handle large motion situations and will produce ghosting problems in large motion situations. Therefore, a solution is needed that can effectively solve the problem of inaccurate video interpolation under large motion. Summary of the invention

[0004] According to a first aspect of the present disclosure, a video processing method is provided, including: a video processing method, characterized in that it includes: obtaining a second video frame sequence of multiple scales corresponding to a first video frame sequence, the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of each scale corresponding to each first video frame; for a first scale and a second scale of the multiple scales, obtaining motion information of the second video frame of the first scale relative to a target video frame of the first scale, and obtaining a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale, the second scale being larger than the first scale.

[0005] According to a first aspect of the present disclosure, the method includes: when the second scale is a maximum scale of the multiple scales, inserting a target video frame of the second scale into the first video frame sequence.

[0006] According to a first aspect of the present disclosure, the method includes: when the second scale is smaller than the maximum scale of the multiple scales, taking the second scale as the new first scale and taking a third scale as the new second scale, repeatedly executing the process of acquiring the motion information, and acquiring the target video frame of the second scale according to the motion information and the second video frame of the second scale, until the second scale is the maximum scale of the multiple scales.

[0007] According to a first aspect of the present disclosure, obtaining a target video frame of the second scale according to motion information of the first scale and a second video frame of the second scale includes: obtaining fusion features of the second video frame of the second scale; amplifying the motion information of the first scale; obtaining correction parameters for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion features of the second video frame of the second scale, and the amplified motion information of the first scale; obtaining motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameters of the motion information of the first scale and the amplified motion information of the first scale; and obtaining the target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale.

[0008] According to a first aspect of the present disclosure, obtaining fused features of the second video frame at the second scale includes: combining the second video frames at the second scale, and extracting features of the combined second video frames at the second scale to obtain the fused features of the second video frame at the second scale.

[0009] According to a first aspect of the present disclosure, a correction parameter for the enlarged motion information of the first scale is obtained based on the second video frame of the second scale, the fusion features of the second video frame of the second scale, and the enlarged motion information of the first scale, including: based on the second video frame of the second scale, the fusion features of the second video frame of the second scale, and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale.

[0010] According to a first aspect of the present disclosure, obtaining motion information of a second video frame at the second scale relative to a target video frame at the second scale according to a correction parameter of the motion information at the first scale and the amplified motion information at the first scale includes: obtaining a deformable convolution kernel at the second scale based on the residual and the amplified deformable convolution kernel at the first scale.

[0011] According to a first aspect of the present disclosure, predicting a residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale includes: converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting depth features of the second video frame of the second scale and the converted video frame; combining the extracted depth features and the fusion features of the second video frame of the second scale, and predicting the residual based on the combined features.

[0012] According to a first aspect of the present disclosure, converting the enlarged deformable convolution kernel of the first scale into a video frame includes: performing deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0013] According to a first aspect of the present disclosure, extracting depth features of the second video frame at the second scale and the converted video frame includes: combining the converted video frame and the second video frame at the second scale; and extracting depth features of the combined video frame layer by layer.

[0014] According to a first aspect of the present disclosure, predicting the residual based on the combined features includes: predicting a correction term for the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale based on the combined features, and a weight and a convolution kernel of the pixels of the second video frame of the second scale, wherein obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale includes: correcting the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0015] According to a second aspect of the present disclosure, a video frame processing device is provided, including: a video frame acquisition unit, configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, the first video frame sequence including multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of the scale corresponding to each first video frame at the each scale; a video frame processing unit, configured to acquire, for a first scale and a second scale of the multiple scales, motion information of the second video frame of the first scale relative to a target video frame of the first scale, and acquire a target video frame of the second scale based on the motion information of the first scale and the second video frame of the second scale, the second scale being larger than the first scale.

[0016] According to a second aspect of the present disclosure, the video frame processing unit is configured to: when the second scale is a maximum scale of the multiple scales, insert the target video frame of the second scale into the first video frame sequence.

[0017] According to a second aspect of the present disclosure, the video frame processing unit is configured to: when the second scale is smaller than the maximum scale of the multiple scales, use the second scale as the new first scale and the third scale as the new second scale, and repeatedly execute the process of obtaining the motion information, and obtaining the target video frame of the second scale according to the motion information and the second video frame of the second scale, until the second scale is the maximum scale of the multiple scales.

[0018] According to a second aspect of the present disclosure, the video frame processing unit is configured to: obtain a fusion feature of a second video frame at the second scale; amplify the motion information of the first scale; obtain a correction parameter for the amplified motion information of the first scale based on the second video frame at the second scale, the fusion feature of the second video frame at the second scale, and the amplified motion information of the first scale; obtain motion information of the second video frame at the second scale relative to a target video frame at the second scale according to the correction parameter of the motion information at the first scale and the amplified motion information of the previous scale; and obtain the target video frame at the second scale according to the motion information at the second scale and the second video frame at the second scale.

[0019] According to a second aspect of the present disclosure, the video frame processing unit is configured to: combine the second video frames of the second scale, and extract features of the combined second video frames of the second scale to obtain fused features of the second video frames of the second scale.

[0020] According to a second aspect of the present disclosure, a video frame processing unit is configured to: predict a residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes a pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and a weight and convolution kernel of the pixels of the second video frame of each scale; and obtain the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale.

[0021] According to a second aspect of the present disclosure, the video frame processing unit is configured to: convert the enlarged deformable convolution kernel of the first scale into a video frame; extract depth features of the second video frame of the second scale and the converted video frame; combine the extracted depth features and the fusion features of the second video frame of the second scale, and predict the residual based on the combined features.

[0022] According to a second aspect of the present disclosure, the video frame processing unit is configured to: perform deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0023] According to a second aspect of the present disclosure, the video frame processing unit is configured to: combine the converted video frame and the second video frame of the second scale; and extract depth features of the combined video frame layer by layer.

[0024] According to a second aspect of the present disclosure, the video frame processing unit is configured to: predict a correction term of a pixel coordinate offset of a second video frame of the second scale relative to a target video frame of the second scale based on the combined features, and a weight and a convolution kernel of pixels of the second video frame of the second scale; and correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0025] According to a third aspect of the present disclosure, a method for training a video processing model is provided, comprising: obtaining a second video frame sequence of multiple scales corresponding to a first video frame sequence, the first video frame sequence including multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale including second video frames of each scale corresponding to each first video frame; for the first scale and the second scale of the multiple scales, obtaining motion information of the second video frame of the first scale relative to a target video frame of the first scale through a video frame processing model, and obtaining a target video frame of the second scale based on the motion information of the first scale and the second video frame of the second scale, the second scale being larger than the first scale; and training the video processing model based on a comparison between the obtained target image frame of the second scale and the training video frame of the second scale.

[0026] According to a third aspect of the present disclosure, the training method includes: when the second scale is smaller than the maximum scale of the multiple scales, taking the second scale as the new first scale and the third scale as the new second scale, repeatedly executing the process of obtaining the motion information, obtaining the target video frame of the second scale according to the motion information and the second video frame of the second scale, and training the video processing model according to the comparison between the target image frame of the second scale and the training video frame of the second scale, until the second scale is the maximum scale of the multiple scales.

[0027] According to a third aspect of the present disclosure, obtaining a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale includes: obtaining a fusion feature of the second video frame of the second scale; amplifying the motion information of the first scale; obtaining a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale; obtaining motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameter of the motion information of the first scale and the amplified motion information of the first scale; and obtaining the target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale.

[0028] According to a third aspect of the present disclosure, obtaining fusion features of the second video frame of the second scale includes: combining the second video frames of the second scale, and extracting features of the combined second video frames of the second scale to obtain the fusion features of the second video frame of the second scale.

[0029] According to a third aspect of the present disclosure, obtaining correction parameters for the enlarged motion information of the first scale based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged motion information of the first scale includes: based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale, wherein obtaining the motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameters of the motion information of the first scale and the enlarged motion information of the first scale includes: obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale.

[0030] According to a third aspect of the present disclosure, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale includes: converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting depth features of the second video frame of the second scale and the converted video frame; combining the extracted depth features and the fusion features of the second video frame of the second scale, and predicting the residual based on the combined features.

[0031] According to a third aspect of the present disclosure, converting the enlarged deformable convolution kernel of the first scale into a video frame includes: performing deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0032] According to a third aspect of the present disclosure, extracting depth features of the second video frame at the second scale and the converted video frame includes: combining the converted video frame and the second video frame at the second scale; and extracting depth features of the combined video frame layer by layer.

[0033] According to a third aspect of the present disclosure, predicting the residual based on the combined features includes: predicting a correction term for the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale based on the combined features, and a weight and a convolution kernel of the pixels of the second video frame of the second scale, wherein obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale includes: correcting the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0034] According to a third aspect of the present disclosure, training a video processing model based on a comparison between the acquired target image frame of the second scale and the training video frame of the second scale includes: adjusting parameters of the video processing model based on a loss function constructed based on the acquired target image frame of the second scale and the training image frame of the second scale.

[0035] According to a fourth aspect of the present disclosure, a training device for a video processing model is provided, comprising: a video frame acquisition unit, configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, the first video frame sequence including multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of each scale corresponding to each first video frame; a video frame processing unit, configured to acquire, for a first scale and a second scale of the multiple scales, motion information of the second video frame of the first scale relative to a target video frame of the first scale through a video frame processing model, and acquire a target video frame of the second scale based on the motion information of the first scale and the second video frame of the second scale, the second scale being larger than the first scale; and a training unit, configured to train the video processing model based on a comparison between the acquired target image frame of the second scale and the training video frame of the second scale.

[0036] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: when the second scale is smaller than the maximum scale of the multiple scales, use the second scale as the new first scale and the third scale as the new second scale, and repeat the process of obtaining the motion information, and obtaining the target video frame of the second scale according to the motion information and the second video frame of the second scale; the training unit is configured to repeat the process of training the video processing model according to the comparison between the obtained target image frame of the second scale and the training video frame of the second scale until the second scale is the maximum scale of the multiple scales.

[0037] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: obtain a fusion feature of the second video frame at the second scale; amplify the motion information of the first scale; obtain a correction parameter for the amplified motion information of the first scale based on the second video frame at the second scale, the fusion feature of the second video frame at the second scale, and the amplified motion information of the first scale; obtain motion information of the second video frame at the second scale relative to a target video frame at the second scale according to the correction parameter of the motion information at the first scale and the amplified motion information of the first scale; obtain the target video frame of the second scale according to the motion information of the second scale and the second video frame at the second scale.

[0038] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: combine the second video frames of the second scale, and extract features of the combined second video frames of the second scale to obtain fused features of the second video frames of the second scale.

[0039] According to a fourth aspect of the present disclosure, a video frame processing unit is configured to: predict a residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes a pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and a weight and a convolution kernel of the pixels of the second video frame of each scale; and obtain the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale.

[0040] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: convert the enlarged deformable convolution kernel of the first scale into a video frame; extract depth features of the second video frame of the second scale and the converted video frame; combine the extracted depth features and the fusion features of the second video frame of the second scale, and predict the residual based on the combined features.

[0041] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: perform deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0042] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: combine the converted video frame and the second video frame of the second scale; and extract depth features of the combined video frame layer by layer.

[0043] According to a fourth aspect of the present disclosure, the video frame processing unit is configured to: predict a correction term of a pixel coordinate offset of a second video frame of the second scale relative to a target video frame of the second scale based on the combined features, and a weight and a convolution kernel of pixels of the second video frame of the second scale; and correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0044] According to a fourth aspect of the present disclosure, the training unit is configured to adjust parameters of the video processing model based on a loss function constructed according to the acquired target image frame of the second scale and the training image frame of the second scale.

[0045] According to a fifth aspect of the present disclosure, a video processing system is provided, comprising: a video frame acquisition unit, configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, the first video frame sequence comprising multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale comprises a second video frame of the scale corresponding to each of the first video frames; a video frame processing network, comprising a plurality of video frame processing modules connected in sequence, each of the plurality of video frame processing modules corresponding to each scale of the multiple scales, respectively, wherein each video frame processing module is configured to: acquire motion information of a second video frame of the first scale corresponding to a module preceding the video frame processing module relative to a target video frame of the first scale, and acquire a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale corresponding to each video frame processing module.

[0046] According to a fifth aspect of the present disclosure, the video frame processing network includes a feature extraction network, which is configured to obtain fused features of the second video frame at each scale of the multiple scales.

[0047] According to a fifth aspect of the present disclosure, the feature extraction network includes: a combination module, configured to combine the second video frames at each scale; and a feature extraction module, configured to extract features of the combined second video frames to obtain fused features of the second video frames at each scale.

[0048] According to a fifth aspect of the present disclosure, a video frame processing module at each scale includes: an amplification module, configured to amplify motion information of the first scale; a motion information acquisition module, configured to acquire correction parameters for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the amplified motion information of the first scale, and acquire motion information of the second video frame of the second scale relative to a target video frame of the second scale according to the correction parameters of the motion information of the first scale and the amplified motion information of the first scale; and an output module, configured to acquire the target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale.

[0049] According to a fifth aspect of the present disclosure, the motion information acquisition module is configured to: based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predict the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, and acquire the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale.

[0050] According to a fifth aspect of the present disclosure, the motion information acquisition module includes: a deformable convolution module, configured to convert the enlarged deformable convolution kernel of the first scale into a video frame; a depth feature extraction module, configured to extract depth features of the second video frame of the second scale and the converted video frame; a residual prediction module, configured to combine the extracted depth features and the fusion features of the second video frame of the second scale, and predict the residual based on the combined features.

[0051] According to a fifth aspect of the present disclosure, the deep feature extraction module includes: a combination module configured to perform combination on the converted video frame and the second video frame of the second scale; and multiple channel attention modules configured to extract deep features of the combined image frame layer by layer.

[0052] According to a fifth aspect of the present disclosure, the residual prediction module includes: a combination module, configured to combine the extracted depth features and the fusion features of the second video frame of the second scale; a prediction neural network, configured to predict a correction term for the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale based on the combined features, as well as the weights and convolution kernels of the pixels of the second video frame of the second scale; a correction module, configured to correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0053] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the video frame processing method and / or the video frame processing model training method as described above.

[0054] According to the seventh aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is enabled to execute the video frame processing method and / or the video frame processing model training method as described above.

[0055] According to an eighth aspect of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor to execute the video frame processing method and / or the video frame processing model training method as described above.

[0056] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: multiple hierarchical scales are used to gradually enlarge the image frames, and the correlation between image frames can be used to optimize the accuracy of deformable convolution kernels at each scale, so that stable and accurate processing effects can be achieved for image frames with large motions, and the implemented structure is simple, and it performs well in high-resolution video processing.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0059] Figure 1is a schematic diagram showing an overall framework of a video frame processing scheme for video frame insertion according to an exemplary embodiment of the present disclosure.

[0060] Figure 2 is a flowchart illustrating a video frame processing method according to an exemplary embodiment of the present disclosure.

[0061] Figure 3 is a structural schematic diagram showing a module for correcting a deformable convolution kernel used in a video frame processing method according to an exemplary embodiment of the present disclosure.

[0062] Figure 4 is a schematic diagram illustrating a video frame processing apparatus according to an exemplary embodiment of the present disclosure.

[0063] Figure 5 It is a schematic diagram showing the effect of a video frame processing method according to an exemplary embodiment of the present disclosure.

[0064] Figure 6 FIG. 4 is another schematic diagram showing the effect of a video frame processing method according to an exemplary embodiment of the present disclosure.

[0065] Figure 7 is a flowchart illustrating a training method of a video frame processing model according to an exemplary embodiment of the present disclosure.

[0066] Figure 8 is a block diagram illustrating a video frame processing model training apparatus according to an exemplary embodiment of the present disclosure.

[0067] Fig. 9 is a schematic diagram illustrating a video frame processing system according to an exemplary embodiment of the present disclosure.

[0068] Fig.10 A schematic diagram showing an electronic device for video frame processing according to an exemplary embodiment of the present disclosure is shown.

[0069] Fig.11 is a schematic diagram illustrating an electronic device for video frame processing according to another exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0072] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0073] According to an exemplary embodiment of the present disclosure, a video frame processing scheme for video interpolation is to convert large motions into small motions and perform motion estimation after downsampling the video frames, and then gradually amplify the resolution of the video frames and adjust the motion estimation between video frames, thereby improving the robustness against large motions.

[0074] Figure 1 It is a schematic diagram of the overall framework of a video frame processing scheme for video interpolation according to an exemplary embodiment of the present disclosure. It should be understood that the video frame processing scheme according to the exemplary embodiment of the present disclosure can be implemented in any device with video processing capabilities. For example, the method can be performed in a terminal device with video processing capabilities. Here, the terminal device can be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a netbook, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device. Applications such as local video playback applications, short video applications, network video platform applications, game applications, etc. can be run on the terminal device, and video processing can be implemented in the application. The method can also be performed on a dedicated video processing device such as a server. For example, a user can upload the video to be processed to a server on the terminal device, and the server performs processing on the uploaded video frame, and returns the processed video to the terminal device. The exemplary embodiments of the present disclosure do not limit it.

[0075] like Figure 1 As shown, the framework may generally include a multi-scale feature extraction part 110 and a motion prediction and correction part 120 .

[0076] The multi-scale feature extraction section 110 is used to extract multi-scale features of input video frames. The input video frames are multiple video frames to be processed and used to generate interpolated intermediate video frames. For example, Figure 1 4 video frames are used as input frames, and the 4 video frames can be 4 consecutive frames in a video frame sequence. It should be understood that the number and order of video frames are only examples, and those skilled in the art can adjust them as needed. The multi-scale feature extraction part can convolve the input video frame, and then extract the multi-scale features of the input video frame (hereinafter also referred to as multi-scale fusion features) through a hierarchical neural network, so as to obtain Figure 1 The pyramid-shaped time domain features shown. The multi-scale features here refer to the fusion features of multiple input video frames at multiple lower resolutions. For example, assuming that the resolution of the input video frame is 4K (3840×2160) and 4 scales are used, the feature extraction part 110 can obtain video frames with resolutions of 1920×1080, 960×540, and 480×2100 by downsampling the input video frame, and perform feature extraction on the downsampled video frame and the video frame with the original resolution to obtain the fusion features of the four scales. Figure 1 FIG. 4 shows that an L-HRnet neural network is used to perform feature extraction, but those skilled in the art should understand that any other multi-scale feature extraction network with deep feature extraction may be used instead.

[0077] The fused multi-scale features contain the temporal and spatial information between the image frames of each scale and the target image frame (i.e., the image frame to be obtained), so the input motion prediction and correction part 120 can initialize the motion information of the lowest scale, estimate the motion of the fused features of each scale layer by layer according to the multi-scale features and perform correction, and obtain the final motion information through multiple iterations of the upsampling deformable convolution module (UDBlock). Figure 1 As shown, the motion prediction and correction part 120 may have a plurality of UDBlocks corresponding to the resolution scale. In each layer of UDBlock, the deformable convolution kernel K including motion information obtained in the previous layer is s Upsample (enlarge) and match the multi-scale feature F of this layer s And the input image frame I s, the residual ΔKs of the deformable convolution kernel at this scale can be obtained, where s is a positive integer corresponding to the level of each scale, and 0 < s <= L, and L represents the total number of scales. This residual is used to correct the motion vector obtained in the previous layer, so as to obtain an accurate deformable convolution kernel at this resolution scale. And so on, multiple stages can be used to layer by layer optimize the accuracy of the deformable convolution kernel at different scales, and finally obtain the processed interpolated frame image (I L,out ).

[0078] In addition, during the training process, at each scale stage, the output image at this scale can also be compared with the training target image, so as to supervise and adjust the parameters of the entire architecture. For example, according to the comparison result, the operation parameters of the feature extraction network and / or the UDBlock can be adjusted through an auxiliary loss function.

[0079] The following will be combined with Figure 2 to detail the flowchart of the method for implementing video processing through the Figure 1 architecture.

[0080] First, in step S210, a second video frame sequence of multiple scales corresponding to the first video frame sequence is obtained. The first video frame sequence includes multiple first video frames. For each scale of the multiple scales, the second video frame sequence of each scale includes second video frames corresponding to each of the first video frames and having a scale of each scale. Each scale corresponds to a resolution. The second video frame sequence of other resolutions except the original resolution can be obtained by performing at least one downsampling on the original video frame sequence (that is, the first video frame sequence), that is, obtaining the second video frame corresponding to each resolution scale. For example, 4 consecutive video frames can be used as the first video frame sequence, and L = 4 resolution scales are used, that is, the original video frame sequence is downsampled 3 times to obtain 4 second video frame sequences of different resolutions: the second video frame sequence 1 with the original resolution 1, the second video frame sequence 2 with the resolution 2 after 1 downsampling, the second video frame sequence 3 with the resolution 3 after 2 downsamplings, and the second video frame sequence 4 with the resolution 4 after 3 downsamplings. Among them, the second video frame sequence 1 is the video frame sequence of scale 1 corresponding to the original resolution 1, the second video frame sequence 2 is the video frame sequence of scale 2 corresponding to the resolution 2, the second video frame sequence 3 is the video frame sequence of scale 3 corresponding to the resolution 3, and the second video frame sequence 4 is the video frame sequence of scale 4 corresponding to the resolution 4.

[0081] In one embodiment of the present application, each downsampling multiple performed on the original video frame sequence is different, and each scale obtained is also different. This is not limited here. The first video frame sequence can be a video frame obtained by various means, for example, it can be a video frame read from the local storage of the electronic device, a video frame obtained from an external network or an external device, or a video frame shot or generated in real time, which is not limited here.

[0082] In step S220, for a first scale and a second scale of the multiple scales, motion information of a second video frame at the first scale relative to a target video frame at the first scale is obtained, and a target video frame at the second scale is obtained according to the motion information at the first scale and the second video frame at the second scale, wherein the second scale is larger than the first scale.

[0083] As described above, the self-encoder of the related art needs to downsample the input features and then use transposed convolution to upsample. On the one hand, this destroys the detailed information of the image, easily causes the loss of image features, and has low parameter utilization. On the other hand, it does not utilize the correlation between image frames of different scales, making it difficult to handle large motion situations, and will cause ghosting problems in the case of large motion. The video processing method according to an exemplary embodiment of the present disclosure uses the motion information of a smaller-scale video frame and a larger-scale video frame to obtain a larger-scale target video frame. The motion correlation between the smaller-scale target video frame and the larger-scale target video frame obtained by the desired interpolation can be used to more accurately predict the motion between the larger-scale video frame and the target video frame, so that more accurate interpolation processing can be performed under large motion.

[0084] According to an exemplary embodiment of the present disclosure, when the second scale is smaller than the maximum scale of the multiple scales, the second scale is used as the new first scale, the third scale is used as the new second scale, and the process of acquiring the motion information and acquiring the target video frame of the second scale according to the motion information and the second video frame of the second scale is repeated until the second scale is the maximum scale of the multiple scales.

[0085] That is, at each scale, the motion information of the previous scale can be used to obtain the motion information of the scale, and then the target video frame can be obtained based on the motion information of the scale until the maximum scale (for example, the original resolution of the video) is reached. In this way, the larger motion at the original resolution scale can be decomposed into smaller motions, thereby improving the accuracy of motion prediction.

[0086] After obtaining the target video frame of the maximum scale, in step S230, the target video frame of the second scale (i.e., the maximum scale) may be inserted into the first video frame sequence, thereby completing the video frame insertion process. Here, the target video frame may be inserted into a position in the first video frame sequence. For example, if four video frames are used for frame insertion, the obtained target video frame may be inserted between the second video frame and the third video frame.

[0087] According to an exemplary embodiment of the present disclosure, the operation of acquiring a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale in step S220 may include: acquiring a fusion feature of the second video frame of the second scale; amplifying the motion information of the first scale; acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale; acquiring motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameter of the motion information of the first scale and the amplified motion information of the first scale; and acquiring the target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale.

[0088] That is to say, the amplified motion information of the previous scale may have errors relative to the expected motion information of the current scale. Therefore, the amplified motion information can be corrected by the video frame of the current scale and its fusion features to obtain accurate motion information of the current scale.

[0089] According to an exemplary embodiment of the present disclosure, obtaining the fusion feature of the second video frame at the second scale may include: combining the second video frames at the second scale, and extracting features of the combined second video frames at the second scale to obtain the fusion feature of the second video frame at the second scale. Figure 1 As shown in FIG. 1 , the first video frame sequence can be combined through a cascade module, and a hierarchical neural network such as L-HRnet can be used to extract fusion features F1, F2, ..., F at each scale. L .

[0090] According to an exemplary embodiment of the present disclosure, a deformable convolution kernel at each scale can be used to obtain motion information of the corresponding scale. The deformable convolution kernel at each scale may include the pixel coordinate offset of the second video frame at each scale relative to the target video frame at each scale and the weight and convolution kernel of the pixels of the second video frame at each scale. That is, the deformable convolution kernel at each scale contains the motion information of the second video frame at each scale relative to the target video frame.

[0091] Accordingly, obtaining correction parameters for the motion information of the first scale after enlargement based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the motion information of the first scale after enlargement may include: based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the deformable convolution kernel of the first scale after enlargement, predicting the residual of the deformable convolution kernel of the first scale after enlargement relative to the deformable convolution kernel of the second scale as the correction parameter. The embodiment of the present disclosure uses a deformable convolution kernel to represent motion information, which can accurately obtain the difference in motion information between different scales, and correct the motion information of each scale based on the difference in motion information, that is, converting the correction of motion information at different scales into correction between deformable convolution kernels, which can accurately reflect the change of motion information at different scales. Accordingly, according to an exemplary embodiment of the present disclosure, obtaining the motion information of the second video frame of the second scale relative to the target video frame of the second scale based on the correction parameters of the motion information of the first scale and the motion information of the first scale after enlargement includes: obtaining the deformable convolution kernel of the second scale based on the residual and the deformable convolution kernel of the first scale after enlargement. The following will refer to Figure 1 To illustrate the process of modifying a deformable convolution kernel between different scales.

[0092] like Figure 1 As shown, the deformable convolution kernel K0 can be initialized first. For example, all vectors of K0 can be set to 0, that is, no transformation is performed on the deformable convolution kernel. The initialized deformable convolution kernel K0 can then be upsampled and input into the first UDBlock, that is, the lowest scale UDBlock corresponding to the lowest resolution. In this UDBlock, a correction term for K0 is obtained based on the lowest scale fusion feature F1, the lowest scale input video frame I1 and the upsampled deformable convolution kernel K0, that is, the deformable convolution kernel residual ΔK1. Then, the deformable convolution kernel residual ΔK1 is added to the upsampled deformable convolution kernel K0 to obtain the lowest scale deformable convolution kernel K1. By analogy, at the sth scale (s=1,2,..L), the deformable convolution kernel K s-1 Upsample and input to the sth UDBlock, based on the fusion feature F of the sth scale s , the input image frame I of the s-th scale s And the upsampled deformable convolution kernel K s-1 Get K s-1 The correction term, namely the deformable convolution kernel residual ΔK s In this way, operations are performed for each scale, so that the highest scale deformable convolution kernel K can be obtained. L .

[0093] In the related art, a self-encoder-decoder is used as the backbone network to extract features from two input frames, and then multiple prediction heads are used to predict deformable convolution. The self-encoder-decoder first downsamples the input features, and then uses transposed convolution to upsample. On the one hand, this process destroys the detailed information of the image, easily causing image feature loss and low parameter utilization; on the other hand, the backbone network still has difficulty in handling large motion situations, and ghosting problems will occur in the case of large motion. The essence of these problems is that the deformable convolution kernel is not accurately solved. According to the embodiments of the present disclosure, the correction of motion information at different scales is converted into corrections between deformable convolution kernels, which can accurately reflect the changes in motion information at different scales and predict accurate deformable convolution kernels.

[0094] According to an exemplary embodiment of the present disclosure, the following may be used: Figure 3 The UDBlock shown is used to predict the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale, and the operation of predicting the residual may include: converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting depth features of the second video frame of the second scale and the converted video frame; combining the extracted depth features and the fusion features of the second video frame of the second scale, and predicting the residual based on the combined features.

[0095] According to an exemplary embodiment of the present disclosure, converting the enlarged deformable convolution kernel of the first scale into a video frame includes: performing deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame. According to an exemplary embodiment of the present disclosure, extracting depth features of the second video frame of the second scale and the converted video frame includes: combining the converted video frame and the second video frame of the second scale; extracting depth features of the combined video frame layer by layer. According to the present invention, converting the deformable convolution kernel of the previous scale into image content and extracting depth features instead of directly convolving it to extract depth features can maintain the consistency of the feature space.

[0096] According to an exemplary embodiment of the present disclosure, predicting the residual based on the combined features may include: predicting the correction term of the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale based on the combined features, as well as the weights and convolution kernels of the pixels of the second video frame of the second scale, wherein obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale may include: correcting the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale. As described above, the combined features of each scale contain the motion information of the second video frame of that scale relative to the target video frame of that scale, and therefore, the correction term of the enlarged deformable convolution kernel of the previous scale may be predicted based on the combined features, thereby obtaining accurate motion information.

[0097] The following will refer to Figure 3 To illustrate the structure of a module UDBlock for modifying a deformable convolution kernel according to an exemplary embodiment of the present disclosure.

[0098] like Figure 3 As shown, for the UDBlock of the sth scale (i.e., Figure 3 sth-UDBlock in the first step, first use the deformable convolution kernel K of the previous scale s-1 is upsampled and magnified, and compared with the video frame I at the current scale s s Perform deformable convolution to convert the video frame. The converted video frame is the same as the video frame I of the current scale s. s The deep features are extracted layer by layer through M channel attention modules (CAB). The input and output of the CAB module are connected and input to the next CAB module to extract deep features. The final extracted deep features are fused with the features F at the current scale s. s After the cascade, it is sent to the prediction head DeH, so as to predict the deformable convolution kernel residual ΔK of the current scale s based on the cascade features s ΔK s and the deformable convolution kernel K of the previous scale after enlargement s-1 Add together to get the deformable convolution kernel K of the current scale s s Here, DeH ​​can be a three-part neural network, which can be based on the fusion feature F s ,I s The deep features obtain the coordinate offsets, weights, and convolution kernels of the pixels of the video frame.

[0099] As mentioned above, since multiple hierarchical scales are used to gradually enlarge the video frames, the correlation between video frames can be used to optimize the accuracy of deformable convolution kernels at each scale, so that stable and accurate processing effects can be achieved for video frames with large motions. The implemented structure is simple and performs well in high-resolution video processing.

[0100] Figure 4 4 is a diagram showing a video frame processing apparatus 400 according to an exemplary embodiment of the present disclosure, including a video frame acquisition unit 410 and a video frame processing unit 420. It should be understood that the video frame processing apparatus 400 can be implemented in a device having video frame processing capability in the form of software / hardware or a combination of software and hardware.

[0101] The video frame acquisition unit 410 is configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of each scale corresponding to each of the first video frames.

[0102] The video frame processing unit 420 is configured to obtain, for a first scale and a second scale of the multiple scales, motion information of a second video frame at the first scale relative to a target video frame at the first scale, and obtain a target video frame at the second scale based on the motion information at the first scale and the second video frame at the second scale, where the second scale is larger than the first scale.

[0103] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: when the second scale is a maximum scale of the multiple scales, insert the target video frame of the second scale into the first video frame sequence.

[0104] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: when the second scale is smaller than the maximum scale of the multiple scales, use the second scale as the new first scale and the third scale as the new second scale, and repeatedly perform the process of acquiring the motion information, and acquiring the target video frame of the second scale according to the motion information and the second video frame of the second scale, until the second scale is the maximum scale of the multiple scales.

[0105] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: obtain fusion features of the second video frame at the second scale; amplify the motion information of the first scale; obtain correction parameters for the amplified motion information of the first scale based on the second video frame at the second scale, the fusion features of the second video frame at the second scale, and the amplified motion information of the first scale; obtain motion information of the second video frame at the second scale relative to the target video frame at the second scale according to the correction parameters of the motion information at the first scale and the amplified motion information of the previous scale; obtain the target video frame of the second scale according to the motion information of the second scale and the second video frame at the second scale.

[0106] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: combine the second video frames of the second scale, and extract features of the combined second video frames of the second scale to obtain fused features of the second video frames of the second scale.

[0107] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: predict the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale. The video frame processing unit 410 is also configured to obtain the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale.

[0108] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: convert the enlarged deformable convolution kernel of the first scale into a video frame; extract depth features of the second video frame of the second scale and the converted video frame; combine the extracted depth features and the fusion features of the second video frame of the second scale, and predict the residual based on the combined features.

[0109] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to perform a deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0110] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: combine the converted video frame and the second video frame of the second scale; and extract depth features of the combined video frame layer by layer.

[0111] According to an exemplary embodiment of the present disclosure, the video frame processing unit 410 is configured to: predict a correction term of a pixel coordinate offset of the second video frame at the second scale relative to a target video frame at the second scale based on the combined features, and a weight and a convolution kernel of pixels of the second video frame at the second scale; and correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0112] The above has been referred to Figure 1-Figure 3 The operations of the video frame processing method according to the exemplary embodiment of the present disclosure have been described. It should be understood that these operations can also be implemented in the video frame processing device according to the exemplary embodiment of the present disclosure, and will not be repeated here.

[0113] Figure 5 is a schematic diagram showing the effect of a video frame processing method according to an exemplary embodiment of the present disclosure. The overlapping input in the upper left corner is a video frame that needs to be processed. Figure 5 It can be seen from the comparison that compared with other video processing algorithms, such as EDSC (Enhanced Deformable Separable Convolution) and FLAVR (Flow-Agnostic Video Representations), the video processing method (H-VFI) according to the exemplary embodiment of the present disclosure has clear boundaries in the case of large motion and obtains highly reliable prediction results. The EDSC algorithm obtains a result with ghosting in the case of large motion, while FLAVR directly blurs the result that is difficult to reconstruct.

[0114] Figure 6 is a schematic diagram showing the effect of a video frame processing method according to an exemplary embodiment of the present disclosure. The overlapping input in the upper left corner is a video frame that needs to be processed. Figure 6 It can be seen from the comparison that compared with other video processing algorithms, such as CAIN (Channel Attention Is All You Need for Video Frame Interpolation), XVFI (eXtreme Video Frame Interpolation), EDSC and FLAVR, the video processing method (H-VFI) according to the exemplary embodiment of the present disclosure presents a more stable and accurate reconstruction effect regardless of large motion or normal motion, in complex or simple occlusion situations, and in the movement of large objects or small objects.

[0115] Figure 7 is a flowchart illustrating a training method of a video processing model according to an exemplary embodiment of the present disclosure.

[0116] like Figure 7 As shown, in step S710, a second video frame sequence of multiple scales corresponding to the first video frame sequence is obtained, the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes second video frames of each scale corresponding to each first video frame. Here, the first video frame sequence can be obtained from the video frames in the video training set used for training, and the second video frame sequence of multiple scales with lower resolution can be obtained by downsampling the first video frame sequence. The training video has preset training video frames of each scale for comparison with the output results of the video processing model. In other words, the training video frames are video frames with the desired processing effect for comparison.

[0117] In step S720, for a first scale and a second scale of the multiple scales, motion information of a second video frame at the first scale relative to a target video frame at the first scale is obtained through a video frame processing model, and a target video frame at the second scale is obtained according to the motion information at the first scale and the second video frame at the second scale, where the second scale is larger than the first scale.

[0118] In step S730, a video processing model is trained based on a comparison between the acquired target image frame of the second scale and the training video frame of the second scale.

[0119] According to an exemplary embodiment of the present disclosure, a video processing model training method uses motion information of smaller-scale video frames and larger-scale video frames to obtain larger-scale target video frames. The motion correlation between the smaller-scale target video frames and the larger-scale target video frames obtained by the desired interpolation can be used to more accurately predict the motion between the larger-scale video frames and the target video frames. The network parameters can be adjusted by comparing the obtained target video frames with the training video frames to obtain a processing model that can accurately perform motion interpolation.

[0120] According to an exemplary embodiment of the present disclosure, the training method may include: when the second scale is smaller than the maximum scale of the multiple scales, the second scale is used as the new first scale, the third scale is used as the new second scale, and the process of obtaining the motion information, obtaining the target video frame of the second scale according to the motion information and the second video frame of the second scale, and training the video processing model according to the comparison of the target image frame of the second scale and the training video frame of the second scale is repeated until the second scale is the maximum scale of the multiple scales. That is, at each scale, the motion information of the previous scale of the scale can be used to obtain the motion information of the scale, and then the target video frame is obtained based on the motion information of the scale until the maximum scale (for example, the original resolution of the video) is reached. In this way, larger motions at the original resolution scale can be decomposed into smaller motions, thereby improving the accuracy of motion prediction, and at each scale, the network parameters can be adjusted by comparing the output target video frame with the training video frame, which can ensure that even with a deeper hierarchical structure, it is easy to train to convergence.

[0121] According to an exemplary embodiment of the present disclosure, according to the motion information of the first scale and the second video frame of the second scale, obtaining the target video frame of the second scale includes: obtaining the fusion feature of the second video frame of the second scale; amplifying the motion information of the first scale; obtaining the correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale and the amplified motion information of the first scale; obtaining the motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameter of the motion information of the first scale and the amplified motion information of the first scale; obtaining the target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale. That is, the amplified motion information of the previous scale may have an error relative to the desired motion information of the current scale, so the amplified motion information can be corrected by the video frame of the current scale and its fusion feature, so as to obtain accurate motion information of the current scale.

[0122] According to an exemplary embodiment of the present disclosure, obtaining the fused features of the second video frame at the second scale includes: combining the second video frames at the second scale, and extracting features of the combined second video frames at the second scale to obtain the fused features of the second video frame at the second scale.

[0123] According to an exemplary embodiment of the present disclosure, based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged motion information of the first scale, a correction parameter for the enlarged motion information of the first scale is obtained, including: based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixel of the second video frame of each scale, wherein, based on the correction parameter of the motion information of the first scale and the enlarged motion information of the first scale, the motion information of the second video frame of the second scale relative to the target video frame of the second scale is obtained, including: based on the residual and the enlarged deformable convolution kernel of the first scale, the deformable convolution kernel of the second scale is obtained. Converting the correction of motion information at different scales into correction between deformable convolution kernels can accurately reflect the changes of motion information at different scales.

[0124] According to an exemplary embodiment of the present disclosure, predicting the residual of the deformable convolution kernel of the first scale after enlargement relative to the deformable convolution kernel of the second scale includes: converting the deformable convolution kernel of the first scale after enlargement into a video frame; extracting the depth features of the second video frame of the second scale and the converted video frame; combining the extracted depth features and the fusion features of the second video frame of the second scale, and predicting the residual based on the combined features. Converting the deformable convolution kernel of the previous scale into image content and extracting the depth features instead of directly convolving it to extract the depth features can maintain the consistency of the feature space.

[0125] According to an exemplary embodiment of the present disclosure, converting the enlarged deformable convolution kernel of the first scale into a video frame includes: performing deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0126] According to an exemplary embodiment of the present disclosure, extracting depth features of the second video frame at the second scale and the converted video frame includes: combining the converted video frame and the second video frame at the second scale; and extracting depth features of the combined video frame layer by layer.

[0127] According to an exemplary embodiment of the present disclosure, predicting the residual based on the combined features includes: predicting the correction term of the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale based on the combined features, as well as the weights and convolution kernels of the pixels of the second video frame of the second scale, wherein obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale includes: correcting the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale. The combined features of each scale contain the motion information of the second video frame of that scale relative to the target video frame of that scale, and therefore, the correction term of the enlarged deformable convolution kernel of the previous scale can be predicted based on the combined features, thereby obtaining accurate motion information.

[0128] According to an exemplary embodiment of the present disclosure, training a video processing model based on a comparison between the acquired target image frame of the second scale and the training video frame of the second scale includes: adjusting parameters of the video processing model based on a loss function constructed based on the acquired target image frame of the second scale and the training image frame of the second scale.

[0129] The above has been referred to Figure 1-Figure 3 The process of acquiring the motion information and the target video frame in step S720 has been described, and will not be repeated here.

[0130] As described above, in the training phase, at each scale, the target video frame of the current scale is obtained based on the motion information of the current scale and the video frame of the current scale, and the operation of the video frame processing model at each scale is supervised based on the comparison between the output video frame of each scale and the training video frame of each scale. Here, the video frames used for training include the training video frames of each scale, which are pre-configured video frames used as comparison targets. At each scale, the loss cost of the output video frame can be calculated by an auxiliary loss function and the gradient can be returned, so that the network structure of the video frame processing model can be adjusted based on the loss cost. For example, the parameters of the feature extraction network can be adjusted, and the parameters of the CAB module set in the UDBlock can also be adjusted. In this way, it can be ensured that even with a deeper hierarchical structure, it is easy to train to convergence. In the test application phase, it is not necessary to reconstruct the video frames of the intermediate scales, but only output the final interpolated image I L,out , that is, no output is required Figure 1 As shown in I 1,out ,…I L-1,out , which can improve the efficiency of test application.

[0131] Figure 8 is a block diagram illustrating a video frame processing apparatus according to an exemplary embodiment of the present disclosure.

[0132] like Figure 8 As shown, the video frame processing apparatus 800 may include a video frame acquisition unit 810 , a video frame processing unit 820 and a training unit 830 .

[0133] The video frame acquisition unit 810 is configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of each scale corresponding to each of the first video frames.

[0134] The video frame processing unit 820 is configured to obtain, for a first scale and a second scale of the multiple scales, motion information of a second video frame of the first scale relative to a target video frame of the first scale through a video frame processing model, and obtain a target video frame of the second scale based on the motion information of the first scale and the second video frame of the second scale, where the second scale is larger than the first scale.

[0135] The training unit 830 is configured to train a video processing model according to a comparison between the acquired target image frame of the second scale and the training video frame of the second scale.

[0136] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: when the second scale is smaller than the maximum scale of the multiple scales, use the second scale as the new first scale and the third scale as the new second scale, and repeat the process of acquiring the motion information, and acquiring the target video frame of the second scale according to the motion information and the second video frame of the second scale; the training unit is configured to repeat the process of training the video processing model according to the comparison between the acquired target image frame of the second scale and the training video frame of the second scale until the second scale is the maximum scale of the multiple scales.

[0137] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: obtain fusion features of the second video frame at the second scale; amplify the motion information of the first scale; obtain correction parameters for the amplified motion information of the first scale based on the second video frame at the second scale, the fusion features of the second video frame at the second scale, and the amplified motion information of the first scale; obtain motion information of the second video frame at the second scale relative to the target video frame at the second scale according to the correction parameters of the motion information at the first scale and the amplified motion information of the first scale; obtain the target video frame of the second scale according to the motion information of the second scale and the second video frame at the second scale.

[0138] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: combine the second video frames of the second scale, and extract features of the combined second video frames of the second scale to obtain fused features of the second video frames of the second scale.

[0139] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: predict the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale; and obtain the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale.

[0140] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: convert the enlarged deformable convolution kernel of the first scale into a video frame; extract depth features of the second video frame of the second scale and the converted video frame; combine the extracted depth features and the fusion features of the second video frame of the second scale, and predict the residual based on the combined features.

[0141] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to perform a deformable convolution on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

[0142] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: combine the converted video frame and the second video frame of the second scale; and extract depth features of the combined video frame layer by layer.

[0143] According to an exemplary embodiment of the present disclosure, the video frame processing unit 820 is configured to: predict the correction term of the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale based on the combined features, and the weight and convolution kernel of the pixels of the second video frame of the second scale; and correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale. .

[0144] According to an exemplary embodiment of the present disclosure, the training unit 830 is configured to adjust parameters of the video processing model based on a loss function constructed according to the acquired target image frame of the second scale and the training image frame of the second scale.

[0145] Fig. 9 is a schematic diagram showing a video frame processing system according to an exemplary embodiment of the present disclosure. The video frame processing system can be implemented by a neural network.

[0146] like Fig. 9 As shown, according to an exemplary embodiment of the present disclosure, a video frame processing system 900 may include a video frame acquisition unit 910 and a video frame processing network 920. The video frame acquisition unit 910 may acquire a video frame sequence locally or externally, and the video frame processing network 920 may process the acquired video frame sequence.

[0147] According to an exemplary embodiment of the present disclosure, the video frame acquisition unit 910 is configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of each scale corresponding to each of the first video frames.

[0148] The video frame processing network 920 includes a plurality of video frame processing modules 920-1, 920-2, 920-n connected in sequence, each of the plurality of video frame processing modules corresponding to each of the plurality of scales, respectively, wherein each video frame processing module is configured to: obtain motion information of a second video frame of a first scale corresponding to a module preceding the video frame processing module relative to a target video frame of the first scale, and obtain a target video frame of the second scale based on the motion information of the first scale and the second video frame of the second scale corresponding to each video frame processing module.

[0149] According to an exemplary embodiment of the present disclosure, the video frame processing network 920 includes a feature extraction network 921, and the feature extraction network 921 is configured to obtain a fusion feature of the second video frame of each scale of the multiple scales. As described above, the fusion feature can be extracted by a feature extraction network such as an L-HRnet neural network.

[0150] According to an exemplary embodiment of the present disclosure, the feature extraction network 921 includes: a combination module 9211, configured to combine the second video frames at each scale; and a feature extraction module 9212, configured to extract features of the combined second video frames to obtain fused features of the second video frames at each scale.

[0151] According to an exemplary embodiment of the present disclosure, each scale video frame processing module 920-n includes: an amplification module, configured to amplify the motion information of the first scale; a motion information acquisition module, configured to acquire correction parameters for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the amplified motion information of the first scale, and acquire motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameters of the motion information of the first scale and the amplified motion information of the first scale; an output module, configured to acquire the target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale. As described above, the video frame can be used as follows: Figure 3 The UDBlock shown is used to implement the video frame processing module.

[0152] According to an exemplary embodiment of the present disclosure, the motion information acquisition module is configured to: predict the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, and acquire the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale.

[0153] According to an exemplary embodiment of the present disclosure, the motion information acquisition module includes: a deformable convolution module, configured to convert the enlarged deformable convolution kernel of the first scale into a video frame; a depth feature extraction module, configured to extract depth features of the second video frame of the second scale and the converted video frame; a residual prediction module, configured to combine the extracted depth features and the fusion features of the second video frame of the second scale, and predict the residual based on the combined features.

[0154] According to an exemplary embodiment of the present disclosure, the deep feature extraction module includes: a combination module (not shown), configured to perform combination on the converted video frame and the second video frame of the second scale; and multiple channel attention modules (not shown), configured to extract deep features of the combined image frame layer by layer.

[0155] According to an exemplary embodiment of the present disclosure, the residual prediction module includes: a combination module (not shown), configured to combine the extracted depth features and the fusion features of the second video frame of the second scale; a prediction neural network (not shown), configured to predict the correction term of the pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale and the weights and convolution kernel of the pixels of the second video frame of the second scale based on the combined features; a correction module (not shown), configured to correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

[0156] Fig.10 1 is a block diagram showing a structure of an electronic device 1000 for video processing for video frame insertion according to an exemplary embodiment of the present disclosure. The electronic device 1000 may be, for example, a smart phone, a tablet computer, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The electronic device 1000 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0157] Typically, the electronic device 1000 includes a processor 1001 and a memory 1002 .

[0158] The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor, the main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In an exemplary embodiment of the present disclosure, the processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0159] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction, which is used to be executed by the processor 1001 to implement the video processing method of the exemplary embodiment of the present disclosure.

[0160] In some embodiments, the electronic device 1000 may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1003 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1004, a touch display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008 and a power supply 1009.

[0161] The peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0162] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1004 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0163] The display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on the surface or above the surface of the display screen 1005. The touch signal can be input to the processor 1001 as a control signal for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, arranged on the front panel of the electronic device 1000; in other embodiments, the display screen 1005 can be at least two, respectively arranged on different surfaces of the terminal 1000 or in a folding design; in some further embodiments, the display screen 1005 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 1000. Even, the display screen 1005 can also be arranged as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0164] The camera assembly 1006 is used to capture images or videos. Optionally, the camera assembly 1006 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0165] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1001 for processing, or input them into the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1000. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0166] The positioning component 1008 is used to locate the current geographic location of the electronic device 1000 to implement navigation or LBS (Location Based Service). The positioning component 1008 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system, or the European Union's Galileo system.

[0167] The power supply 1009 is used to power various components in the electronic device 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0168] In some embodiments, the electronic device 1000 further includes one or more sensors 1010 , including but not limited to: an acceleration sensor 1011 , a gyroscope sensor 1012 , a pressure sensor 1013 , a fingerprint sensor 1014 , an optical sensor 1015 , and a proximity sensor 1016 .

[0169] The acceleration sensor 311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1000. For example, the acceleration sensor 1011 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1001 can control the touch display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used for collecting game or user motion data.

[0170] The gyro sensor 1012 can detect the body direction and rotation angle of the terminal 1000, and the gyro sensor 1012 can cooperate with the acceleration sensor 1011 to collect the user's 3D actions on the terminal 1000. The processor 1001 can implement the following functions based on the data collected by the gyro sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0171] The pressure sensor 1013 can be set on the side frame of the terminal 1000 and / or the lower layer of the touch display screen 1005. When the pressure sensor 1013 is set on the side frame of the terminal 1000, the user's holding signal of the terminal 1000 can be detected, and the processor 1001 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is set on the lower layer of the touch display screen 1005, the processor 1001 controls the operability controls on the UI according to the user's pressure operation on the touch display screen 1005. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0172] The fingerprint sensor 1014 is used to collect the user's fingerprint, and the processor 1001 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1001 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1014 can be set on the front, back, or side of the electronic device 1000. When a physical button or a manufacturer logo is set on the electronic device 1000, the fingerprint sensor 1014 can be integrated with the physical button or the manufacturer logo.

[0173] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the touch display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1005 is reduced. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015.

[0174] The proximity sensor 1016, also called a distance sensor, is usually arranged on the front panel of the electronic device 1000. The proximity sensor 1016 is used to collect the distance between the user and the front of the electronic device 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the touch display screen 1005 to switch from the screen-on state to the screen-off state; when the proximity sensor 1016 detects that the distance between the user and the front of the electronic device 1000 is gradually increasing, the processor 1001 controls the touch display screen 1005 to switch from the screen-off state to the screen-on state.

[0175] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the electronic device 1000, and the electronic device 1000 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0176] Fig.11 FIG. 1 is a block diagram of another electronic device 1100. For example, the electronic device 1100 may be provided as a server. Fig.11 , the electronic device 1100 includes one or more processing processors 1110 and a memory 1120. The memory 1120 may include one or more programs for executing the above video processing method. The electronic device 1100 may also include a power supply component 1130 configured to perform power management of the electronic device 1100, a wired or wireless network interface 1140 configured to connect the electronic device 1100 to the network, and an input / output (I / O) interface 1150. The electronic device 1100 may operate based on an operating system stored in the memory 1120, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0177] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the video processing method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0178] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the above-mentioned video processing method.

[0179] The video processing method disclosed in the present invention adopts multiple hierarchical scales to gradually enlarge the image frames, and can utilize the correlation between image frames to optimize the accuracy of deformable convolution kernels at various scales, thereby achieving stable and accurate processing effects for image frames with large motions, and the implemented structure is simple, and has good performance in high-resolution video processing.

[0180] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the schemes disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0181] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: include: Acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of the scale corresponding to each of the first video frames; For a first scale and a second scale of the multiple scales, motion information of a second video frame of the first scale relative to a target video frame of the first scale is obtained, and a target video frame of the second scale is obtained according to the motion information of the first scale and the second video frame of the second scale, wherein the second scale is larger than the first scale, When the second scale is smaller than the maximum scale of the multiple scales, the second scale is used as the new first scale, the third scale is used as the new second scale, and the process of obtaining the motion information and obtaining the target video frame of the second scale according to the motion information and the second video frame of the second scale is repeated until the second scale is the maximum scale of the multiple scales.

2. The method according to claim 1, characterized in that include: When the second scale is the maximum scale of the multiple scales, the target video frame of the second scale is inserted into the first video frame sequence.

3. The method according to claim 1, characterized in that Acquiring a target video frame at the second scale according to the motion information at the first scale and the second video frame at the second scale includes: Obtaining a fusion feature of a second video frame at the second scale; amplifying the motion information of the first scale; acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale; Acquire motion information of a second video frame at the second scale relative to a target video frame at the second scale according to a correction parameter of the motion information at the first scale and the amplified motion information at the first scale; A target video frame at the second scale is acquired according to the motion information at the second scale and the second video frame at the second scale.

4. The method according to claim 3, characterized in that Acquiring a fusion feature of the second video frame at the second scale includes: The second video frames at the second scale are combined, and features of the combined second video frames at the second scale are extracted to obtain fusion features of the second video frames at the second scale.

5. The method according to claim 3, characterized in that Acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale, including: Based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale, Acquiring motion information of a second video frame at the second scale relative to a target video frame at the second scale according to the correction parameter of the motion information at the first scale and the amplified motion information at the first scale includes: A deformable convolution kernel of the second scale is obtained based on the residual and the enlarged deformable convolution kernel of the first scale.

6. The method according to claim 5, characterized in that Predicting a residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale, comprising: Converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting deep features of the second video frame and the transformed video frame at the second scale; The extracted depth feature and the fused feature of the second video frame at the second scale are combined, and the residual is predicted based on the combined feature.

7. The method according to claim 6, characterized in that Converting the enlarged deformable convolution kernel of the first scale into a video frame includes: A deformable convolution is performed on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

8. The method according to claim 6, characterized in that Extracting deep features of the second video frame and the transformed video frame at the second scale includes: combining the converted video frame and the second video frame of the second scale; Deep features of the combined video frames are extracted layer by layer.

9. The method according to claim 6, characterized in that in, Predicting the residual based on the combined features includes: Predicting a correction term for pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale and a weight and a convolution kernel of pixels of the second video frame of the second scale based on the combined features, The step of obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale includes: The pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale is corrected based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

10. A video frame processing device, comprising: a video frame acquisition unit configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of the scale corresponding to each of the first video frames; a video frame processing unit configured to obtain, for a first scale and a second scale of the multiple scales, motion information of a second video frame of the first scale relative to a target video frame of the first scale, and obtain a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale, wherein the second scale is larger than the first scale; Wherein, the video frame processing unit is configured as follows: When the second scale is smaller than a maximum scale of the multiple scales, the second scale is used as a new first scale, the third scale is used as a new second scale, and the process of acquiring the motion information, and acquiring the target video frame of the second scale according to the motion information and the second video frame of the second scale is repeated until the second scale is the maximum scale of the multiple scales.

11. The device according to claim 10, characterized in that The video frame processing unit is configured to: when the second scale is a maximum scale of the multiple scales, insert the target video frame of the second scale into the first video frame sequence.

12. The device according to claim 10, characterized in that The video frame processing unit is configured as follows: Obtaining a fusion feature of a second video frame at the second scale; amplifying the motion information of the first scale; acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale; Acquire motion information of a second video frame at the second scale relative to a target video frame at the second scale according to a correction parameter of the motion information at the first scale and the amplified motion information at the first scale; A target video frame at the second scale is acquired according to the motion information at the second scale and the second video frame at the second scale.

13. The device according to claim 12, characterized in that The video frame processing unit is configured as follows: The second video frames at the second scale are combined, and features of the combined second video frames at the second scale are extracted to obtain fusion features of the second video frames at the second scale.

14. The device according to claim 12, characterized in that The video frame processing unit is configured as follows: Based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale; A deformable convolution kernel of the second scale is obtained based on the residual and the enlarged deformable convolution kernel of the first scale.

15. The device according to claim 14, characterized in that The video frame processing unit is configured as follows: Converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting deep features of the second video frame and the transformed video frame at the second scale; The extracted depth feature and the fused feature of the second video frame at the second scale are combined, and the residual is predicted based on the combined feature.

16. The device according to claim 15, characterized in that The video frame processing unit is configured as follows: A deformable convolution is performed on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

17. The device according to claim 15, characterized in that The video frame processing unit is configured as follows: combining the converted video frame and the second video frame of the second scale; Deep features of the combined video frames are extracted layer by layer.

18. The device according to claim 15, characterized in that The video frame processing unit is configured as follows: Predicting, based on the combined features, a correction term for pixel coordinate offset of the second video frame at the second scale relative to the target video frame at the second scale, and weights and convolution kernels of pixels of the second video frame at the second scale; The pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale is corrected based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

19. A video processing model training method, characterized in that: include: Acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of the scale corresponding to each of the first video frames; For a first scale and a second scale of the multiple scales, obtaining motion information of a second video frame of the first scale relative to a target video frame of the first scale through a video frame processing model, and obtaining a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale, wherein the second scale is larger than the first scale; Training a video processing model according to a comparison between the acquired target image frame of the second scale and the training video frame of the second scale, When the second scale is smaller than the maximum scale of the multiple scales, the second scale is used as the new first scale, the third scale is used as the new second scale, and the process of obtaining the motion information, obtaining the target video frame of the second scale according to the motion information and the second video frame of the second scale, and training the video processing model according to the comparison between the target image frame of the second scale and the training video frame of the second scale is repeated until the second scale is the maximum scale of the multiple scales.

20. The training method according to claim 19, characterized in that: Acquiring a target video frame at the second scale according to the motion information at the first scale and the second video frame at the second scale includes: Obtaining a fusion feature of a second video frame at the second scale; amplifying the motion information of the first scale; acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale; Acquire motion information of a second video frame at the second scale relative to a target video frame at the second scale according to a correction parameter of the motion information at the first scale and the amplified motion information at the first scale; A target video frame at the second scale is acquired according to the motion information at the second scale and the second video frame at the second scale.

21. The training method according to claim 20, characterized in that: Acquiring a fusion feature of the second video frame at the second scale includes: The second video frames at the second scale are combined, and features of the combined second video frames at the second scale are extracted to obtain fused features of the second video frames at the second scale.

22. The training method according to claim 20, characterized in that: Acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale, including: Based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale, The method of acquiring the motion information of the second video frame at the second scale relative to the target video frame at the second scale according to the correction parameter of the motion information at the first scale and the amplified motion information at the first scale includes: A deformable convolution kernel of the second scale is obtained based on the residual and the enlarged deformable convolution kernel of the first scale.

23. The training method according to claim 22, characterized in that: Predicting a residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale, comprising: Converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting deep features of the second video frame and the transformed video frame at the second scale; The extracted depth feature and the fused feature of the second video frame at the second scale are combined, and the residual is predicted based on the combined feature.

24. The training method according to claim 23, characterized in that: Converting the enlarged deformable convolution kernel of the first scale into a video frame includes: A deformable convolution is performed on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

25. The training method according to claim 23, characterized in that: Extracting deep features of the second video frame and the transformed video frame at the second scale includes: combining the converted video frame and the second video frame of the second scale; Deep features of the combined video frames are extracted layer by layer.

26. The training method according to claim 23, characterized in that: Predicting the residual based on the combined features includes: Predicting a correction term for pixel coordinate offset of the second video frame of the second scale relative to the target video frame of the second scale and a weight and a convolution kernel of pixels of the second video frame of the second scale based on the combined features, The step of obtaining the deformable convolution kernel of the second scale based on the residual and the enlarged deformable convolution kernel of the first scale includes: The pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale is corrected based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

27. The training method according to claim 19, characterized in that: Training the video processing model according to the comparison between the acquired target image frame of the second scale and the training video frame of the second scale includes: The parameters of the video processing model are adjusted based on a loss function constructed according to the acquired target image frame of the second scale and the training image frame of the second scale.

28. A video processing model training device, characterized in that: include: a video frame acquisition unit configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of the scale corresponding to each of the first video frames; a video frame processing unit configured to obtain, for a first scale and a second scale of the multiple scales, motion information of a second video frame of the first scale relative to a target video frame of the first scale through a video frame processing model, and obtain a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale, wherein the second scale is larger than the first scale; a training unit configured to train a video processing model according to a comparison between the acquired target image frame of the second scale and the training video frame of the second scale, Wherein, the video frame processing unit is configured as follows: When the second scale is smaller than the maximum scale of the multiple scales, the second scale is used as the new first scale and the third scale is used as the new second scale, and the process of acquiring the motion information and acquiring the target video frame of the second scale according to the motion information and the second video frame of the second scale is repeated. The training unit is configured to repeatedly train the video processing model according to the comparison between the acquired target image frame of the second scale and the training video frame of the second scale until the second scale is the maximum scale of the multiple scales.

29. The training device according to claim 28, characterized in that The video frame processing unit is configured as follows: Obtaining a fusion feature of a second video frame at the second scale; amplifying the motion information of the first scale; acquiring a correction parameter for the amplified motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the amplified motion information of the first scale; Acquire motion information of a second video frame at the second scale relative to a target video frame at the second scale according to a correction parameter of the motion information at the first scale and the amplified motion information at the first scale; A target video frame at the second scale is acquired according to the motion information at the second scale and the second video frame at the second scale.

30. The training device according to claim 29, characterized in that The video frame processing unit is configured as follows: The second video frames at the second scale are combined, and features of the combined second video frames at the second scale are extracted to obtain fused features of the second video frames at the second scale.

31. The training device according to claim 29, characterized in that The video frame processing unit is configured as follows: Based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, predicting the residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale as the correction parameter, wherein the deformable convolution kernel of each scale includes the pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and the weight and convolution kernel of the pixels of the second video frame of each scale; A deformable convolution kernel of the second scale is obtained based on the residual and the enlarged deformable convolution kernel of the first scale.

32. The training device according to claim 31, characterized in that The video frame processing unit is configured as follows: Converting the enlarged deformable convolution kernel of the first scale into a video frame; extracting deep features of the second video frame and the transformed video frame at the second scale; The extracted depth feature and the fused feature of the second video frame at the second scale are combined, and the residual is predicted based on the combined feature.

33. The training device according to claim 32, characterized in that The video frame processing unit is configured as follows: A deformable convolution is performed on the enlarged deformable convolution kernel of the first scale and the second video frame of the second scale to obtain a converted video frame.

34. The training device according to claim 32, characterized in that The video frame processing unit is configured as follows: combining the converted video frame and the second video frame of the second scale; Deep features of the combined video frames are extracted layer by layer.

35. The training device according to claim 32, characterized in that in, The video frame processing unit is configured as follows: Predicting, based on the combined features, a correction term for pixel coordinate offset of the second video frame at the second scale relative to the target video frame at the second scale, and weights and convolution kernels of pixels of the second video frame at the second scale; The pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale is corrected based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

36. The training device according to claim 28, characterized in that The training unit is configured as: The parameters of the video processing model are adjusted based on a loss function constructed according to the acquired target image frame of the second scale and the training image frame of the second scale.

37. A video processing system, characterized in that: include: a video frame acquisition unit configured to acquire a second video frame sequence of multiple scales corresponding to a first video frame sequence, wherein the first video frame sequence includes multiple first video frames, and for each scale of the multiple scales, the second video frame sequence of each scale includes a second video frame of the scale corresponding to each of the first video frames; A video frame processing network comprises a plurality of video frame processing modules connected in sequence, wherein each of the plurality of video frame processing modules corresponds to each of the plurality of scales, Wherein, each video frame processing module is configured as follows: Obtain motion information of a second video frame of a first scale corresponding to a module before the video frame processing module relative to a target video frame of the first scale, and obtain a target video frame of the second scale according to the motion information of the first scale and the second video frame of the second scale corresponding to each video frame processing module.

38. The system of claim 37, wherein: The video frame processing network includes a feature extraction network configured to obtain fused features of the second video frame at each scale of the multiple scales.

39. The system of claim 38, wherein: The feature extraction network includes: A combination module is configured to combine the second video frames at each scale; The feature extraction module is configured to extract features of the combined second video frames to obtain fused features of the second video frames at each scale.

40. The system of claim 38, wherein: The video frame processing module for each scale includes: an amplification module, configured to amplify the motion information of the first scale; a motion information acquisition module, configured to acquire a correction parameter for the enlarged motion information of the first scale based on the second video frame of the second scale, the fusion feature of the second video frame of the second scale, and the enlarged motion information of the first scale, and acquire motion information of the second video frame of the second scale relative to the target video frame of the second scale according to the correction parameter of the motion information of the first scale and the enlarged motion information of the first scale; The output module is configured to obtain a target video frame of the second scale according to the motion information of the second scale and the second video frame of the second scale.

41. The system of claim 40, wherein: The motion information acquisition module is configured as follows: Based on the second video frame of the second scale, the fusion features of the second video frame of the second scale and the enlarged deformable convolution kernel of the first scale, a residual of the enlarged deformable convolution kernel of the first scale relative to the deformable convolution kernel of the second scale is predicted as the correction parameter, and the deformable convolution kernel of the second scale is obtained based on the residual and the enlarged deformable convolution kernel of the first scale, wherein the deformable convolution kernel of each scale includes a pixel coordinate offset of the second video frame of each scale relative to the target video frame of each scale and a weight and a convolution kernel of the pixels of the second video frame of each scale.

42. The system of claim 40, wherein: The motion information acquisition module includes: a deformable convolution module, configured to convert the enlarged deformable convolution kernel of the first scale into a video frame; a depth feature extraction module configured to extract depth features of the second video frame and the converted video frame at the second scale; The residual prediction module is configured to combine the extracted depth feature and the fused feature of the second video frame at the second scale, and predict the residual based on the combined feature.

43. The system of claim 42, wherein: The deep feature extraction module includes: a combining module configured to combine the converted video frame and the second video frame of the second scale; Multiple channel attention modules are configured to extract deep features of the combined image frames layer by layer.

44. The system of claim 42, wherein: The residual prediction module comprises: a combining module configured to combine the extracted depth features and the fused features of the second video frame at the second scale; A prediction neural network configured to predict, based on the combined features, a correction term for pixel coordinate offset of the second video frame at the second scale relative to the target video frame at the second scale, and weights and convolution kernels of pixels of the second video frame at the second scale; The correction module is configured to correct the pixel coordinate offset included in the enlarged deformable convolution kernel of the first scale based on the correction term to obtain the pixel coordinate offset included in the deformable convolution kernel of the second scale.

45. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method of any one of claims 1 to 9 or claims 19 to 27.

46. ​​A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by at least one processor, enables the at least one processor to perform the method of any one of claims 1 to 9 or claims 19 to 27.

Images