Video frame synthesis method, device, equipment and storage medium

Abstract

The present invention discloses a video frame synthesis method, apparatus, device and storage medium. The method comprises inputting a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales; performing feature fusion on the semantic features to obtain fused semantic features; and determining the video frame synthesis result based on the fused semantic features. Since the present invention obtains semantic features of a video frame sequence at different spatiotemporal scales, performs feature fusion on the semantic features to obtain fused semantic features, and determines the video frame synthesis result based on the fused semantic features. Compared with the existing method of synthesizing video frames by relying on densely estimating the motion between given video frames, the above method of the present invention can synthesize higher quality synthetic video frames.

Description

Technical Field

[0001] The present invention relates to the technical field of video frame synthesis, and in particular to a video frame synthesis method, device, equipment and storage medium. Background Art

[0002] The basic idea behind video frame synthesis is to estimate the motion relationship between given video frames and then interpolate and extrapolate frames based on existing video frames. Interpolation refers to synthesizing one or more frames between two given frames or two video clips, so that the two given frames or clips can achieve a visually smooth and reasonable transition. Extrapolation involves synthesizing corresponding video frames or clips at the beginning or end of a given video based on existing video content, so that the synthesized clips can visually serve as the content before or after the beginning or end of the existing video.

[0003] Traditional video frame synthesis methods rely primarily on densely estimating the motion between given video frames, such as dense optical flow. Based on the estimated motion relationships, the input video frames are then deformed and transformed accordingly, ultimately achieving interpolation and extrapolation of video frames. The performance of such optical flow-guided video frame synthesis methods depends heavily on the quality of the optical flow estimation, and the edges of moving objects in the resulting synthesized video frames often exhibit noticeable artifacts, severely impacting the visual quality of the synthesized frames. Summary of the Invention

[0004] The main purpose of the present invention is to provide a video frame synthesis method, device, equipment and storage medium, aiming to solve the technical problem of low visual quality of synthesized video frames in the prior art.

[0005] To achieve the above object, the present invention provides a video frame synthesis method, which includes the following steps:

[0006] Inputting the video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales;

[0007] Performing feature fusion on the semantic features to obtain fused semantic features;

[0008] The video frame synthesis result is determined according to the fused semantic features.

[0009] Optionally, the preset hybrid spatiotemporal convolutional network includes: a preset 3D convolutional layer and a preset time domain pooling layer;

[0010] The step of inputting the video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales includes:

[0011] Inputting the video frame sequence into the preset 3D convolution layer for convolution operation to obtain a video frame sequence after 3D convolution;

[0012] Performing a time-domain pooling operation on the video frame sequence after the 3D convolution through the preset time-domain pooling layer to obtain a video frame sequence with a time dimension of a preset dimension;

[0013] A convolution operation is performed on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0014] Optionally, the preset hybrid spatiotemporal convolutional network further includes: a preset 2D convolutional layer;

[0015] The performing a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales includes:

[0016] The preset 2D convolution layer is used to perform a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0017] Optionally, the step of determining a video frame synthesis result according to the fused semantic features includes:

[0018] Performing adaptive convolution on the fused semantic features to obtain an adaptive convolution result;

[0019] Performing motion blur correction on the fused semantic features to obtain a motion blur correction result;

[0020] The adaptive convolution result and the motion blur correction result are superimposed to obtain a video frame synthesis result.

[0021] Optionally, the step of performing motion blur correction on the fused semantic features to obtain a motion blur correction result includes:

[0022] determining a motion degree of each pixel in the video frame sequence based on the fused semantic features;

[0023] Determining pixels that are motion blurred according to the degree of motion, and determining a motion mask for the pixels that are motion blurred;

[0024] determining a correction bias for correcting motion blur based on the fused semantic features;

[0025] The motion blur correction result is determined according to the motion mask and the correction bias.

[0026] Optionally, the step of performing adaptive convolution on the fused semantic features to obtain an adaptive convolution result includes:

[0027] Determining a vertical motion convolution kernel and a horizontal motion convolution kernel corresponding to the video frame sequence according to the fused semantic features;

[0028] A 2D convolution kernel is simulated according to the vertical motion convolution kernel and the horizontal motion convolution kernel, and an adaptive convolution result is determined by the 2D convolution kernel.

[0029] Optionally, the step of performing feature fusion on the semantic features to obtain fused semantic features includes:

[0030] Scaling the semantic features to the target spatial resolution by using a preset multi-scale spatiotemporal feature fusion model;

[0031] Merge each scaled semantic feature in the channel dimension to obtain the semantic feature to be fused;

[0032] The semantic features to be fused are fused to obtain fused semantic features.

[0033] In addition, to achieve the above-mentioned object, the present invention further provides a video frame synthesis device, the device comprising:

[0034] A semantic feature determination module, configured to input a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales;

[0035] A fusion module, configured to perform feature fusion on the semantic features to obtain fused semantic features;

[0036] A synthesis module is used to determine a video frame synthesis result based on the fused semantic features.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a video frame synthesis device, which includes: a memory, a processor, and a video frame synthesis program stored on the memory and runnable on the processor, and the video frame synthesis program is configured to implement the steps of the video frame synthesis method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a video frame synthesis program is stored. When the video frame synthesis program is executed by a processor, the steps of the video frame synthesis method described above are implemented.

[0039] The present invention includes inputting a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales; performing feature fusion on the semantic features to obtain fused semantic features; and determining the video frame synthesis result based on the fused semantic features. Since the present invention obtains semantic features of the video frame sequence at different spatiotemporal scales, performs feature fusion on the semantic features to obtain fused semantic features, and determines the video frame synthesis result based on the fused semantic features. Compared with the existing method of synthesizing video frames by densely estimating the motion between given video frames, the above method of the present invention does not rely on optical flow to guide the synthesis of video frames, but determines the video frame synthesis result by performing feature fusion on the semantic features of the video frames at different spatiotemporal scales, which can avoid obvious artifacts in the synthesized video and improve the quality of the synthesized video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the structure of a video frame synthesis device in a hardware operating environment according to an embodiment of the present invention;

[0041] Figure 2 1 is a flow chart of a first embodiment of a method for synthesizing video frames according to the present invention;

[0042] Figure 3 2 is a flow chart of a second embodiment of a video frame synthesis method according to the present invention;

[0043] Figure 4 1 is a flow chart of a third embodiment of a video frame synthesis method according to the present invention;

[0044] Figure 5 This is a structural block diagram of the first embodiment of the video frame synthesis device of the present invention.

[0045] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a video frame synthesis device in the hardware operating environment involved in an embodiment of the present invention.

[0048] like Figure 1As shown, the video frame synthesis device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0049] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation to the video frame synthesis device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0050] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a video frame synthesis program.

[0051] exist Figure 1 In the video frame synthesis device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the video frame synthesis device of the present invention can be set in the video frame synthesis device, and the video frame synthesis device calls the video frame synthesis program stored in the memory 1005 through the processor 1001, and executes the video frame synthesis method provided by the embodiment of the present invention.

[0052] Based on the above video frame synthesis device, the embodiment of the present invention provides a video frame synthesis method, referring to Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a video frame synthesis method according to the present invention.

[0053] In this embodiment, the video frame synthesis method includes the following steps:

[0054] Step S10: Inputting the video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, personal computer, etc., or an electronic device with the same or similar functions, or a frame synthesis network (FSN) network architecture. The following uses the FSN as an example to illustrate this embodiment and the following embodiments.

[0056] It should be noted that the video frame sequence may be a sequence or video clip consisting of a specified number of video frames that need to be synthesized. The preset hybrid spatio-temporal convolutional network may be a 3D+2D hybrid spatio-temporal convolutional network (Hybrid Spatio-Temporal Convolutional Neural Network, HSTCNN) with a time domain pooling structure. In this embodiment, the preset hybrid spatio-temporal convolutional network includes: a preset 3D convolution layer, a preset time domain pooling layer, and a preset 2D convolution layer. The semantic features may include high-level features, mid-level features, and shallow features. The high-level features may include features related to semantic correctness, the mid-level features may include features related to object coherence, and the shallow features may include features related to edge sharpness.

[0057] Step S20: performing feature fusion on the semantic features to obtain fused semantic features.

[0058] It should be noted that the feature fusion of the semantic features can be performed by scaling the three levels of features in the semantic features to a uniform spatial resolution through a multi-scale spatio-temporal aggration (MSTA).

[0059] Furthermore, in order to improve the visual quality of the synthesized frame, step S20 may include: scaling the semantic features to the target spatial resolution through a preset multi-scale spatiotemporal feature fusion model; merging each scaled semantic feature in the channel dimension to obtain the semantic features to be fused; and fusing the semantic features to be fused to obtain the fused semantic features.

[0060] It should be noted that the target spatial resolution may be the spatial resolution of shallow features. The merging of the channel dimensions of each scaled semantic feature may be the merging of the features after resolution adjustment in the channel dimension after unifying the spatial resolutions of high-level features and mid-level features to the spatial resolution of shallow features. The merging of the features after resolution adjustment in the channel dimension may be the merging of the features after resolution adjustment into the same dimension. The semantic features to be fused may be the semantic features after merging channels. The fusing of the semantic features to be fused may be the fusing of the above three features using a set of convolutions to obtain the fused semantic features.

[0061] Step S30: determining a video frame synthesis result according to the fused semantic features.

[0062] In a specific implementation, after obtaining the fused semantic features, the fused semantic features can be adaptively convolutionally operated on through a convolutional neural network to obtain the video frame synthesis result.

[0063] Furthermore, to improve the quality of the synthesized video frame, considering that some areas of the adaptively convolved image frame may be blurred, this embodiment also performs motion blur correction on the fused semantic features. The adaptively convolved image frame and the motion blur corrected image frame are then fused to obtain a synthesized video frame. Specifically, the fused semantic features can be adaptively convolved and motion blur corrected separately, and the adaptive convolution and motion blur correction results are superimposed to form the synthesized video frame.

[0064] The adaptive convolution and motion blur correction for the fused semantic features may be performed by using an anti-motion blur adaptive convolution model (Motion-blur Agonistic AdaptiveConvolution, MA-AdaConv) to perform adaptive convolution and motion blur correction for the fused semantic features.

[0065] The MA-AdaConv described in this embodiment is composed of an adaptive convolution module (Adaptive Convolution Stream, ACS) and a motion blur correction module (Motion-blur Correction Stream, MCS). The ACS is used to estimate the convolution kernel of the adaptive convolution to achieve the goal of generating each pixel in the synthesized frame through adaptive convolution operations on the input frame; the MCS detects and marks the locations of intense motion in the video, and then generates correction offsets for the pixels at the corresponding locations. Finally, the correction offset output by the MCS is superimposed on the synthesized video frame obtained by the ACS corresponding to the correction offset in an additive manner, and finally a high-quality video frame synthesis result is obtained.

[0066] It should be noted that the adaptive convolution of the fused semantic features can be performed by forming a predictor through several convolution layer kernel upsampling layers contained in the adaptive convolution module, which is used to perform dense convolution kernel prediction on each image block. The video frame pixels are finally synthesized by the adaptive convolution module to obtain the adaptive convolution result. The adaptive convolution module can be an adaptive convolution stream (ACS). The motion blur correction of the fused semantic features can be performed by using a preset motion blur correction model to solve the problem of blurring of the corresponding area in the synthesized frame due to violent motion. The preset motion blur correction model can be a motion blur correction stream (MCS).

[0067] It should be understood that, based on the high-quality video features, ACS is already able to provide good results for synthesized video frames. However, for some pixels with more intense motion, synthesis using only ACS may still result in blurred synthesis results, because these pixels are difficult to obtain by performing adaptive convolution operations on pixels near the corresponding positions in the input video frame that can be covered by the adaptive convolution kernel corresponding to the current position. Although the academic community has tried to propose perceptual loss and gradient loss to help optimize adaptive convolution and thus alleviate the above-mentioned blurring problem, the overall effect is not good. In order to solve the above-mentioned problem of blurring of corresponding areas in adaptive convolution synthesized frames due to intense motion, this embodiment innovatively designs a motion blur correction model to solve the problem, that is, the problem of blurring of corresponding areas in synthesized frames due to intense motion is solved by presetting a motion blur correction model.

[0068] In practice, this embodiment proposes a novel Frame Synthesis Network (FSN) network architecture. In summary, the FSN network architecture consists of three core components: a 3D+2D hybrid spatiotemporal convolutional network (HSTCNN) with a temporal pooling structure, a multi-scale spatiotemporal feature fusion structure (MST), and a motion-blur-resistant adaptive convolutional structure (MA-AdaConv). For an input video frame sequence of T frames with a spatial size of H×W, where H and W represent the height and width of the video frame, respectively, the FSN outputs a synthesized frame obtained by interpolation or extrapolation. The input video frame sequence is first fed into the HSTCNN, completing the network's forward propagation process with minimal computational overhead. The MSTA structure then extracts three sets of low-level, mid-level, and high-level video feature descriptions from the HSTCNN at different spatiotemporal scales, namely the high-level features, mid-level features, and shallow-level features mentioned above. Through feature fusion, the video features subsequently used by MA-AdaConv are guaranteed to possess both precise spatial localization capabilities and rich semantic expression capabilities. MA-AdaConv first calculates the convolution kernel for adaptive convolution based on the video features it receives through the adaptive convolution stream (ACS). Then, the motion blur correction stream (MCS) determines the intensity of motion in different spatial regions of the video based on the input features and marks the areas with intense motion using a motion mask. At the same time, the MCS estimates the correction bias map used to correct the motion blur in the adaptive convolution output based on the video features. Finally, the synthesized frame result after motion blur correction is output by the model. The structural design, parameter setting, and working mechanism of each module will be described in detail in subsequent embodiments. During the training phase, this embodiment can perform end-to-end training optimization of the entire FSN by minimizing the pixel-level difference between the synthesized frame output by the model and the real frame used as the label. During the inference process, the FSN can be directly used to achieve end-to-end video frame synthesis.

[0069] This embodiment includes inputting a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales; performing feature fusion on the semantic features to obtain fused semantic features; and determining a video frame synthesis result based on the fused semantic features. Since this embodiment obtains semantic features of a video frame sequence at different spatiotemporal scales, performs feature fusion on the semantic features to obtain fused semantic features, and determines a video frame synthesis result based on the fused semantic features. Compared with the existing method of synthesizing video frames by densely estimating the motion between given video frames, the above method of this embodiment does not rely on optical flow to guide the synthesis of video frames, but determines the video frame synthesis result by performing feature fusion on the semantic features of video frames at different spatiotemporal scales, which can avoid obvious artifacts in the synthesized video and improve the quality of the synthesized video frames.

[0070] refer to Figure 3 , Figure 3 FIG. 4 is a flow chart of a second embodiment of a video frame synthesis method according to the present invention.

[0071] Based on the above first embodiment, in this embodiment, step S10 includes:

[0072] Step S101: inputting the video frame sequence into the preset 3D convolution layer for convolution operation to obtain the video frame sequence after 3D convolution.

[0073] It should be noted that the preset 3D convolution layer includes at least one 3D convolution layer and a preset time domain pooling layer. When there are two 3D convolution layers, the first 3D convolution layer uses a convolution operation with a stride of 2, which will downsample the feature map input to this stage, that is, downsample the input video frame sequence. The stride of the second 3D convolution layer can be set to 1 to keep the resolution of the feature map unchanged. Therefore, the output of each convolution stage has a different spatial resolution. 3D convolution operations are used in the first and second stages of convolution to obtain a video frame sequence after 3D convolution.

[0074] Step S102: performing a time-domain pooling operation on the video frame sequence after the 3D convolution through the preset time-domain pooling layer to obtain a video frame sequence whose time dimension is a preset dimension.

[0075] It should be noted that, in order to synthesize a single video frame according to the video frame sequence after the 3D convolution, the preset dimension may be a dimension in which the time dimension is 1. The performing of the time domain pooling operation on the video frame sequence after the 3D convolution by the preset time domain pooling layer may be merging the time dimension of the input video frame sequence to 1 by the time domain pooling operation of the preset time domain pooling layer.

[0076] Step S103: performing a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0077] It should be noted that performing a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales may be performed by performing a convolution operation on the video frame sequence of the preset dimension using a preset 2D convolution layer or a 3D convolution layer to obtain semantic features of the video frame sequence at different spatiotemporal scales. In order to save the overhead of the convolution operation, a preset 2D convolution layer may be used to perform a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0078] It should be noted that the preset 2D convolution layer can be two convolution layers with a stride of 1 to keep the resolution of the feature map unchanged.

[0079] It should be understood that the four basic convolution operation units available in convolutional neural networks include straight-through 2D convolution, residual 2D convolution, straight-through 3D convolution and residual 3D convolution.

[0080] Among them, the straight-through or residual 2D convolution block: The straight-through 2D convolution module achieves the goal of feature learning by directly cascading the general 2D convolution layer x t+1 =F Conv2D (x t ), where F Conv2D Represents a nonlinear function based on 2D convolution. In order to solve the gradient vanishing problem faced by straight-through 2D convolution, the residual 2D convolution designed based on the residual learning idea can be expressed as follows:

[0081] x t+1 =x t +F Conv2D (x t )

[0082] Among them F Conv2D Used to learn x t+1 with x t The residual between .

[0083] Direct or residual 3D convolutional blocks: A natural way to encode spatiotemporal information is to directly upgrade the 2D convolutions in the two 2D convolutional blocks mentioned above to 3D convolutions, thereby simultaneously modeling the spatial information in each frame of the input video sequence and the spatial correlation between frames. The corresponding operation is expressed as follows:

[0084] x t+1 =F Conv3D (x t )

[0085] x t+1 =x t +F Conv3D (x t )

[0086] In practice, this embodiment adheres to the principle of using only the four different types of convolutional computing units described above. When using only 2D blocks or both 2D and 3D blocks, this embodiment designs a CNN structure for encoding the spatiotemporal information of the input video, namely, HSTCNN, which includes two possible design schemes. One is the "Early Fusion" scheme.

[0087] It should be understood that the technical feature of synthesizing video frames through 2D networks in the prior art is to first concatenate the T input video frames in the channel dimension and expand the number of channels of the network input from 3 to 3*T. Therefore, it is only necessary to modify the filter of the first convolution layer in the convolution block to a version that supports 3*T channels, and then the information in the video frame sequence can be directly mined through the cascaded 2D-CNN. However, in practice, it can be found that this direct "early fusion" solution at the pixel level is difficult to capture larger motion information and complex time relationships. Therefore, in contrast, the present embodiment innovatively proposes a 2D+3D fusion design scheme, which is a method of realizing the fusion of time dimension information by combining 3D and 2D computing units. It can regard the T input video frames as a three-dimensional cube of size T*H*W, and use multiple 3D convolution modules to learn low-level visual motion information in the time and space dimensions, and then use temporal average pooling operations and multiple 2D convolution modules to further learn high-level feature expressions to help the synthesis of video frames.

[0088] In the specific implementation, this embodiment uses 6 different CNN structures for HSTCNN in FSN. In Plain2D-15, the time information is "early fused" and followed by cascaded 2D convolution, while Res2D-15 replaces the Plain2D block with the Res2D block in stages 2 to 4. Res(3D+2D)-15 chooses a 3D+2D fusion solution, which uses Plain3D and Res3D blocks in the first and second stages respectively. Plain2D-29, Res2D-29 and Res(3D+2D)-29 are all obtained by adding additional computing units in stages 2, 3, and 4. For details, please refer to the following Table 1-The specific structural configuration of the network. Table 1 summarizes the specific structural configuration of the network:

[0089] Table 1 Specific structure configuration of the network

[0090]

[0091] Referring to Table 1, 3*3, 32 in Table 1 represent 2D convolutions with a length, width, and height of 3, respectively, and 32 convolution kernels or channels. 3*3*3, 32 in Table 1 represent 3D convolutions with a length, width, and height of 3, respectively, and 32 convolution kernels or channels. [3*3*3, 128] is used to represent a convolution unit, and 2 / 4 is used to represent a cascade connection of 2 or 4 convolution units. With the help of HSTCNN, this embodiment can obtain a good spatiotemporal feature representation of the input video frame sequence while maintaining controllable computational overhead. However, unlike traditional tasks such as video classification and action recognition that only require video features to be rich in semantic information, in video frame synthesis tasks based on adaptive convolution, the video features obtained by the network must not only contain sufficient semantic information to represent the high-level semantic content of the video, but also have sufficiently accurate spatial positioning capabilities. Only in this way can the convolution kernel parameters of high-quality adaptive convolution be further estimated based on the video features.

[0092] Typically, the richness of semantic information in convolutional neural networks (CNNs) and their ability to accurately locate features in space are in conflict. Shallow networks, limited by their receptive field size, can only focus on small and medium-scale local regions of the input image or video, such as edges, textures, and colors. This makes it difficult to reflect the semantic content of the video image at a global scale, but it is precisely this property that leads to their high spatial positioning accuracy. Deep networks can achieve a larger receptive field through multiple pooling operations within the network, allowing them to abstract semantic information over a larger range. However, this large coverage area leads to insufficient spatial positioning accuracy. The most direct way to resolve this contradiction is to use networks of different depths and sizes in parallel to simultaneously obtain features with different attribute tendencies from the input image or video, but this approach incurs a huge overhead in computational resources.

[0093] Based on the above-mentioned defects, this embodiment adopts a multi-scale spatiotemporal feature fusion structure MSTA. Similar to the idea of ​​connecting networks of different depths in parallel, by reusing a relatively shallow network structure, three networks of different depths are fused into one network (i.e., the HSTCNN mentioned above), thereby avoiding the consumption of more computing resources while having video features at different scales. The features output by the shallow network contain a large amount of local edge information that looks like "noise"; the middle-level network begins to focus on structural information within a larger range, such as human contours, road contours, umbrella contours, building contours, etc.; the deep network mainly reflects the areas closely related to the semantic content of the video (people running), at which time the local edge information has almost disappeared. In order to make full use of video features at different scales to obtain a good adaptive convolution kernel for the input video, and obtain reasonable (high-level features, semantic correctness), coherent (mid-level features, object coherence), and clear (shallow features, edge sharpness) video frame synthesis results, MSTA first scales the above three levels of features to a unified spatial resolution (i.e., the spatial resolution of shallow features), and uses an additional set of convolutions to fuse the three features after merging the channel dimension for use in the subsequent anti-motion blur adaptive convolution model.

[0094] This embodiment inputs a video frame sequence into the preset 3D convolution layer for convolution operation to obtain a 3D convolved video frame sequence; performs a temporal pooling operation on the 3D convolved video frame sequence through the preset temporal pooling layer to obtain a video frame sequence with a preset temporal dimension; and performs a convolution operation on the video frame sequence with the preset dimension through the preset 2D convolution layer to obtain semantic features of the video frame sequence at different temporal and spatial scales. This allows for high-quality video frame synthesis results.

[0095] refer to Figure 4 , Figure 4 FIG. 4 is a flow chart of a third embodiment of a video frame synthesis method according to the present invention.

[0096] Based on the above embodiments, in this embodiment, step S30 includes:

[0097] Step 301: performing adaptive convolution on the fused semantic features to obtain an adaptive convolution result.

[0098] It should be noted that the adaptive convolution is performed on the fused semantic features to obtain the adaptive convolution result. The adaptive convolution result can be obtained by determining the vertical motion convolution kernel and the horizontal motion convolution kernel corresponding to the video frame sequence according to the fused semantic features; simulating a 2D convolution kernel according to the vertical motion convolution kernel and the horizontal motion convolution kernel, and determining the adaptive convolution result through the 2D convolution kernel.

[0099] Determining the vertical motion convolution kernel and the horizontal motion convolution kernel corresponding to the video frame sequence based on the fused semantic features may include establishing the vertical motion convolution kernel and the horizontal motion convolution kernel corresponding to the video frame sequence using a preset 1D convolution kernel based on the fused semantic features; simulating a 2D convolution kernel in an adaptive convolution model based on the vertical motion convolution kernel and the horizontal motion convolution kernel, and determining an adaptive convolution result using the 2D convolution kernel. The adaptive convolution result may be a video frame synthesized by the preset adaptive convolution model.

[0100] In specific implementations, the existing adaptive convolution module often uses a two-step scheme of motion estimation and pixel synthesis in the classic video frame interpolation and extrapolation work based on optical flow or motion maps. This type of scheme relies heavily on the results of unreliable motion estimation. However, even the more advanced adaptive convolution technology routes have certain defects. The basic adaptive convolution method is to directly regard pixel synthesis as the result of a local convolution operation on the input video frame, which is called a single-step scheme here. At each pixel position (x, y), the above single-step scheme can estimate a 2D convolution kernel K for each input video frame. i (x,y)∈R s×s , and use the convolution kernel and the i The 2D image block P of the frame centered at pixel (x, y) i (x,y)∈R s×s Perform convolution operation. The output pixel value is the result of the above convolution operation on each input frame. In this process, the convolution kernel K i (x,y) captures both motion information and pixel resampling information, and uses this information for subsequent synthesis. This elegant mechanism can seamlessly embed the task of video frame synthesis into convolutional neural networks and model the changes in visual content caused by motion. One limitation of this solution is that the 2D convolution kernel K i (x, y). On the one hand, the quality requirements of the synthesized frame drive the convolution kernel to be large enough to contain sufficient information. At the same time, limited computing resources, parameter size, and memory space make it difficult to estimate the corresponding convolution kernel for all pixels simultaneously. Therefore, this method can only estimate the 2D convolution kernel pixel by pixel.

[0101] To solve the above problem, inspired by separable convolution, this embodiment uses a pair of 1D convolution kernels to model the vertical motion K and i,v (x,y)∈R s×1 and horizontal motion K i,h (x,y)∈R 1×s, thus achieving the goal of simulating 2D convolution kernels in the adaptive convolution flow. In particular, the adaptive convolution model includes several convolution layer kernel upsampling layers to form a predictor for performing dense convolution kernel prediction on a per-image block basis. Finally, the video frame I synthesized by the adaptive convolution module i The pixel (x, y) in is calculated by the following formula:

[0102]

[0103] Among them, * represents the convolution operation, represents the outer product calculation, The video frames synthesized by the adaptive convolution module are used to represent i (x, y) is used to represent the 2D image patch centered at pixel (x, y), K i (x,y) is used to represent the convolution of size s*s, k i,v (x,y) is used to represent the convolution in the vertical direction. Used to represent convolution in the horizontal direction.

[0104] Step S302: performing motion blur correction on the fused semantic features to obtain a motion blur correction result.

[0105] It should be noted that performing motion blur correction on the fused semantic features to obtain the motion blur correction result may be performing motion blur correction on the fused semantic features using a preset motion blur correction model to obtain the motion blur correction result. The preset motion blur correction model may be a motion blur correction stream (MCS). Performing motion blur correction on the fused semantic features using the preset motion blur correction model may be performing certain correction on pixels with relatively strong motion in the fused semantic features using the preset motion blur correction model.

[0106] Furthermore, in order to obtain a high-quality synthesized video frame, the step S302 may include: determining the degree of motion of each pixel in the video frame sequence based on the fused semantic features; determining the pixels that are motion blurred according to the degree of motion, and determining the motion mask of the pixels that are motion blurred; determining the correction bias for correcting motion blur according to the fused semantic features; and determining the motion blur correction result according to the motion mask and the correction bias.

[0107] It should be noted that the method of determining the degree of motion of each pixel in the video frame sequence based on the fused semantic features; determining the pixels belonging to motion blur according to the degree of motion, and determining the motion mask of the pixels belonging to motion blur; determining the correction bias for correcting motion blur according to the fused semantic features; determining the motion blur correction result according to the motion mask and the correction bias may be determining the motion blur correction result according to a preset motion blur correction model.

[0108] Determining the degree of motion of each pixel in the video frame sequence based on the fused semantic features may be determining the degree of motion of each pixel in the video frame sequence based on the fused semantic features through a preset motion blur correction model. It may be detecting the motion area in the video frame through the preset motion blur correction model and obtaining the degree of motion of each pixel in the motion area. Determining the motion mask of the pixels that are motion blurred may be determining the initial motion mask of the pixels with intense motion based on the degree of motion of each pixel in the motion area; and normalizing the initial motion mask to obtain a motion mask. Normalizing the initial motion mask may be normalizing the initial motion mask to the interval (0,1) through a sigmoid function. Determining the correction bias for correcting motion blur based on the fused semantic features may be obtaining the correction bias for correcting motion blurred pixels based on video feature regression.

[0109] Step S303: superimposing the adaptive convolution result and the motion blur correction result to obtain a video frame synthesis result.

[0110] It should be noted that the adaptive convolution result and the motion blur correction result are superimposed to obtain a video frame synthesis result, which can be calculated by the following formula:

[0111]

[0112] Among them, (·) represents the element-by-element multiplication operation, Used to characterize synthetic video frames, The video frames synthesized by the adaptive convolution module are used to represent M I Used to characterize the motion mask, R I Used to characterize the correction bias.

[0113] It should be understood that, based on high-quality video features, the above-mentioned ACS can already provide good results for synthesized video frames. However, for some pixels with more intense motion, synthesis using only ACS may still lead to blurred synthesis results, because these pixels are difficult to obtain by performing adaptive convolution operations on pixels near the corresponding positions in the input video frame that can be covered by the adaptive convolution kernel corresponding to the current position.

[0114] In the specific implementation, the results output by ACS A good approximation of the synthesized video frame in most areas For those pixels with poor approximation and strong motion, some correction is required. To indirectly determine the value of the pixel that needs to be corrected, this helps simplify the learning objectives of the neural network and helps the solver optimize the model. MCS has the same prediction structure as ACS. Its output consists of two parts: 1. The motion mask M obtained by detecting the violent motion area in the video to mark the violent motion pixels I , the mask is normalized to the (0,1) interval by the sigmoid function; 2. The correction bias R for correcting motion blurred pixels obtained by video feature regression I Although it is possible to generate a correction bias that covers the entire spatial range of the video frame without a mask, this scheme takes into account that doing so may introduce additional noise to the originally well-approximated area and thus have a negative impact, so the above-mentioned mask scheme is adopted. Finally, the FSN synthesized video frame is calculated by the following formula:

[0115]

[0116] In a specific implementation, the training and optimization of FSN can be: to measure the synthesized video frame and the true value I gt We use a simple L1-norm loss function on each color channel to evaluate the loss function pixel by pixel. In FSN, the real value I is used gt To impose additional constraints on the adaptive convolution flow, which implicitly constitutes the constraints of residual learning. Therefore, the final loss function L FSN It can be expressed as:

[0117]

[0118] λ is a hyperparameter used to balance the two loss function terms and can be set to 1 in this embodiment. This embodiment uses the Caffe deep learning framework for code implementation. During training, the network parameters are optimized using the Adam optimizer, where β1 = 0.9 and β2 = 0.999. The learning rate is initialized to 0.0001 and the batch size is set to 128. In this embodiment, the entire training process requires 50,000 training optimization iterations.

[0119] This paper experiments on two video datasets: UCF-101 and Kinetics-HD. Using MAE, RMSE, PSNR, and SSIM as performance evaluation metrics, the paper demonstrates its feasibility and performance advantages of video frame synthesis. First, the paper experimentally compares the video frame synthesis performance of different spatiotemporal convolutional neural network architectures. Table 2, Performance of Video Frame Interpolation and Extrapolation Synthesis Tasks Using Different Network Architectures, summarizes the performance (PSNR) of video frame interpolation and extrapolation synthesis tasks using different network architectures on the Kinetics-HD dataset.

[0120] Table 2 - Performance of different network structures for video frame interpolation and extrapolation synthesis tasks

[0121]

[0122] As shown in Table 2, "Architecture" represents the network architecture, "#param" represents the number of parameters corresponding to the network architecture, "Inter" represents the interpolation evaluation score, and "Extra" represents the extrapolation evaluation score. Table 2 shows that the interpolation and extrapolation evaluation scores of the Res(3D+2D)-29 network architecture are both higher than those of other network architectures. Therefore, the CNN architecture using 3D+2D fusion achieves better frame synthesis performance than the "early fusion" approach when using either the Plain2D or Res2D computation blocks. This demonstrates the significant advantages of combining 3D and 2D computation blocks to learn spatiotemporal feature representations for videos. At the same time, the 3D+2D fusion approach slightly increases the number of parameters. Res(3D+2D)-29, leveraging the deep architecture of FSN, achieves better performance than Res(3D+2D)-15. Unless otherwise specified, in the following evaluations, we use Res(3D+2D)-29 as the CNN architecture in FSN. Next, we compare the FSN proposed in this example with other baseline methods on two datasets. The Average method directly uses the average value of the input frame as the interpolation result; TVL1 refers to a method for video frame synthesis based on the optical flow estimation method; AdaConv, SepConv and DVF are all benchmark methods based on convolutional neural networks.

[0123] The following Table 3 - Performance of various methods in the video frame interpolation synthesis task summarizes the performance of various methods in the video frame interpolation synthesis task. By comparison, it can be seen that the FSN of this application has a relatively obvious performance advantage on both evaluation datasets.

[0124] Table 3 - Performance of various methods in video frame interpolation synthesis task

[0125]

[0126] Table 4 below summarizes the performance comparison in the video frame extrapolation task, which also verifies the advantages of the FSN method.

[0127] Table 4 - Performance comparison of video frame extrapolation tasks

[0128]

[0129] This embodiment simulates a 2D convolution kernel within a preset adaptive convolution model based on the fused semantic features, and determines an adaptive convolution result using the 2D convolution kernel. Motion blur correction is performed on the fused semantic features using a preset motion blur correction model to obtain a motion blur correction result. The adaptive convolution result and the motion blur correction result are superimposed to obtain a video frame synthesis result. This embodiment can better model the spatiotemporal information in a video, thereby facilitating better video frame synthesis results. The designed network structure is more efficient, and resource consumption is more manageable.

[0130] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the video frame synthesis device of the present invention.

[0131] like Figure 5 As shown, the video frame synthesis device proposed in the embodiment of the present invention includes:

[0132] A semantic feature determination module 10 is configured to input a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales;

[0133] A fusion module 20 is used to perform feature fusion on the semantic features to obtain fused semantic features;

[0134] The synthesis module 30 is configured to determine a video frame synthesis result based on the fused semantic features.

[0135] This embodiment includes inputting a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales; performing feature fusion on the semantic features to obtain fused semantic features; and determining a video frame synthesis result based on the fused semantic features. Since this embodiment obtains semantic features of a video frame sequence at different spatiotemporal scales, performs feature fusion on the semantic features to obtain fused semantic features, and determines a video frame synthesis result based on the fused semantic features. Compared with the existing method of synthesizing video frames by densely estimating the motion between given video frames, the above method of this embodiment does not rely on optical flow to guide the synthesis of video frames, but determines the video frame synthesis result by performing feature fusion on the semantic features of video frames at different spatiotemporal scales, which can avoid obvious artifacts in the synthesized video and improve the quality of the synthesized video frames.

[0136] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the embodiment scheme, and no limitation is made here.

[0137] In addition, for technical details not fully described in this embodiment, reference can be made to the video frame synthesis method provided in any embodiment of the present invention, and will not be repeated here.

[0138] Based on the first embodiment of the video frame synthesis device of the present invention, a second embodiment of the video frame synthesis device of the present invention is proposed.

[0139] In this embodiment, the semantic feature determination module 10 is further configured to input the video frame sequence into the preset 3D convolution layer for convolution operation to obtain a video frame sequence after 3D convolution;

[0140] Performing a time-domain pooling operation on the video frame sequence after the 3D convolution through the preset time-domain pooling layer to obtain a video frame sequence with a time dimension of a preset dimension;

[0141] A convolution operation is performed on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0142] Furthermore, the semantic feature determination module 10 is further configured to perform a convolution operation on the video frame sequence of the preset dimension through the preset 2D convolution layer to obtain semantic features of the video frame sequence at different spatiotemporal scales.

[0143] Furthermore, the synthesis module 30 is further configured to perform adaptive convolution on the fused semantic features to obtain an adaptive convolution result;

[0144] Performing motion blur correction on the fused semantic features to obtain a motion blur correction result;

[0145] The adaptive convolution result and the motion blur correction result are superimposed to obtain a video frame synthesis result.

[0146] Furthermore, the synthesis module 30 is further configured to determine the degree of motion of each pixel in the video frame sequence based on the fused semantic features;

[0147] Determining pixels that are motion blurred according to the degree of motion, and determining a motion mask for the pixels that are motion blurred;

[0148] determining a correction bias for correcting motion blur based on the fused semantic features;

[0149] The motion blur correction result is determined according to the motion mask and the correction bias.

[0150] Furthermore, the synthesis module 30 is also used to determine the vertical motion convolution kernel and the horizontal motion convolution kernel corresponding to the video frame sequence based on the fused semantic features; simulate the 2D convolution kernel based on the vertical motion convolution kernel and the horizontal motion convolution kernel, and determine the adaptive convolution result through the 2D convolution kernel.

[0151] Furthermore, the fusion module 20 is also used to scale the semantic features to the target spatial resolution through a preset multi-scale spatiotemporal feature fusion model; merge each scaled semantic feature in the channel dimension to obtain the semantic feature to be fused; and fuse the semantic features to be fused to obtain the fused semantic feature.

[0152] Other embodiments or specific implementations of the video frame synthesis device of the present invention can refer to the above-mentioned method embodiments and will not be described in detail here.

[0153] In addition, an embodiment of the present invention further provides a storage medium, on which a video frame synthesis program is stored. When the video frame synthesis program is executed by a processor, the steps of the video frame synthesis method described above are implemented.

[0154] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0155] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0156] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0157] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video frame synthesis method, characterized in that: The video frame synthesis method comprises the following steps: Inputting the video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales; Performing feature fusion on the semantic features to obtain fused semantic features; Determining a video frame synthesis result according to the fused semantic features; The preset hybrid spatiotemporal convolutional network includes: a preset 3D convolutional layer, a preset time domain pooling layer and a preset 2D convolutional layer. The semantic features include high-level features, middle-level features and shallow-level features. The high-level features include semantic correctness-related features, the middle-level features include object coherence-related features, and the shallow-level features include edge sharpness-related features.

2. The video frame synthesis method according to claim 1, wherein: The preset hybrid spatiotemporal convolutional network includes: a preset 3D convolutional layer and a preset time domain pooling layer; The step of inputting the video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales includes: Inputting the video frame sequence into the preset 3D convolution layer for convolution operation to obtain a video frame sequence after 3D convolution; Performing a time-domain pooling operation on the video frame sequence after the 3D convolution through the preset time-domain pooling layer to obtain a video frame sequence with a time dimension of a preset dimension; A convolution operation is performed on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

3. The video frame synthesis method according to claim 2, wherein: The preset hybrid spatiotemporal convolutional network further includes: a preset 2D convolutional layer; The performing a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales includes: The preset 2D convolution layer is used to perform a convolution operation on the video frame sequence of the preset dimension to obtain semantic features of the video frame sequence at different spatiotemporal scales.

4. The video frame synthesis method according to claim 1, wherein: The step of determining the video frame synthesis result according to the fused semantic features includes: Performing adaptive convolution on the fused semantic features to obtain an adaptive convolution result; Performing motion blur correction on the fused semantic features to obtain a motion blur correction result; The adaptive convolution result and the motion blur correction result are superimposed to obtain a video frame synthesis result.

5. The video frame synthesis method according to claim 4, wherein: The step of performing motion blur correction on the fused semantic features to obtain a motion blur correction result includes: determining a motion degree of each pixel in the video frame sequence based on the fused semantic features; Determining pixels that are motion blurred according to the degree of motion, and determining a motion mask for the pixels that are motion blurred; determining a correction bias for correcting motion blur based on the fused semantic features; The motion blur correction result is determined according to the motion mask and the correction bias.

6. The video frame synthesis method according to claim 4, wherein: The step of performing adaptive convolution on the fused semantic features to obtain an adaptive convolution result includes: Determining a vertical motion convolution kernel and a horizontal motion convolution kernel corresponding to the video frame sequence according to the fused semantic features; A 2D convolution kernel is simulated according to the vertical motion convolution kernel and the horizontal motion convolution kernel, and an adaptive convolution result is determined by the 2D convolution kernel.

7. The video frame synthesis method according to any one of claims 1 to 6, wherein: The step of fusing the semantic features to obtain fused semantic features includes: Scaling the semantic features to the target spatial resolution by using a preset multi-scale spatiotemporal feature fusion model; Merge each scaled semantic feature in the channel dimension to obtain the semantic feature to be fused; The semantic features to be fused are fused to obtain fused semantic features.

8. A video frame synthesis device, characterized in that: The video frame synthesis device comprises: A semantic feature determination module, configured to input a video frame sequence into a preset hybrid spatiotemporal convolutional network to obtain semantic features of the video frame sequence at different spatiotemporal scales; A fusion module, configured to perform feature fusion on the semantic features to obtain fused semantic features; A synthesis module, configured to determine a video frame synthesis result based on the fused semantic features; The preset hybrid spatiotemporal convolutional network includes: a preset 3D convolutional layer, a preset time domain pooling layer and a preset 2D convolutional layer. The semantic features include high-level features, middle-level features and shallow-level features. The high-level features include semantic correctness-related features, the middle-level features include object coherence-related features, and the shallow-level features include edge sharpness-related features.

9. A video frame synthesis device, characterized in that: The device includes: a memory, a processor, and a video frame synthesis program stored in the memory and executable on the processor, wherein the video frame synthesis program is configured to implement the steps of the video frame synthesis method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a video frame synthesis program, which, when executed by a processor, implements the steps of the video frame synthesis method according to any one of claims 1 to 7.

Images