A lightweight deep video frame interpolation method based on attention mechanism
By constructing a lightweight video frame interpolation model based on an attention mechanism, the problems of complexity and low efficiency in existing technologies are solved, achieving efficient video frame interpolation and improving interpolation quality and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-28
- Publication Date
- 2026-04-03
AI Technical Summary
Existing video frame interpolation techniques have complex processing procedures and low efficiency. Furthermore, deep learning models are large and complex, making them difficult to train and unable to meet real-time frame interpolation requirements. They also perform poorly in situations involving large motion displacement, occlusion, and brightness changes.
A lightweight deep video frame interpolation method based on attention mechanism is adopted to construct an initial interpolation model including a preprocessing module, a motion enhancement module, a U-Net backbone network and a generation sub-module. The model parameters are optimized by end-to-end training using L1 loss function and perceptual loss function. The motion enhancement module is used to enhance feature representation, and a ghost module is introduced to replace the basic module of U-Net backbone network to reduce the number of parameters.
It improves frame interpolation quality and visual effects, reduces model memory consumption, enhances frame interpolation efficiency, and balances objective evaluation metrics and subjective visual effects.
Smart Images
Figure CN116320246B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to a lightweight depth video frame interpolation method based on an attention mechanism. Background Technology
[0002] Currently, with the rapid development of terminal and multimedia technologies, high-quality video applications have become an urgent need. Frame rate is an important indicator for measuring video quality, defined as the number of frames refreshed per second by a display device.
[0003] Video frame interpolation algorithms aim to synthesize a non-existent intermediate frame using information from two consecutive input video frames. They have practical applications in numerous fields such as visual quality enhancement, video compression, and multi-view synthesis. Frame interpolation technology can effectively improve the frame rate of a video, ensure the continuity of moving objects in the video, and effectively alleviate visual problems such as flickering, motion blur, and image stuttering caused by low frame rate videos, providing users with a more comfortable viewing experience.
[0004] Traditional video frame interpolation techniques primarily involve two steps: motion estimation and motion compensation. Classic algorithms include optical flow and block matching. Optical flow requires enormous computational resources, making it unsuitable for real-time interpolation, and its predictions are inaccurate under conditions of significant motion displacement, occlusion, or brightness variations. Block matching is prone to issues such as image holes and overlap. Many existing deep learning-based frame interpolation models are large, complex, and have numerous parameters, making them difficult to train.
[0005] It is evident that there is an urgent need for an efficient and simple lightweight deep video frame interpolation method based on an attention mechanism. Summary of the Invention
[0006] In view of this, the present disclosure provides a lightweight depth video frame interpolation method based on an attention mechanism, which at least partially solves the problems of complex processing flow and poor processing efficiency in the prior art.
[0007] In a first aspect, embodiments of this disclosure provide a lightweight depth video frame interpolation method based on an attention mechanism, including:
[0008] Step 1: Construct a lightweight video frame interpolation model based on the attention mechanism. This model includes a preprocessing module, a motion enhancement module, a U-Net backbone network, and a generation sub-module.
[0009] Step 2: Obtain multiple image triples as a training set, wherein the image triples include the input frames of the two preceding and following frames and the reference frame in the middle.
[0010] Step 3: Initialize model parameters, set the number of iterations, set the L1 loss function and perceptual loss function according to the difference between the interpolated frame synthesized by the initial model and the reference frame, and perform end-to-end training on the initial model, iteratively optimize the parameters of the initial model, and obtain the target interpolation model;
[0011] Step 4: Input the data to be processed into the target interpolation model to obtain the synthesized intermediate frame image as the interpolation frame.
[0012] According to one specific implementation of this disclosure, the preprocessing module includes three two-dimensional convolutional layers.
[0013] According to a specific implementation of an embodiment of this disclosure, the motion enhancement module includes a channel enhancement submodule and a spatial enhancement submodule, wherein the channel enhancement submodule includes a pooling layer and two two-dimensional convolutional layers, and the spatial enhancement submodule includes a global average pooling layer and a two-dimensional convolutional layer.
[0014] According to one specific implementation of this disclosure, the U-Net backbone network includes a ghost module.
[0015] According to a specific implementation of an embodiment of this disclosure, the generation submodule includes a convolution kernel submodule, an offset submodule, an occlusion mask submodule, and an adaptive deformation layer.
[0016] According to a specific implementation of an embodiment of this disclosure, step 4 specifically includes:
[0017] The data to be processed is input into the target interpolation model. The preprocessing module expands the input frame in the channel dimension to obtain the feature map of the input frame.
[0018] The channel enhancement submodule of the motion enhancement module performs global average pooling on the feature map of the input frame and then expands the channel dimension through a two-dimensional convolutional layer. The corresponding channel weights are then taken by difference and compressed to the original number of channels through a two-dimensional convolutional layer. The obtained channel attention coefficients are then multiplied with the corresponding channel features.
[0019] The spatial enhancement submodule of the motion enhancement module obtains spatial attention coefficients by calculating the similarity between channels at each location through a global average pooling layer and a two-dimensional convolutional layer, and then multiplies them with the features of the preceding and following frames.
[0020] The Ling module performs regular 2D convolution on the feature map to generate a partial feature map. The remaining feature map is generated using a cheap operation and depthwise separable convolution to generate a redundant feature map. Finally, the two parts of the feature map are stacked in the channel dimension as the final output feature.
[0021] After the output features are input into the generation submodule for convolution kernel prediction, offset prediction, and occlusion mask prediction, an adaptive deformation layer is used to sample and linearly combine the corresponding pixels according to the mapping relationship to generate an intermediate frame image as an interpolation frame.
[0022] According to a specific implementation of this disclosure, before step 2, the method further includes:
[0023] Data augmentation of all initial triples is performed by randomly cropping, flipping, and swapping the order of preceding and following frames to obtain the corresponding image triples.
[0024] According to a specific implementation of this disclosure, the expression for the L1 loss is:
[0025]
[0026] in, , These represent the model's output frame and reference frame, respectively.
[0027] The expression for the perception loss is:
[0028]
[0029] It is the feature extractor of the two-dimensional convolutional layer in the pre-trained initial interpolation model.
[0030] The lightweight deep video frame interpolation scheme based on the attention mechanism in this embodiment includes: Step 1, constructing an initial interpolation model for lightweight video frames based on the attention mechanism, which sequentially includes a preprocessing module, a motion enhancement module, a U-Net backbone network, and a generation submodule; Step 2, acquiring multiple image triplets as a training set, wherein the image triplets include two input frames and an intermediate reference frame; Step 3, initializing model parameters, setting the number of iterations, setting the L1 loss function and perceptual loss function according to the difference between the interpolated frame synthesized by the initial model and the reference frame, and performing end-to-end training on the initial model, iteratively optimizing the parameters of the initial model to obtain a target interpolation model; Step 4, inputting the data to be processed into the target interpolation model to obtain the synthesized intermediate frame image as the interpolation frame.
[0031] The beneficial effects of this disclosure are as follows: By using the motion enhancement module, the motion-related feature expression is enhanced by comparing the differences between the feature maps in the channel dimension and the spatial dimension, while suppressing the feature expression of unimportant information such as background. This improves the interpolation quality index and visual effect, while greatly reducing the number of parameters and the memory consumption of the model, thus improving the interpolation efficiency. By adjusting the coefficients of the L1 loss function and the perceptual loss function, the model can achieve both objective evaluation index and subjective visual effect. Attached Figure Description
[0032] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 A flowchart illustrating a lightweight depth video frame interpolation method based on an attention mechanism provided in this disclosure embodiment;
[0034] Figure 2 This is a schematic diagram of the structure of a motion enhancement module provided in an embodiment of the present disclosure;
[0035] Figure 3 This is a schematic diagram of the structure of a ghost module provided in an embodiment of the present disclosure;
[0036] Figure 4 A comparison diagram of the output results of a lightweight depth video frame interpolation method based on an attention mechanism provided in this disclosure and existing methods. Detailed Implementation
[0037] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0038] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0039] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0040] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0041] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0042] This disclosure provides a lightweight deep video frame interpolation method based on an attention mechanism, which can be applied to video processing in Internet multimedia scenarios.
[0043] See Figure 1 This is a flowchart illustrating a lightweight depth video frame interpolation method based on an attention mechanism provided in an embodiment of this disclosure. Figure 1 As shown, the method mainly includes the following steps:
[0044] Step 1: Construct a lightweight video frame interpolation model based on the attention mechanism. This model includes a preprocessing module, a motion enhancement module, a U-Net backbone network, and a generation sub-module.
[0045] Furthermore, the preprocessing module includes three two-dimensional convolutional layers.
[0046] Furthermore, the motion enhancement module includes a channel enhancement submodule and a spatial enhancement submodule, wherein the channel enhancement submodule includes a pooling layer and two two-dimensional convolutional layers, and the spatial enhancement submodule includes a global average pooling layer and a two-dimensional convolutional layer.
[0047] Furthermore, the U-Net backbone network includes ghost modules.
[0048] Furthermore, the generation submodule includes a convolution kernel submodule, an offset submodule, an occlusion mask submodule, and an adaptive deformation layer.
[0049] In practice, the preprocessing module consists of three two-dimensional convolutional networks, the purpose of which is to expand the feature map in the channel dimension and provide more reference information for subsequent modules;
[0050] like Figure 2 As shown, the motion enhancement module is divided into two stages. In the first stage, the feature map is processed through a global average pooling layer and a two-dimensional convolutional layer. By comparing the differences between the channel features of the previous and next frames, a set of channel attention coefficients are obtained. These coefficients are then multiplied with the corresponding channel features to obtain the feature output, thereby achieving motion enhancement in the channel dimension. In the second stage, the spatial differences of the features are also considered. By calculating the similarity between the features at different positions in the previous and next frames, a set of spatial attention coefficients are obtained and then multiplied with the corresponding features to achieve motion enhancement in the spatial dimension.
[0051] The U-Net backbone network, also known as the encoder-decoder structure, has a first half used for feature extraction and a second half for upsampling. There are skip connections between the two parts, which realizes the fusion of deep and shallow features.
[0052] The generation submodule consists of the convolution kernel submodule. , Offset submodule and occlusion mask submodule It consists of an adaptive deformation layer, where the weight submodule is used to predict the weights of the convolution kernel for each pixel, and the size of the convolution kernel is optional. The offset submodule is used to predict the offset at each position on the convolution kernel, and the occlusion mask predicts the proportion of each pixel in the previous and next frames. The frame interpolation result can be expressed as:
[0053] like Figure 3 As shown, the ghost module first performs a conventional two-dimensional convolution operation on the feature map to generate a partial feature map, which can be represented as:
[0054]
[0055] in, This represents the input feature map, with dimension 1. , The kernel is a convolutional layer with dimension 1. ,in The number of channels for the output features. For compression ratio, the remaining Each feature map uses a cheap operation, typically employing depthwise separable convolution. Redundant feature maps are generated, and finally the two feature maps are stacked along the channel dimension to obtain the final feature output. , can be represented as:
[0056] .
[0057] Step 2: Obtain multiple image triples as a training set, wherein the image triples include the input frames of the two preceding and following frames and the reference frame in the middle.
[0058] Optionally, before step 2, the method further includes:
[0059] Data augmentation of all initial triples is performed by randomly cropping, flipping, and swapping the order of preceding and following frames to obtain the corresponding image triples.
[0060] In specific implementation, the triplet refers to three consecutive frames of images in the video. The first and last two frames are taken as the input frames of the model, and the middle frame is taken as the reference frame of the model. The selected training set should cover various motions and scenes to better train the model and improve its generalization ability. The bias caused by prior information can also be eliminated by randomly cropping, flipping, and changing the order of the first and last frames to achieve data augmentation. The processed image triplet is used as the training set to train the initial model in subsequent steps.
[0061] Step 3: Initialize model parameters, set the number of iterations, set the L1 loss function and perceptual loss function according to the difference between the interpolated frame synthesized by the initial model and the reference frame, and perform end-to-end training on the initial model, iteratively optimize the parameters of the initial model, and obtain the target interpolation model;
[0062] Furthermore, the expression for the L1 loss is:
[0063]
[0064] in, , These represent the model's output frame and reference frame, respectively.
[0065] The expression for the perception loss is:
[0066]
[0067] It is the feature extractor of the two-dimensional convolutional layer in the pre-trained initial interpolation model.
[0068] In practice, the initialization parameter method described above adopts standard initialization.
[0069] The initial iteration count is 0, and the maximum iteration count is 50.
[0070] The loss function consists of two parts: L1 loss and perceptual loss. , Let represent the model's output frame and reference frame, respectively. The expression for L1 loss is:
[0071]
[0072] Perceptual loss makes the synthesized image appear more realistic in visual effect; the expression is:
[0073]
[0074] It is the feature extractor of the conv4_3 layer in the pre-trained VGG16 model.
[0075] Then, the initial model is trained end-to-end, and the parameters of the initial model are iteratively optimized to obtain the target interpolation model.
[0076] Step 4: Input the data to be processed into the target interpolation model to obtain the synthesized intermediate frame image as the interpolation frame.
[0077] Based on the above embodiments, step 4 specifically includes:
[0078] The data to be processed is input into the target interpolation model. The preprocessing module expands the input frame in the channel dimension to obtain the feature map of the input frame.
[0079] The channel enhancement submodule of the motion enhancement module performs global average pooling on the feature map of the input frame and then expands the channel dimension through a two-dimensional convolutional layer. The corresponding channel weights are then taken by difference and compressed to the original number of channels through a two-dimensional convolutional layer. The obtained channel attention coefficients are then multiplied with the corresponding channel features.
[0080] The spatial enhancement submodule of the motion enhancement module obtains spatial attention coefficients by calculating the similarity between channels at each location through a global average pooling layer and a two-dimensional convolutional layer, and then multiplies them with the features of the preceding and following frames.
[0081] The Ling module performs regular 2D convolution on the feature map to generate a partial feature map. The remaining feature map is generated using a cheap operation and depthwise separable convolution to generate a redundant feature map. Finally, the two parts of the feature map are stacked in the channel dimension as the final output feature.
[0082] After the output features are input into the generation submodule for convolution kernel prediction, offset prediction, and occlusion mask prediction, an adaptive deformation layer is used to sample and linearly combine the corresponding pixels according to the mapping relationship to generate an intermediate frame image as an interpolation frame.
[0083] In practice, after stopping the iteration to obtain the target interpolation model, the data to be processed can be input into the target interpolation model. The preprocessing module expands the input frame in terms of channel dimension to obtain the feature map of the input frame. Then, the channel enhancement submodule of the motion enhancement module performs global average pooling on the feature map of the input frame and expands the channel dimension through a two-dimensional convolutional layer. The corresponding channel weights are then taken as a difference and compressed to the original number of channels through a two-dimensional convolutional layer. The obtained channel attention coefficients are multiplied with the corresponding channel features. The spatial enhancement submodule of the motion enhancement module obtains the spatial attention coefficients by calculating the similarity between channels at each position through a global average pooling layer and a two-dimensional convolutional layer, and multiplies them with the features of the preceding and following frames.
[0084] The Ling module then performs a standard 2D convolution operation on the feature maps to generate partial feature maps. The remaining feature maps are processed using a cheap operation, employing depthwise separable convolution to generate redundant feature maps. Finally, the two feature maps are stacked along the channel dimension to obtain the final output feature. This output feature is then input into the generation submodule for convolution kernel prediction, offset prediction, and occlusion mask prediction. An adaptive deformation layer then samples and linearly combines the corresponding pixels according to the mapping relationship to generate an intermediate frame image as the interpolated frame.
[0085] The lightweight deep video frame interpolation method based on the attention mechanism provided in this embodiment enhances motion-related feature representation and suppresses the representation of unimportant information such as background by using a motion enhancement module and comparing the differences between feature maps in the channel and spatial dimensions. It also introduces a ghost module to replace the two-dimensional convolutional layer in the U-Net backbone, improving frame interpolation quality and visual effects while significantly reducing the number of parameters, lowering memory consumption, and increasing interpolation efficiency. The loss function used to train the model includes both L1 loss and perceptual loss; adjusting their coefficients allows the model to consider both objective evaluation metrics and subjective visual effects.
[0086] The following will illustrate this solution through a specific embodiment. The specific steps of the attention-based lightweight depth video frame interpolation method are as follows:
[0087] S1: Obtain the training and test sets of the model. For an image triple, the first and last two frames are the input frames of the model, and the middle frame is the reference frame of the model.
[0088] In this embodiment, the training set uses the large-scale, high-quality dataset Vimeo90K, which contains 73,171 image triples, covering various motions and scenes. The input frames are first randomly cropped into... The data augmentation process involves sub-blocks with a probability of 0.5, horizontal and vertical flipping, and swapping the order of preceding and following frames to eliminate biases introduced by prior information. The test set is the Middlebury OTHER dataset, commonly used for evaluating frame interpolation effects, which contains 12 scenes.
[0089] S2: Construct a lightweight frame interpolation network model based on the attention mechanism, which includes a preprocessing module, a motion enhancement module, a U-Net backbone network, and a generation sub-module. The U-Net backbone network reduces parameters by introducing a Ghost Module, thus making the network model lightweight.
[0090] The preprocessing module consists of three two-dimensional convolutional networks with 32, 64, and 64 kernels, respectively. The purpose is to expand the feature map in the channel dimension and provide more reference information for the subsequent motion enhancement module.
[0091] like Figure 2 As shown, the motion enhancement module in this embodiment is divided into two stages. The first stage is motion enhancement in the channel dimension, which first involves inputting the feature maps of the previous and next frames. Global average pooling is performed separately. To provide more dimensional information, two single-layer 2D convolutions are used to expand the channel dimension. The feature values obtained after global average pooling of the feature maps of the corresponding channels in the two frames are subtracted.
[0092]
[0093] Then, a single-layer two-dimensional convolutional network is used to compress the channels back to their original number. In this embodiment, a compression factor of 0.5 is selected.
[0094]
[0095] in These are the feature values obtained after average pooling of features from the preceding and following frames. These represent the parameters that the two-dimensional convolutional layer needs to learn. It is the sigmoid activation function. The obtained channel attention coefficients.
[0096] Finally, the obtained channel attention coefficients are multiplied by the corresponding channel features to obtain the final feature output, thus achieving motion enhancement in the channel dimension:
[0097]
[0098] Phase two considers motion enhancement in the spatial dimension, calculating the feature similarity at each location between two frames. High similarity represents regions with little change in the background, which are assigned low weights, while low similarity represents motion-related regions, which are assigned high weights. Finally, an attention map in the spatial dimension is generated. The formula for calculating feature similarity is expressed as:
[0099]
[0100] A second 2D convolution layer is applied to obtain spatial attention coefficients, which are then multiplied with the features of the preceding and following frames. This achieves motion enhancement in the spatial channel dimension and adds residual connections to better aid model training and convergence. This can be represented as:
[0101]
[0102] The U-Net backbone network, also known as the encoder-decoder structure, has a front half for feature extraction and a back half for upsampling. There are skip connections between the front and back parts, which enables the fusion of deep and shallow features.
[0103] The generation submodule consists of the convolution kernel submodule. , Offset submodule and occlusion mask submodule The interpolation process consists of several modules, including a weighting submodule to predict the weights of the convolutional kernel for each pixel (the kernel size is optional, here we choose 3), an offset submodule to predict the offset at each position on the convolutional kernel, an occlusion mask to predict the proportion of each pixel in the preceding and following frames, and a sigmoid activation function with values ranging from 0 to 1 for each pixel, representing the sampling weights of the preceding and following frames. The final interpolation result can be represented as follows:
[0104] like Figure 3 As shown, in this embodiment, the compression ratio of the ghost module is set to 2. That is, the feature map is first generated by performing a conventional two-dimensional convolution operation to generate half of the original feature map, and the remaining half of the feature map is generated by a depthwise separable convolution operation. Finally, the two parts of the feature map are stacked in the channel dimension as the final feature output.
[0105] S3: Initialize model parameters, set the number of iterations, set the loss function based on the difference between the synthesized frame and the reference frame, train the frame interpolation model end-to-end, and continuously iterate to optimize the network parameters;
[0106] In this embodiment, the parameter initialization method adopts standard initialization;
[0107] In this embodiment, the initial number of iterations is 0, and the total number of iterations is 50.
[0108] The loss function consists of two parts: L1 loss and perceptual loss. , Let represent the model's output frame and reference frame, respectively. The expression for L1 loss is:
[0109]
[0110] Perceptual loss makes the synthesized image appear more realistic in visual effect; the expression is:
[0111]
[0112] This is the feature extractor of the conv4_3 layer in the pre-trained VGG16 model. In this embodiment, the coefficients of the two losses are 0.1 and 0.01, respectively.
[0113] S4: Obtain the interpolation results. After training the interpolation model, input the test dataset into the model to obtain the synthesized intermediate frame image.
[0114] The frame interpolation results can be categorized into subjective and objective interpolation effects. Subjective interpolation effects are measured by human perception, while objective interpolation effects are measured by two parameters: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The evaluation of frame interpolation quality is validated for complex scenarios such as large motion, blur, or occlusion.
[0115] The above video frame interpolation method will be illustrated below with specific examples.
[0116] In this example, the video frame interpolation models are: SepConv (Video frame interpolation via adaptive separable convolution), DVF (Video frame synthesis using deep voxelflow), MIND (Learning image matching by simply watching video), SuperSlomo (Super slomo: High quality estimation of multiple intermediate frames for video interpolation), Phase Based (Phase-based frame interpolation for video), and AdaCof (Adacof: Adaptive collaboration of flows for video frame interpolation). The output images obtained by each method are as follows: Figure 4 As shown.
[0117] In this example, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are also used to illustrate the image interpolation effect of the embodiments of the present invention. PSNR is the most common and widely used objective evaluation method for assessing image quality; a higher PSNR value indicates better quality output image. SSIM is an index that measures the similarity between two images and can better represent visual perception; a higher SSIM value indicates that the output image is closer to the real image, and the better the image quality. The various metrics of each method on the Middlebury test set are shown in Table 1. It should be noted that in Table 1, the last row represents the interpolation method used in this invention.
[0118]
[0119] Table 1
[0120] from Figure 4 As can be seen from Table 1, the intermediate frame images obtained using the attention-based lightweight video interpolation method proposed in this invention are significantly better than those obtained using other methods.
[0121] The units described in the embodiments of this disclosure can be implemented in software or in hardware.
[0122] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.
[0123] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.
Claims
1. A lightweight depth video frame interpolation method based on an attention mechanism, characterized in that, include: Step 1: Construct an initial interpolation model for lightweight video frames based on an attention mechanism. This model includes a preprocessing module, a motion enhancement module, a U-Net backbone network, and a generation sub-module. The U-Net backbone network includes a ghost module. Step 2: Obtain multiple image triples as a training set, wherein the image triples include the input frames of the two preceding and following frames and the reference frame in the middle. Step 3: Initialize model parameters, set the number of iterations, set the L1 loss function and perceptual loss function according to the difference between the interpolated frame synthesized by the initial model and the reference frame, and perform end-to-end training on the initial model, iteratively optimize the parameters of the initial model, and obtain the target interpolation model; Step 4: Input the data to be processed into the target interpolation model to obtain the synthesized intermediate frame image as the interpolation frame; Step 4 specifically includes: The data to be processed is input into the target interpolation model. The preprocessing module expands the input frame in the channel dimension to obtain the feature map of the input frame. The channel enhancement submodule of the motion enhancement module performs global average pooling on the feature map of the input frame and then expands the channel dimension through a two-dimensional convolutional layer. The corresponding channel weights are then taken by difference and compressed to the original number of channels through a two-dimensional convolutional layer. The obtained channel attention coefficients are then multiplied with the corresponding channel features. The spatial enhancement submodule of the motion enhancement module obtains spatial attention coefficients by calculating the similarity between channels at each location through a global average pooling layer and a two-dimensional convolutional layer, and then multiplies them with the features of the preceding and following frames. The ghost module performs regular 2D convolution on the feature map to generate a partial feature map. The remaining feature map is generated using a cheap operation and depthwise separable convolution to generate redundant feature maps. Finally, the two parts of the feature map are stacked in the channel dimension as the final output feature. After the output features are input into the generation submodule for convolution kernel prediction, offset prediction, and occlusion mask prediction, an adaptive deformation layer is used to sample and linearly combine the corresponding pixels according to the mapping relationship to generate an intermediate frame image as an interpolation frame.
2. The method according to claim 1, characterized in that... The preprocessing module includes three two-dimensional convolutional layers.
3. The method according to claim 2, characterized in that... The motion enhancement module includes a channel enhancement submodule and a spatial enhancement submodule. The channel enhancement submodule includes a pooling layer and two two-dimensional convolutional layers, and the spatial enhancement submodule includes a global average pooling layer and a two-dimensional convolutional layer.
4. The method according to claim 3, characterized in that... The generation submodule includes a convolution kernel submodule, an offset submodule, an occlusion mask submodule, and an adaptive deformation layer.
5. The method according to claim 4, characterized in that... Before step 2, the method further includes: Data augmentation of all initial triples is performed by randomly cropping, flipping, and swapping the order of preceding and following frames to obtain the corresponding image triples.
6. The method according to claim 5, characterized in that... The expression for the L1 loss function is: in, , These represent the model's output frame and reference frame, respectively. The expression for the perceptual loss function is: It is the feature extractor of the two-dimensional convolutional layer in the pre-trained initial interpolation model.