A video moire elimination method based on selective time domain fusion

Abstract

The application discloses a video moire removal method based on selective time domain fusion. The method inputs the features spliced after frame alignment module output into the selective time domain fusion module. After the spliced features are segmented by the time domain channel fusion module, the convolution layer and the activation function are used for processing and then fusion, the spatial feature adaptive module is input to refine the fusion features, the time domain information of adjacent frames is effectively fused, and the useful features of the fusion frame are adaptively explored. Meanwhile, the selective time domain fusion module effectively guides the learning of the parameters of the shallow adjacent frame alignment through the spatial exploration of the deep fusion features, and avoids the system instability caused by the excessive feature offset. In addition, the moire removal sub-network used in the method effectively removes the moire of each scale in the fusion frame by embedding the multi-scale attention module into each level of the encoder of the encoder-decoder network.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video moiré removal, and particularly relates to a video moiré removal method based on selective temporal fusion. BACKGROUND

[0002] Moiré is a kind of noise interference caused in the case of signal aliasing, which belongs to aliasing distortion. It is usually caused by the mismatch between the sampling frequency of the mobile phone and the spatial frequency of the display when the mobile phone captures the display screen image or video, and is manifested as the appearance of strange waves or color distortion in the image. The video image with moiré not only seriously affects people's visual experience, but also seriously hinders the technical development of downstream tasks of computer vision. Therefore, it is an important task to restore high-quality video images from the video affected by moiré.

[0003] In recent years, with the help of deep learning algorithms, many moiré removal techniques for images have emerged, but compared with images, the development of video moiré technology is relatively slow. After the moiré removal algorithm, the video moiré removal not only needs to ensure the high quality of the video image, but also needs to ensure the temporal consistency of the recovered video. The current video moiré removal method mainly has the following limitations: (1) Feature offset in video frames. The displacement or movement between consecutive frames in a video sequence due to camera motion, object motion or compression. The current method mainly relies on optical flow network assistance for alignment or deformable convolution to predict the offset, but the former needs more computational complexity, and the latter will cause the overall network to be unstable due to too large offset. (2) Unable to effectively use the temporal information of consecutive frames in the video. Due to the dynamic characteristics of the changing moiré, the temporal information of adjacent frames is particularly important on consecutive frames with little overall change in the background. The current method mainly uses direct channel splicing or simple convolution prediction weight weighting. (3) Unable to effectively remove moiré of different scales. Moiré is manifested in different forms at different scales, and in addition, the directionality of moiré is not fixed. The existing method brings it into the frequency domain for processing, which undoubtedly increases the computational burden. SUMMARY

[0004] The application proposes a video moiré removal method and system based on selective temporal fusion to solve the problems in the current video moiré removal field.

[0005] The video moiré removal method based on selective temporal fusion provided by the application comprises the following steps:

[0006] Step one, obtain a data set, which includes videos with moiré and corresponding videos without moiré, and pre-process the data set.

[0007] Step two, a moire removal model is constructed, the moire removal model includes two frame alignment modules, a selective temporal fusion module and a moire subnetwork. The two frame alignment modules are used to make up the feature offset between the input continuous three frames of images. The outputs of the two frame alignment modules are spliced and input into the selective temporal fusion module. The selective temporal fusion module includes a temporal channel fusion module and a spatial feature adaptive module. The temporal channel fusion module fuses the temporal information of the spliced image features through convolution, and the channel number remains unchanged. The fused features are divided into three features with equal channels, and each of the three divided frames of features passes through a layer of convolution and an activation function to predict a respective weight map, and the fused features are weighted to enhance the temporal information of the fused features. The output of the temporal channel fusion module is processed by the spatial feature adaptive module and input into the moire subnetwork; the moire subnetwork outputs moire removal intermediate frames of different scales.

[0008] Step three, using the data in the data set to train the moire removal model.

[0009] Step four, using the trained moire removal model to remove moire in the moire video.

[0010] As a preferred, in step two, the adjacent three frames of images input into the moire removal model are down-sampled by pixel unshuffle, and the input frames are down-sampled to three scales using bilinear interpolation. The adjacent three frames of images are divided into two groups of adjacent frames and input into two frame alignment modules respectively. The frame alignment module adopts a network structure of feature pyramid; at the third scale, a convolution layer is used to predict the feature offset of the adjacent frames, and a deformable convolution layer is used to combine the feature offset and the intermediate frame feature to obtain the third scale aligned feature; at the second scale, the feature offset at the third scale and the predicted feature offset at the second scale are spliced, and combined with the third scale aligned feature, input into a deformable convolution layer to obtain the second scale aligned feature; at the first scale, the feature offset at the second scale and the predicted feature offset at the first scale are spliced, and combined with the second scale aligned feature, input into a deformable convolution layer to obtain the first scale aligned feature.

[0011] As a preferred, in step two, the spatial feature adaptive module reduces the fused features output by the temporal channel fusion module to one quarter of the original size through a pixel-level adaptive filter, and inputs three series of residual blocks after splitting to refine the fused features.

[0012] As a preferred, in step two, the moire subnetwork adopts an encoder-decoder model structure.

[0013] As preferred, the demoising subnetwork comprises three encoder modules and three decoder modules, each of which has the same structure; the entire encoding stage and decoding stage are divided into three levels, corresponding to the semantic features of the fusion frame at three different depths. The output of each level encoder is used as the input of the next level encoder and the corresponding level decoder; the spatial resolution of the fusion frame feature output by each level encoder is half of the previous level. The output of each level decoder is used as the input of the previous level decoder; at the same time, the output of each level decoder is upsampled to the RGB color space through convolution and pixelshuffle, outputting demoised intermediate frames of different scales.

[0014] As preferred, the encoder module is encoded by a residual dilated convolution block and a multi-scale attention module, and the decoder module is decoded by a residual dilated convolution block and a multi-scale attention module. The fusion frame after the selective temporal fusion module is sent to the first level encoder module, and after passing through a residual dilated convolution block composed of different dilated coefficients, the feature information is sent to the multi-scale attention module. The multi-scale attention module uses bilinear interpolation and a multilayer perceptron to predict the vector of each scale to fuse the multi-scale information of the current feature.

[0015] As preferred, in step three, when training the model, use multi-scale perception loss and multi-scale L1 as the loss function of the model to train the model before 40-50 cycles, and then add a temporal consistency loss function to train the model.

[0016] As preferred, in step three, when training the model using the data set, the model is trained for 60 cycles.

[0017] As preferred, the hyperparameters of the multi-scale perception loss and the multi-scale L1 are 1 and 0.5, respectively.

[0018] As preferred, in step one, the pre-processing process is: sequentially divide the continuous three frames of images of the video into a group, and use the serial number of the middle frame image as the serial number of the image in the data set. Take the middle frame of the continuous three frames in each data group as the pre-recovery frame, and take the clean image corresponding to the pre-recovery frame as the label of the data group. According to the self-defined image size parameter, the image is randomly cropped to a fixed resolution.

[0019] The present application has the beneficial effects of:

[0020] 1、The frame alignment module outputs the features, which are spliced and then input into the selective time domain fusion module, the spliced features are segmented by the time domain channel fusion module, and then fused after being processed by the convolution layer and the activation function, and input into the spatial feature adaptive module to refine the fused features, effectively fusing the time domain information of adjacent frames, and adaptively exploring useful features of the fused frame; at the same time, the selective time domain fusion module effectively guides the learning of the parameters of the adjacent frame alignment in the shallow layer through the spatial exploration of the deep fusion features, avoiding system instability caused by excessive feature offset.

[0021] 2、The moire removal sub-network provided by the application effectively removes moires of various scales in the fused frame by embedding a multi-scale attention module into each level of the encoder of the encoder-decoder network; at the same time, the moire removal sub-network reserves an interface of the RGB color space image of each scale for each level of the encoder, facilitating the selection of different supervision modes.

[0022] 3、The application provides a video moire removal framework by using a deep learning method, the whole process is completed in an end-to-end manner, does not need to be trained in multiple stages, works in the spatial domain, has a simple structure, and can achieve excellent results on different moire datasets. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is the overall flowchart of the application.

[0024] Figure 2 is the video moire removal network structure diagram in the application.

[0025] Figure 3 is the image generation quality test result schematic diagram in the application.

[0026] Figure 4 is the video consistency test result schematic diagram in the application. DETAILED DESCRIPTION

[0027] The application will be further described below in combination with the drawings.

[0028] Embodiment 1

[0029] As shown in Figure 1 , a video moire removal method based on selective time domain fusion comprises the following steps:

[0030] Step 1, dataset processing

[0031] The VDmoire iPhonev2 dataset was selected as the dataset for model training and testing. The dataset includes 290 videos with moiré patterns and corresponding videos without moiré patterns, with each video divided into 60 image frames. Images within the same video in the dataset are divided into different data groups based on three consecutive frames, and the sequence number of the middle frame image is used as the sequence number of the processed data group in the dataset. The middle frame of three consecutive frames in each data group is used as the pre-recovery frame, and the clean image corresponding to the pre-recovery frame is used as the label of the data group. Images are randomly cropped to a fixed resolution based on the custom loaded image size parameter. The processed dataset is divided into a training set and a test set; the training set contains 247 videos and the test set contains 43 videos.

[0032] Step 2: Build a model to remove moiré patterns

[0033] like Figure 2 As shown in Figure 1, the moiré removal model includes two frame alignment modules, a selective time domain fusion module, and a moiré sub-network.

[0034] 2-1. Frame alignment module

[0035] The three adjacent frames input to the moiré removal model are downsampled to half the resolution using pixel unshuffle, and bilinear interpolation is used to downsample the input frames to three scales. The three adjacent frames are divided into two groups of adjacent frames and fed into the frame alignment module. The frame alignment module employs a feature pyramid network structure. At the third scale, a convolutional layer predicts feature offsets between adjacent frames, and a deformable convolutional layer combines these offsets with intermediate frame features to obtain third-scale aligned features. At the second scale, a convolutional layer predicts feature offsets between adjacent frames, and the third-scale feature offsets are concatenated with the predicted second-scale feature offsets. The combined third-scale aligned features are fed into a deformable convolutional layer to obtain second-scale aligned features. At the first scale, the second-scale feature offsets are concatenated with the predicted first-scale feature offsets, and the combined second-scale aligned features are fed into a deformable convolutional layer to obtain first-scale aligned features. The three frames output by the frame alignment module compensate for feature offsets between different frames.

[0036] 2-2. Selective Time Domain Fusion Module

[0037] After directly splicing the three processed frames, they are fed into a selective temporal fusion module, which fuses the temporal information between the three adjacent frames and explores the spatial features of the fused frame. The selective temporal fusion module consists of a temporal channel fusion module and a spatial feature adaptation module. Since the number of channels in a directly spliced ​​three-frame image is three times that of a single-frame feature, the temporal channel fusion module roughly fuses the temporal information of the spliced ​​image features through convolution, while maintaining the same number of channels. The fused features are then divided into three features using equal channels. These three features now represent a coarse temporal fusion of the three-frame features. Each of these features undergoes a layer of convolution and activation function to predict its own weight map, which is then point-wise multiplied with its own features. Finally, the weighted sum of the three features is weighted to enhance the temporal information of the fused feature. During model training, the weight map is adaptively learned to extract more effective information from the three frames. The spatial feature adaptation module uses a pixel-level adaptive filter to reduce the fused features output by the temporal channel fusion module to one-quarter their original size. After segmentation, the fused features are fed into three concatenated residual blocks to refine the fused features.

[0038] 2-3. Moore's law sub-network

[0039] The moiré sub-network removes moiré from the fused frames using multi-scale attention and multi-scale supervision. The moiré removal sub-network consists of three encoder modules and three decoder modules, each with the same architecture. The encoder module encodes the frame using a residual dilated convolution block and a multi-scale attention module, while the decoder module decodes the frame using a residual dilated convolution block and a multi-scale attention module. After the selective temporal fusion module, the fused frame is fed into the first-level encoder module. After passing through a residual dilated convolution block with different dilation coefficients, the feature information is fed into the multi-scale attention module. This module uses bilinear interpolation and a multi-layer perceptron to predict a vector at each scale to fuse the multi-scale information of the current feature. Different dilation coefficients introduce different receptive fields across the spatial dimensions of the input, capturing a wider range of contextual information. The residual structure allows for faster information propagation within the network. This helps address the vanishing and exploding gradient problems in deep networks, making them easier to train. Residual connections enable deeper networks while still maintaining good training performance. In the convolution block, the number of feature channels increases with the number of convolution layers in order to learn deeper semantic features. Finally, the number of channels is set to the number of input channels in the last convolution layer of the convolution block.

[0040] The whole encoding stage and decoding stage are divided into three levels, which correspond to the semantic features of the fusion frame at three different depths. Each depth of semantic feature will pass through a layer of multi-scale attention, making full use of the semantic information at each depth. The output of each level encoder is used as the input of the next level encoder and the corresponding level decoder. The spatial resolution of the fusion frame feature output by each level encoder is half of the previous level. The output of each level decoder is used as the input of the previous level decoder. At the same time, the output of each level decoder is upsampled to the RGB color space through convolution and pixel shuffle, outputting de-moired intermediate frames at different scales. A visualization processing module is added after the moire subnetwork, which can generate visual results of de-moired frames.

[0041] Step three, training the de-moiring model

[0042] The de-moiring model is trained for 60 cycles using the data in the training set. The first 50 cycles are trained using multi-scale perception loss and multi-scale L1, with hyperparameters of 1 and 0.5, respectively. The output results are compared with the preset labels, and the loss is calculated according to the set loss function. The parameters in the entire network are updated through backpropagation and iteration. In the last 10 cycles, the loss function adds a temporal consistency loss based on the previous one. At this time, the de-moiring model inputs the previous and next two frames of the intermediate frame, and the temporal consistency loss is used to supervise the time domain changes between the two frames through the set labels.

[0043] Step four, test the de-moiring model

[0044] As shown in Figure 3 , the test set data is input into the selective temporal fusion video de-moiring method and the existing de-moiring methods DMCNN, WDNet, ESDNet and VDmoire. LPIPS (Learning Perceptual Image Blocks Similarity), PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) are used to evaluate the quality of the images generated by different de-moiring methods. The smaller the LPIPS, the larger the PSNR and SSIM, and the closer the output image is to the set label. The selective temporal fusion video de-moiring method reaches 0.150, 25.756, and 0.868, respectively. Compared with the current advanced image or video de-moiring methods DMCNN, WDNet, ESDNet, and VDmoire, there is different degree of improvement.

[0045] As shown in Figure 4As shown, FVD (Frechet Video Distance), FSIM (Feature Similarity Index Measure) and FSIMc (Feature Similarity Index Measure with Color) are respectively used to evaluate the consistency of the video frames generated by different moire removal methods; the smaller the FVD, the larger the FSIM and FSIMc, the closer the consistency of the output video and the set video frame label; the video moire removal method based on selective time domain fusion respectively reaches 125.75, 0.974 and 0.973; compared with the current advanced image or video moire removal methods DMCNN, WDNet, ESDNet and VDmoire, there is different degree of improvement.

[0046] The specific embodiments described in the present application are only examples to illustrate the spirit of the present application. Those skilled in the art to which the present application belongs can make various modifications or supplements to the described specific embodiments or replace them with similar ways, but will not deviate from the spirit of the present application or exceed the scope defined by the appended claims.

Claims

1. A video moire reduction method based on selective temporal fusion, characterized in that: The method comprises the following steps: Step one, obtaining a data set, the data set comprising a video with moire and a corresponding video without moire, and preprocessing the data set; Step two, constructing a moire removal model, the moire removal model comprising two frame alignment modules, a selective temporal fusion module and a moire subnetwork; the two frame alignment modules are used to compensate for feature offset between inputted three continuous frames of images; the output of the two frame alignment modules is inputted into the selective temporal fusion module after being spliced; the selective temporal fusion module comprises a temporal channel fusion module and a spatial feature adaptive module; the temporal channel fusion module fuses temporal information of spliced image features through convolution, and the channel number remains unchanged; the fused features are equally divided into three features, and each of the three features is respectively predicted through a layer of convolution and an activation function to obtain a weight map, and the fused features are weighted to enhance the temporal information of the fused features; the output of the temporal channel fusion module is inputted into the moire subnetwork after being processed by the spatial feature adaptive module; the moire subnetwork outputs moire removal intermediate frames of different scales; Step three, training the moire removal model using data in the data set; Step four, removing moire in a video with moire using the trained moire removal model.

2. The video demozaing method based on selective temporal fusion according to claim 1, wherein: In step two, the spatial feature adaptive module reduces the fused features outputted by the temporal channel fusion module to one quarter of the original size through a pixel-level adaptive filter, and inputs the fused features into three residual blocks in series to refine the fused features.

3. The video demozaing method based on selective temporal fusion according to claim 1, wherein: In step two, the moire removal subnetwork adopts an encoder-decoder model structure.

4. The video demozaing method based on selective temporal fusion according to claim 3, wherein: The moire removal subnetwork comprises three encoder modules and three decoder modules, and the structure of each encoder module and decoder module is the same; the entire encoding stage and decoding stage are divided into three levels, which correspond to semantic features of the fused frame at three different depths; the output of each level encoder is inputted into a next level encoder and a corresponding level decoder; the spatial resolution of the fused frame feature outputted by each level encoder is half of that of the previous level; the output of each level decoder is inputted into a previous level decoder; meanwhile, the output of each level decoder is upsampled to an RGB color space through convolution and pixel shuffle, and moire removal intermediate frames of different scales are outputted.

5. The video demozaing method based on selective temporal fusion according to claim 4, characterized in that: The encoder module encodes through a residual dilation convolution block and a multi-scale attention module, and the decoder module decodes through a residual dilation convolution block and a multi-scale attention module; The fused frame after the selective temporal fusion module is inputted into the first level encoder module, and the feature information is inputted into the multi-scale attention module after being processed by a residual dilation convolution block composed of different dilation coefficients; the multi-scale attention module uses bilinear interpolation and a multi-layer perceptron to predict a vector of each scale to fuse multi-scale information of the current feature.

6. The video demozaing method based on selective temporal fusion according to claim 1, wherein: In step three, during the training of the model, a multi-scale perception loss and a multi-scale L1 are used as the loss function of the model to train the model before 40-50 cycles, and a temporal consistency loss function is added to train the model.

7. The video demozaing method based on selective temporal fusion according to claim 1, wherein: In step three, the model is trained for 60 cycles when training the model using the data set.

8. The video demozaing method based on selective temporal fusion according to claim 6, wherein: The hyperparameters of the multi-scale perception loss and the multi-scale L1 are 1 and 0.5, respectively.

9. The video demozaing method based on selective temporal fusion of claim 1, wherein: In step one, the pre-processing process is: sequentially dividing the continuous three frames of images of the video into a group, taking the serial number of the middle frame image as the serial number of the image in the data set; taking the middle frame of the continuous three frames in each data group as the pre-recovery frame, and taking the clean image corresponding to the pre-recovery frame as the label of the data group; and randomly cropping the image to a fixed resolution according to the self-defined image size parameter.

10. The video demozaing method based on selective temporal fusion according to claim 1, characterized in that: In step two, the adjacent three frames of images in the input Moiré removal model are down-sampled by pixel unshuffle, and the input frame is down-sampled to three scales using bilinear interpolation; the adjacent three frames of images are divided into two groups of adjacent frames and input into two frame alignment modules; the frame alignment module adopts a network structure of a feature pyramid; at the third scale, a convolution layer is used to predict the feature offset of the adjacent frames, and a deformable convolution layer is used to combine the feature offset and the middle frame feature to obtain the third scale aligned feature; At the second scale, the feature offset at the third scale and the predicted feature offset at the second scale are spliced, and combined with the third scale aligned feature, a deformable convolution layer is input to obtain the second scale aligned feature; At the first scale, the feature offset at the second scale and the predicted feature offset at the first scale are spliced, and combined with the second scale aligned feature, a deformable convolution layer is input to obtain the first scale aligned feature.

Images