Video moire removing method based on selective time domain fusion
Through the combination of selective time domain fusion module and moiré network, the inter-frame offset and scale problems in video demoiré technology are solved, and the efficient video demoiré effect is achieved, improving inter-frame alignment stability and de-frame removal Molar mass.
Patent Information
- Application Number
- CN202410901527.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-05
AI Technical Summary
The existing video demodeling technology has problems such as unstable inter-frame feature offset, inability to effectively utilize time domain information, and inability to remove molar patterns at each scale, resulting in high computational complexity and poor demodeling effect.
Using a video demolarization method based on selective time domain fusion, a model including frame alignment module, selective time domain fusion module and molarization subnet network is constructed, and a time domain channel fusion and spatial feature adaptive module are used to make up for inter-frame bias. and remove molar patterns of different scales through multi-scale attention module.
Effectively fusing inter-frame time domain information improves the stability of inter-frame alignment, can remove molar patterns on different scales, reduces the computational complexity, and achieves high-quality video demolar patterns.
Smart Images

Figure CN119027338A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video moiré removal, and in particular relates to a video moiré removal method based on selective time domain fusion. Background Art
[0002] Moiré is a type of noise interference caused by signal aliasing, which is an aliasing distortion. It is usually caused by the mismatch between the sampling frequency of the mobile phone and the spatial frequency of the display when the mobile phone shoots images or videos on the display screen, which manifests as strange ripples or color distortion in the image. Video images with moiré not only seriously affect people's visual experience, but also seriously hinder the technical development of downstream computer vision tasks. Therefore, restoring high-quality video images from videos affected by moiré is an important task.
[0003] In recent years, with the help of deep learning algorithms, many moiré removal techniques for images have emerged. However, compared with images, the development of video moiré technology is relatively slow. After the moiré removal algorithm, video moiré removal not only needs to ensure the high quality of the video image, but also needs to ensure that the restored video has the temporal consistency of the original video. The current video moiré removal methods have the following limitations: (1) Feature offset in the video frame. The displacement or movement between consecutive frames in the video sequence due to camera motion, object motion or compression. The current methods mainly rely on optical flow network-assisted alignment or deformable convolution to predict the offset, but the former requires greater computational complexity, and the latter will cause the overall network instability due to excessive offset. (2) It is impossible to effectively utilize the temporal information of consecutive frames in the video. Due to the dynamic characteristics of the ever-changing moiré, the temporal information of adjacent frames is particularly important for consecutive frames where the overall background does not change much. The current methods mainly use direct channel splicing or simple convolution prediction weight weighting. (3) It is impossible to effectively remove moiré of various scales. Moiré has different forms at different scales, and the directionality of moiré is also not fixed. Existing methods bring it into frequency domain processing, which undoubtedly increases the computational burden. Summary of the invention
[0004] The present invention aims at the problems existing in the current video de-moiré field and proposes a video de-moiré method and system based on selective time domain fusion.
[0005] The present invention provides a video moiré removal method based on selective time domain fusion, comprising the following steps:
[0006] Step 1: Obtain a data set, which includes videos with moiré patterns and corresponding videos without moiré patterns, and preprocess the data set.
[0007] Step 2: Construct a model for removing moiré. The model for removing moiré includes two frame alignment modules, a selective time domain fusion module, and a moiré sub-network. The two frame alignment modules are used to compensate for the feature offset between the three consecutive frames of the input image. The outputs of the two frame alignment modules are spliced and input into the selective time domain fusion module. The selective time domain fusion module includes a time domain channel fusion module and a spatial feature adaptation module. The time domain channel fusion module uses convolution to fuse the time domain information of the spliced image features. The number of channels remains unchanged. The fused features are divided into three features by equal channels. The three frame features after division are each predicted by a layer of convolution and activation function. The weight map is then weighted to enhance the time domain information of the fused features. The output of the time domain channel fusion module is processed by the spatial feature adaptation module and input into the moiré sub-network; the moiré sub-network outputs de-moiré intermediate frames of different scales.
[0008] Step 3: Use the data in the dataset to train the model for removing moiré patterns.
[0009] Step 4: Use the trained moiré removal model to remove moiré from the video.
[0010] Preferably, in the step 2, the three adjacent frames of images input into the moiré removal model are downsampled by pixelunshuffle, and the input frames are downsampled to three scales using bilinear interpolation. The three adjacent frames are divided into two groups of adjacent frames and input into two frame alignment modules respectively. The frame alignment module adopts a feature pyramid network structure; at the third scale, a convolution layer is used to predict the feature offset of adjacent frames, and a deformable convolution layer is used to combine the feature offset and the intermediate frame feature to obtain the third scale alignment feature; at the second scale, the feature offset on the third scale and the feature offset predicted on the second scale are spliced, and combined with the third scale alignment feature, the feature is input into the deformable convolution layer to obtain the second scale alignment feature; at the first scale, the feature offset on the second scale and the feature offset predicted on the first scale are spliced, and combined with the second scale alignment feature, the feature is input into the deformable convolution layer to obtain the first scale alignment feature.
[0011] Preferably, in the step 2, the spatial feature adaptation module reduces the fusion features output by the time domain channel fusion module to one quarter of the original size through a pixel-level adaptive filter, and after segmentation, inputs three series-connected residual blocks to refine the fusion features.
[0012] Preferably, in the step 2, the moiré removal sub-network adopts an encoder-decoder model structure.
[0013] Preferably, the de-moiré sub-network includes three encoder modules and three decoder modules, and each encoder module has the same structure as the decoder module; the entire encoding and decoding stages are divided into three levels, corresponding to the semantic features of the fused frame at three different depths. The output of each level of encoder serves as the input of the next level of encoder and the corresponding level of decoder; the spatial resolution of the fused frame features output by each level of encoder is half of that of the previous level. The output of each level of decoder serves as the input of the previous level of decoder; at the same time, the output of each level of decoder is upsampled to the RGB color space through convolution and pixel shuffle, and outputs de-moiré intermediate frames of different scales.
[0014] Preferably, the encoder module is encoded by a residual puffed convolution block and a multi-scale attention module, and the decoder module is decoded by a residual puffed convolution block and a multi-scale attention module. The fused frame after the selective time domain fusion module is sent to the first-level encoder module, and after passing through a residual puffed convolution block composed of different puffing coefficients, the feature information is sent to the multi-scale attention module, which uses bilinear difference and multi-layer perceptron to predict the vector of each scale to fuse the multi-scale information of the current feature.
[0015] Preferably, in the step three, when training the model, the multi-scale perceptual loss and multi-scale L1 are used as the loss function of the model to train the model 40 to 50 cycles ago, and then the time consistency loss function is added to train the model.
[0016] Preferably, in the step three, when using the data set to train the model, the model is trained for 60 cycles.
[0017] Preferably, the hyperparameters of the multi-scale perceptual loss and the multi-scale L1 are 1 and 0.5, respectively.
[0018] Preferably, in step 1, the preprocessing process is: divide three consecutive frames of the video into a group, and use the sequence number of the middle frame image as the sequence number of the group of images in the data set. Use the middle frame of three consecutive frames in each data group as the pre-recovery frame, and use the clean image corresponding to the pre-recovery frame as the label of the data group. Randomly crop the image to a fixed resolution according to the custom loaded image size parameter.
[0019] The present invention has the following beneficial effects:
[0020] 1. The present invention splices the features output by the frame alignment module and inputs them into the selective time domain fusion module. After segmenting the spliced features through the time domain channel fusion module, the spliced features are processed by the convolution layer and the activation function and then fused. The spliced features are input into the spatial feature adaptation module to refine the fusion features, effectively fuse the time domain information of adjacent frames, and adaptively explore the useful features of the fused frames. At the same time, the selective time domain fusion module effectively guides the learning of parameters for alignment of shallow adjacent frames through spatial exploration of deep fusion features, thereby avoiding system instability caused by excessive feature offset.
[0021] 2. The de-moiré sub-network provided by the present invention effectively removes moiré patterns of various scales in the fused frame by embedding the multi-scale attention module into each level of encoders of the codec network; at the same time, the de-moiré sub-network reserves interfaces of RGB color space images of various scales for encoders at each level, so as to facilitate the selection of different supervision modes.
[0022] 3. The present invention provides a video moiré removal framework using deep learning methods. The entire process is completed end-to-end without the need for multi-stage training. It works in the spatial domain and has a simple structure. It can achieve excellent results on different moiré data sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is the overall flow chart of the present invention.
[0024] Figure 2 This is a diagram of the network structure for video moiré removal in the present invention.
[0025] Figure 3 It is a schematic diagram of the image generation quality test results in the present invention.
[0026] Figure 4 It is a schematic diagram of the video consistency test results in the present invention. DETAILED DESCRIPTION
[0027] The present invention will be further described below in conjunction with the accompanying drawings.
[0028] Example 1
[0029] like Figure 1 As shown, a video moiré removal method based on selective time domain fusion includes the following steps:
[0030] Step 1: Dataset processing
[0031] The iPhonev2 dataset of VDmoire is selected as the dataset for model training and testing. The dataset includes 290 moiré videos and corresponding videos without moiré, and a single video is divided into 60 image frames. The images in the same video of the dataset are divided into different data groups in the form of three consecutive frames, and the sequence number of the middle frame image is used as the sequence number of the processed data group in the dataset. The middle frame of three consecutive frames in each data group is used as the pre-recovery frame, and the clean image corresponding to the pre-recovery frame is used as the label of the data group. The image is randomly cropped to a fixed resolution according to the custom loading image size parameter. The processed dataset is divided into a training set and a test set; there are 247 videos in the training set and 43 videos in the test set.
[0032] Step 2: Build a model to remove moiré
[0033] like Figure 2 As shown, the model for removing moiré patterns includes two frame alignment modules, a selective time domain fusion module, and a moiré pattern sub-network.
[0034] 2-1. Frame alignment module
[0035] The three adjacent frames of images in the input moiré removal model are downsampled to half the resolution size through pixel unshuffle, and the input frames are downsampled to three scales using bilinear interpolation. The three adjacent frames are divided into two groups of adjacent frames and input into the frame alignment module respectively. The frame alignment module adopts a feature pyramid network structure; at the third scale, the convolution layer is used to predict the feature offset of the adjacent frames, and the deformable convolution layer is used to combine the feature offset and the intermediate frame features to obtain the third scale alignment features; at the second scale, the convolution layer is used to predict the feature offset of the adjacent frames, the feature offset on the third scale is spliced with the feature offset predicted on the second scale, and the third scale alignment features are combined to input the deformable convolution layer to obtain the second scale alignment features; at the first scale, the feature offset on the second scale is spliced with the feature offset predicted on the first scale, and the second scale alignment features are combined to input the deformable convolution layer to obtain the first scale alignment features. The three frames of images output by the frame alignment module make up for the feature offset between different frames.
[0036] 2-2. Selective time domain fusion module
[0037] After the processed three frames are directly spliced, they are input into the selective time domain fusion module to fuse the time domain information between the three adjacent frames and explore the spatial features of the fused frame. The selective time domain fusion module includes the time domain channel fusion module and the spatial feature adaptive module. Since the number of channels of the directly spliced three frames is three times that of the single frame features, the time domain channel fusion module roughly fuses the time domain information of the spliced image features through convolution, and the number of channels remains unchanged; then the fused features are divided into three features by equal channels, and the three features at this time are three-frame features with rough time domain fusion information; the three frame features after division are each predicted by a layer of convolution and activation function, and point multiplication is performed with each feature, and finally the three frame features after operation are weighted to enhance the time domain information of the fused features. When training the model, the weight map will be adaptively learned to extract more effective information from the three frames. The spatial feature adaptive module reduces the fusion features output by the time domain channel fusion module to one-fourth of the original size through a pixel-level adaptive filter, and after segmentation, three series-connected residual blocks are input to refine the fusion features.
[0038] 2-3. Moore's Law Network
[0039] The moiré sub-network removes the moiré of the fused frame by multi-scale attention and multi-scale supervision. The moiré removal sub-network includes three encoder modules and three decoder modules. Each encoder module has the same structure as the decoder module. The encoder module is encoded by the residual puffed convolution block and the multi-scale attention module, and the decoder module is decoded by the residual puffed convolution block and the multi-scale attention module. The fused frame after the selective time domain fusion module is fed into the first-level encoder module. After passing through a residual puffed convolution block composed of different puffing coefficients, the feature information is fed into the multi-scale attention module. The multi-scale attention module uses bilinear difference and multi-layer perceptron to predict the vector of each scale to fuse the multi-scale information of the current feature. Different puffing coefficients can introduce different receptive fields in the spatial dimension of the input, thereby capturing a wider range of contextual information, and the residual structure allows information to propagate faster in the network. This helps to solve the gradient vanishing and gradient exploding problems in deep networks, making the network easier to train. Through residual connections, the network can be deeper while still maintaining good training results. In the convolution block, the number of feature channels increases with the number of convolution layers in order to learn deeper semantic features. Finally, the number of channels is set to the number of input channels in the last convolution layer of the convolution block.
[0040] The entire encoding and decoding stages are divided into three levels, corresponding to the semantic features of the fused frame at three different depths. The semantic features of each depth will go through a layer of multi-scale attention, making full use of the semantic information at each depth. The output of each level of encoder serves as the input of the next level encoder and the corresponding level decoder; the spatial resolution of the fused frame features output by each level encoder is half of that of the previous level. The output of each level of decoder serves as the input of the previous level decoder; at the same time, the output of each level of decoder is upsampled to the RGB color space through convolution and pixel shuffle, and outputs de-moiré intermediate frames of different scales. Adding a visualization processing module after the moiré subnetwork can generate de-moiré visualization results.
[0041] Step 3: Train a model to remove moiré patterns
[0042] The moiré removal model was trained for 60 cycles using the data in the training set. In the first 50 cycles, multi-scale perceptual loss and multi-scale L1 joint training were used, with hyperparameters of 1 and 0.5 respectively. The output results were compared with the preset labels, and the loss was calculated according to the set loss function. Backpropagation was performed, and the parameters in the entire network were iteratively updated. In the next 10 cycles, the loss function added a temporal consistency loss on the basis of the previous one. At this time, the moiré removal model input the two frames before and after the middle frame respectively, and the temporal consistency loss was used to supervise the temporal changes between the two frames through the set label output.
[0043] Step 4: Test the model after removing moiré
[0044] like Figure 3 As shown in the figure, the video demoirification method based on selective temporal fusion, as well as the existing demoirification methods DMCNN, WDNet, ESDNet and VDmoire are input into the test set data. LPIPS (learning-perceptual image block similarity), PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) are used to evaluate the quality of images generated by different demoirification methods; the smaller the LPIPS, the larger the PSNR and SSIM, and the closer the output image is to the set label; the video demoirification method based on selective temporal fusion reaches 0.150, 25.756, and 0.868 respectively; compared with the current advanced image or video demoirification methods DMCNN, WDNet, ESDNet, and VDmoire, they have been improved to varying degrees.
[0045] like Figure 4As shown in the figure, FVD (Fréchet Video Distance), FSIM (Feature Similarity Index Measure) and FSIMc (Feature Similarity Index Measure with Color) are used to evaluate the consistency of video frames generated by different de-moiré methods; the smaller the FVD, the larger the FSIM and FSIMc, and the closer the consistency between the output video and the set video frame label; the video de-moiré method based on selective time domain fusion reached 125.75, 0.974, and 0.973 respectively; compared with the current advanced image or video de-moiré methods DMCNN, WDNet, ESDNet, and VDmoire, they have all been improved to varying degrees.
[0046] The specific embodiments described in the present invention are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.
Claims
1. A video moiré removal method based on selective temporal fusion, characterized in that: The following steps are involved: Step 1: Obtain a data set, which includes videos with moiré patterns and corresponding videos without moiré patterns, and preprocess the data set; Step 2: Construct a model for removing moiré. The model for removing moiré includes two frame alignment modules, a selective time domain fusion module and a moiré sub-network. The two frame alignment modules are used to compensate for the feature offset between the three consecutive frames of input images. The outputs of the two frame alignment modules are spliced and input into the selective time domain fusion module. The selective time domain fusion module includes a time domain channel fusion module and a spatial feature adaptation module. The time domain channel fusion module uses convolution to fuse the time domain information of the spliced image features, and the number of channels remains unchanged. The fused features are divided into three features by equal channels. The three frame features after division are each predicted by a layer of convolution and activation function. The weighted images are then weighted to enhance the time domain information of the fused features. The output of the time domain channel fusion module is processed by the spatial feature adaptation module and input into the moiré sub-network. The moiré sub-network outputs de-moiré intermediate frames of different scales. Step 3: Use the data in the dataset to train the model for removing moiré patterns; Step 4: Use the trained moiré removal model to remove moiré from the video.
2. The video moiré removal method based on selective time domain fusion according to claim 1, characterized in that: In the step 2, the spatial feature adaptive module reduces the fusion features output by the time domain channel fusion module to one-fourth of the original size through a pixel-level adaptive filter, and after segmentation, inputs three series-connected residual blocks to refine the fusion features.
3. The video moiré removal method based on selective time domain fusion according to claim 1, characterized in that: In the step 2, the de-moiré sub-network adopts an encoder-decoder model structure.
4. The video moiré removal method based on selective time domain fusion according to claim 3, characterized in that: The de-moiré sub-network includes three encoder modules and three decoder modules, and the structures of each encoder module and decoder module are the same; the entire encoding stage and decoding stage are divided into three levels, corresponding to the semantic features of the fused frame at three different depths respectively; the output of each level of encoder is used as the input of the next level encoder and the corresponding level decoder; the spatial resolution of the fused frame features output by each level of encoder is half of that of the previous level; the output of each level of decoder is used as the input of the previous level decoder; at the same time, the output of each level of decoder is upsampled to the RGB color space through convolution and pixel shuffle, and the de-moiré intermediate frames of different scales are output.
5. The video de-moiré method based on selective time-domain fusion according to claim 4, characterized in that: The encoder module is encoded by a residual puffed convolution block and a multi-scale attention module, and the decoder module is decoded by a residual puffed convolution block and a multi-scale attention module; The fused frame after the selective time domain fusion module is sent to the first-level encoder module. After passing through a residual dilation convolution block composed of different dilation coefficients, the feature information is sent to the multi-scale attention module. The multi-scale attention module uses bilinear difference and multi-layer perceptron to predict the vector of each scale to fuse the multi-scale information of the current feature.
6. The video moiré removal method based on selective time domain fusion according to claim 1, characterized in that: In the step three, when training the model, the multi-scale perceptual loss and multi-scale L1 are used as the loss function of the model for training the model before 40 to 50 cycles, and then the time consistency loss function is added to train the model.
7. The video moiré removal method based on selective time domain fusion according to claim 1, characterized in that: In the step 3, when using the data set to train the model, the model is trained for 60 cycles.
8. The video de-moiré method based on selective time-domain fusion according to claim 6, characterized in that: The hyperparameters of multi-scale perceptual loss and multi-scale L1 are 1 and 0.5 respectively.
9. The video moiré removal method based on selective time domain fusion according to claim 1, characterized in that: In the step 1, the preprocessing process is: divide the three consecutive frames of the video into a group in turn, and use the serial number of the middle frame image as the serial number of the group of images in the data set; use the middle frame of three consecutive frames in each data group as the pre-recovery frame, and use the clean image corresponding to the pre-recovery frame as the label of the data group; randomly crop the image to a fixed resolution according to the custom loaded image size parameter.
10. The video moiré removal method based on selective time domain fusion according to claim 1, characterized in that: In the step 2, three adjacent frames of images input into the moiré removal model are downsampled by pixel unshuffle, and bilinear interpolation is used to downsample the input frames to three scales; the three adjacent frames are divided into two groups of adjacent frames and input into two frame alignment modules respectively; the frame alignment module adopts a feature pyramid network structure; at the third scale, a convolution layer is used to predict the feature offset of adjacent frames, and a deformable convolution layer is used to combine the feature offset and the intermediate frame feature to obtain the third scale alignment feature; At the second scale, the feature offset at the third scale and the feature offset predicted at the second scale are concatenated, and combined with the features aligned at the third scale, they are input into the deformable convolution layer to obtain the features aligned at the second scale; At the first scale, the feature offset at the second scale and the feature offset predicted at the first scale are concatenated, and combined with the features aligned at the second scale, they are input into the deformable convolution layer to obtain the features aligned at the first scale.
Citation Information
Patent Citations
Image denoising method and device, electronic equipment and storage medium
CN113240607A
Video moire removing method based on linear sparse attention Transformer
CN114881888A
Raw domain image and video moire removing method based on channel and spatial modulation
CN117333399A
Training method for deep residual network for removing a moire pattern of two-dimensional code
WO2021134874A1
Cited By
Ultrahigh-definition image moire removing method based on pyramid learnable band-pass filter
CN120070260A
High-resolution image moire removing method, system, equipment and medium
CN120259115A
