A video denoising and demosaicking method and system based on two-path space-time enhancement

Abstract

A kind of video denoising demosaicing method based on two-way space-time enhancement, comprising the following steps: according to the definition and acquisition method of video denoising demosaicing, collect and process the Bayer format RAW video data with high noise;Construct a space-time feature extraction branch, extract the global space-time dynamic characteristics of video sequence;Construct RAW domain feature scanning branch, output RAW domain feature representation;Build two-way space-time enhanced video denoising demosaicing model, respectively in noise removal module and final reconstruction stage with L1 loss constraint, optimize reconstruction quality and perceptual quality;Save training parameters, generate reconstruction results according to input data, complete inference and index evaluation.The present application extracts the information between video frames and RAW domain information through two-way space-time enhancement model, combines frequency adaptive dilated convolution and channel attention mechanism, improves the restoration ability of high-frequency details, and at the same time suppresses noise interference, realizes higher reconstruction accuracy in denoising demosaicing task.

Description

Technical Field

[0001] This invention relates to a video denoising and de-mosaic method and system based on dual-channel spatiotemporal enhancement, belonging to the fields of computer vision and deep learning technology. Background Technology

[0002] Joint video denoising and desacrifice is a core task in computer vision, aiming to remove noise signals from RAW videos and obtain high-quality RGB reconstructed videos. This technology plays a crucial role in many downstream tasks such as object detection and trajectory tracking. With the rapid development of deep learning methods, learning-based denoising and desacrifice techniques have achieved significant performance improvements. However, existing research mostly focuses on single-image denoising and desacrifice or burst image denoising and desacrifice, with research on video scenarios still far from sufficient.

[0003] With the continuous development of joint denoising and de-mosaic techniques, some researchers have attempted to introduce perceptual loss and adversarial loss to improve the quality of reconstruction results. While these methods improve reconstruction effectiveness, they exacerbate inconsistencies between reconstructed video frames. To align video frames, some methods introduce recurrent networks and use optical flow to compensate for inter-frame motion; however, optical flow estimation suffers from accumulated errors, and its alignment tends to be smooth, resulting in poor performance in video restoration. Furthermore, existing methods treat all input frames equally, neglecting information mining of the target frames to be reconstructed. Additionally, during denoising, noise is typically concentrated in high-frequency regions; directly suppressing high frequencies easily leads to detail loss, while retaining high frequencies introduces more noise interference. Some researchers have improved the visual quality of reconstruction results by introducing data temporal consistency loss and relational perceptual consistency loss; however, existing methods still cannot effectively balance denoising capability and reconstruction quality when dealing with high-frequency textures and complex motion. Simultaneously, due to the difficulty and high cost of collecting real-world paired video data, datasets based on additive white Gaussian noise synthesis are widely used, limiting the model's generalization ability and performance in real-world scenes.

[0004] Therefore, the key to solving the above problems lies in how to construct an efficient and flexible joint video denoising and de-mosaic method. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies by creatively proposing a video denoising and de-mosaic method and system based on dual-channel spatiotemporal enhancement. This invention not only effectively utilizes spatiotemporal related information between adjacent frames but also focuses on mining information from the frame to be reconstructed itself, thereby further improving the quality of the reconstruction results.

[0006] To achieve the above objectives, the present invention employs the following technical solutions.

[0007] A video denoising and de-mosaic method based on dual-channel spatiotemporal enhancement includes the following steps:

[0008] Step 1: Based on the definition and acquisition method of video denoising and de-mosaicing, collect and process high-noise Bayer format RAW video data to provide basic data for model training. The RAW video to be input is segmented frame by frame to obtain a continuous multi-frame video sequence. The video frames are then cropped and processed into TIFF format. This preserves the high dynamic range characteristics of the RAW domain of the input data while reducing unnecessary information, improving data usability, and reducing storage costs.

[0009] Step 2: Construct a spatiotemporal feature extraction branch. Using a spatiotemporal coding mechanism and an attention module, construct a spatiotemporal feature extraction module, and use transposed convolution to sample the feature map to the target resolution to extract the global spatiotemporal dynamic features of the video sequence.

[0010] Step 2.1: Based on the input video data, generate spatiotemporal feature codes, and combine 3D convolution and spatial location coding with temporal location coding embedding to integrate them into a unified spatiotemporal representation, thereby further improving the model's ability to capture dynamic spatiotemporal information.

[0011] Step 2.2: A multi-head attention mechanism is used to capture long-range spatial dependencies via an attention network. Simultaneously, transposed convolutions are introduced to sample spatiotemporal feature representations down to the target resolution. Finally, 3D convolutions are used to model the spatiotemporal feature encodings, providing accurate and efficient feature representations for joint video denoising and de-mosaic tasks.

[0012] Step 3: Construct a RAW domain feature scanning branch, and use Retinex decomposition, noise removal module, RAW domain scanning mechanism and frequency domain channel attention module to refine the model of the target frame and extract rich RAW domain features of the frame to be reconstructed.

[0013] Step 3.1: Apply Retinex theory to the reconstructed frame to decompose it into illumination and reflection components, effectively reducing interference caused by illumination changes, highlighting the detailed features of the reflectivity component, and providing a cleaner and more stable feature representation for denoising and de-mosaic.

[0014] Step 3.2: Multiply the illumination component element-wise with the input frame, and then perform denoising operations together with the reflection component through the domain adaptive fusion module and the simple denoising module. Repeat this operation using a Unet-like structure to obtain the RAW domain denoising result of the target frame.

[0015] Step 3.3: The RAWMamba module is introduced to perform multi-directional scanning of the denoised target frame, extracting multi-scale features from the target frame and aggregating these features to generate a RAW domain feature map. This operation is repeated using a Unet-like structure to obtain the feature representation map of the target frame. Specifically, the scanning results from the RAWMamba module and the denoising results from the corresponding level in Step 3.2 are adaptively fused through domain integration, thereby improving the reconstruction effect while removing noise.

[0016] Step 3.4: Combine frequency-adaptive extended convolution and channel attention to form a frequency-domain channel attention module. The receptive field is dynamically adjusted according to the frequency distribution of the input features, and each feature is processed separately. Finally, the generated channel weights are multiplied with the input features channel by channel to achieve dynamic weighted fusion and to balance the modeling of high-frequency and low-frequency information in the feature map.

[0017] Step 3.5: Output a high-quality, denoised, and optimized RAW domain feature map to provide a strong feature foundation for the subsequent final reconstruction.

[0018] Step 4: Construct a dual-path spatiotemporal augmented video denoising and de-mosaic model. The spatiotemporal feature representations output from the spatiotemporal feature extraction branch and the RAW domain feature representations output from the RAW domain feature scanning branch are concatenated along the channel dimension, and the final reconstruction result is output through the constructed decoder. Training is performed using real-world noisy video and paired noise-free video data. During the denoising stage of the RAW domain feature scanning branch and the final reconstruction result output stage, L1 loss constraints are applied to the output results using the RAW ground truth and RAW domain RGB ground truth values, respectively, to improve reconstruction and perceptual quality.

[0019] Step 5: Save the optimized network parameters, generate denoising and de-mosaic reconstruction results based on the input high-noise video sequence, and combine quality evaluation indicators (such as peak signal-to-noise ratio, structural similarity index, etc.) to infer and evaluate the model performance.

[0020] Based on the above method, the present invention further proposes a video denoising and de-mosaicing system based on dual-channel spatiotemporal enhancement, including a noisy RAW video data processing subsystem, a video denoising and de-mosaicing subsystem, and a supervised optimization and result evaluation subsystem.

[0021] The connections between the above components are as follows:

[0022] The output of the noisy RAW video data processing subsystem is connected to the input of the video denoising and de-mosaicing subsystem, and the output of the video denoising and de-mosaicing subsystem is connected to the input of the supervised optimization and result evaluation subsystem.

[0023] Beneficial effects

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1. This invention designs a video denoising and de-mosaicing method based on dual-channel spatiotemporal enhancement. It can achieve higher reconstruction accuracy in denoising and de-mosaicing tasks by using spatiotemporal feature extraction branch and RAW domain feature scanning branch, while taking advantage of the rich dynamic information between video frames and the high bit depth and large information content of RAW data.

[0026] 2. In the RAW domain feature scanning branch, this invention designs a frequency domain channel attention module, which combines frequency adaptive dilated convolution and channel attention mechanism. The model can perform differentiated processing on high-frequency and low-frequency regions according to the frequency distribution of input features, which significantly improves the ability to restore high-frequency details, while suppressing noise interference, thereby improving the model's performance in complex modal scenarios. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention.

[0028] Figure 2 This is a schematic diagram of the system composition of the present invention. Detailed Implementation

[0029] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below in conjunction with the accompanying drawings and examples.

[0030] Example

[0031] like Figure 1 As shown, a stereo depth estimation method based on cross-modal data includes the following steps:

[0032] Step 1: Based on the definition and acquisition method of video denoising and de-mosaicing, collect and process Bayer format RAW video data with high noise.

[0033] Specifically, the RAW video to be input is segmented frame by frame to obtain a continuous multi-frame video sequence. The video frames are then cropped and processed into TIFF format. This process maintains the high dynamic range characteristics of the RAW domain of the data to be input while reducing unnecessary information, improving data usability, and reducing storage costs.

[0034] Step 2: Construct a spatiotemporal feature extraction branch to extract global spatiotemporal dynamic features of the video sequence.

[0035] Specifically, a spatiotemporal feature extraction module is constructed using a spatiotemporal coding mechanism and an attention module, and the feature map is sampled to the target resolution using transposed convolution. This branch extracts the global spatiotemporal dynamic features of the video sequence by scanning consecutive input video frames.

[0036] Step 2.1: Generate spatiotemporal feature codes based on the input video data.

[0037] Specifically, the spatiotemporal feature extraction module first concatenates the three input frames along the time dimension, then segments the input video into fixed-size spatiotemporal blocks using 3D convolution, and downsamples the data. This step is defined as follows:

[0038]

[0039] in, It is a spatiotemporal feature block obtained through convolution. The input three consecutive frames are stitched together along the time dimension.

[0040] Furthermore, we inject spatial and temporal location codes into the spatiotemporal features and integrate them into a unified spatiotemporal representation, thereby further enhancing the model's ability to capture dynamic spatiotemporal information. The specific formula is as follows:

[0041]

[0042] in, and These are the encoded time and spatial location embedded information, respectively.

[0043] In this way, the generated spatiotemporal feature codes further enhance the model's ability to express spatiotemporal information.

[0044] Step 2.2: Based on the attention mechanism network and transposed convolution, perform feature modeling for spatiotemporal feature encoding.

[0045] Step 2.1 yields the spatiotemporal feature encoding of the input video sequence. To better mine the spatiotemporal feature information of the video sequence, feature modeling is required on the spatiotemporal feature encoding to obtain the spatiotemporal feature representation of the input sequence.

[0046] Specifically, in the feature modeling stage, an attention encoding network is first constructed to capture long-range spatial dependencies through a multi-head attention mechanism. The specific formula can be expressed as:

[0047]

[0048] in, It is a spatiotemporal feature representation obtained through attention encoding network processing.

[0049] Furthermore, transposed convolution is used to sample spatiotemporal feature representations down to the target resolution. Simultaneously, 3D convolution techniques are introduced to enhance the modeling capability of local spatiotemporal features. This design effectively preserves the global spatiotemporal feature modeling capability while reducing computational complexity, thus providing accurate and efficient feature representations for joint video denoising and de-mosaic tasks.

[0050] Step 3: Construct RAW domain feature scanning branches to mine information from the frame to be reconstructed.

[0051] Specifically, Retinex decomposition, noise removal module, RAW domain scanning mechanism, and frequency domain channel attention module are used to refine the model of the target frame and extract rich RAW domain features of the frame to be reconstructed, providing more accurate guidance for joint video denoising and de-mosaic tasks.

[0052] Step 3.1: Retinex Theoretical Decomposition

[0053] Specifically, for the 3 input frames: The RAW domain feature scanning branch only focuses on the reconstructed target frame. First, regarding Decomposition using Retinex theory can be expressed as follows:

[0054]

[0055] in, These represent the illumination component and reflection component obtained from the decomposition, respectively. This operation effectively reduces interference caused by changes in illumination, highlights the detailed features of the reflectivity component, and provides a cleaner and more stable feature representation for denoising and depigmentation.

[0056] Step 3.2: Noise Removal Module

[0057] The illumination component is multiplied element-wise with the input frame, and then combined with the reflection component for denoising via a domain adaptive fusion module and a simple denoising module. The specific formula can be expressed as follows:

[0058]

[0059] in, These represent the illumination component and the reflection component obtained from the decomposition, respectively. For element-wise multiplication, For noise reduction modules, DAF For domain adaptive fusion module, Input for the module, This is the module output. This operation is repeated using a Unet-like structure to obtain the RAW domain denoising result of the target frame.

[0060] Step 3.3: RAW domain multi-directional scanning mechanism.

[0061] To further capture the high dynamic range characteristics and local detail information of the RAW domain, the RAWMamba module is introduced for the target frame. The denoised results are then scanned in multiple directions. The core function of the RAWMamba module is to extract multi-scale features from the target frame through multi-directional perceptual operations in the RAW domain. The specific steps are as follows:

[0062] Step 3.3.1: Multi-directional scanning. The RAWMamba module uses an eight-directional scanning strategy, specifically from left to right, from top to bottom, from top left to bottom right, from top right to bottom left, and their reverse directions, effectively covering all potential feature directions in the target frame. By performing pixel-by-pixel fine-grained scanning of the target frame's RAW data, feature changes in different directions can be captured, especially edge and texture information.

[0063] Step 3.3.2: Feature Aggregation. The results of multi-directional scanning are weighted and fused to generate a RAW domain feature map.

[0064] Furthermore, this operation is repeated using a Unet-like structure to obtain the feature representation map of the target frame. In particular, the scanning results of the RAWMamba module and the denoising results of the corresponding level in step 3.2 are fused through domain adaptive fusion, thereby improving the reconstruction effect while removing noise.

[0065] Step 3.4: Optimize the frequency domain channel attention module.

[0066] When modeling RAW domain features, a frequency domain channel attention module is introduced on the basis of the multi-directional scanning mechanism. This is the core component of the domain adaptive fusion module. It combines frequency adaptive dilated convolution and channel attention to further perform frequency domain modeling and dynamic optimization of the feature map, so as to balance the preservation of high-frequency details and the suppression of low-frequency background.

[0067] Specifically, this module uses frequency-adaptive dilated convolution to dynamically adjust the receptive field based on the frequency distribution of the input features and processes them separately. In high-frequency regions, higher channel weights are assigned to enhance detail representation. In low-frequency regions, global contextual information is captured through the expanded receptive field. Finally, the generated channel weights are multiplied channel-by-channel with the input features to achieve dynamic weighted fusion and to balance the modeling of high-frequency and low-frequency information in the feature map. The specific formula can be expressed as:

[0068]

[0069] Among them, Pool It is a global average pooling operation. , These are the weight matrices of two frequency-adaptive dilated convolutional fully connected layers, ReLU. It is a corrected linear unit. This represents the Sigmoid activation function. It is a module input. This is the module output.

[0070] Step 3.5: Output RAW domain feature representation.

[0071] The RAW domain feature scanning branch is encoded using a U-net-like structure. After Retinex theoretical decomposition, multi-directional scanning of the RAWMamba module, and frequency domain optimization of the FDCA module, a high-quality, denoised and optimized RAW domain feature map is finally generated, providing a strong feature foundation for subsequent final reconstruction.

[0072] Step 4: Construct a dual-channel spatiotemporal augmentation video denoising and de-mosaic model, trained using real-world noisy video data and paired noise-free video data.

[0073] Specifically, a dual-path spatiotemporal augmentation video denoising and de-mosaic model is constructed. The spatiotemporal feature representations output from the spatiotemporal feature extraction branch and the RAW domain feature representations output from the RAW domain feature scanning branch are concatenated along the channel dimension, and the final reconstruction result is output through a built decoder. The model is trained using real-world noisy video data and paired noise-free video data. The specific training optimization includes two parts: reconstruction quality optimization (step 4.1) and perceptual quality optimization (step 4.2).

[0074] Step 4.1: Reconstruction Quality Optimization

[0075] RAW domain data retains the original details of the image without any image signal processing. To better remove noise, an L1 loss function is used to constrain the output of the denoising stage and the ground truth value in the RAW domain during the RAW domain feature scan branch denoising stage, thereby improving the reconstruction quality.

[0076] Step 4.2: Perceived Quality Optimization

[0077] The quality of the final reconstruction result directly reflects the performance of the model. To ensure the perceptual quality of the final reconstruction result, L1 loss is used in the final output stage to constrain the output of the dual-path spatiotemporal denoising and desacrifice model and the ground truth value of RAW domain RGB.

[0078] Step 5: Save the training parameters, generate denoising and de-mosaic reconstruction results based on the input high-noise video sequence, and complete the inference and index evaluation.

[0079] Specifically, in order to objectively evaluate the effect of the generated reconstructed frames, objective evaluation metrics can be generated using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index).

[0080] This application presents a video denoising and de-mosaicing method based on dual-path spatiotemporal enhancement, utilizing a spatiotemporal feature extraction branch and a RAW domain feature scanning branch to fully exploit the rich dynamic information between video frames and the abundant information in RAW data. By combining frequency-adaptive dilated convolution and channel attention mechanisms, the method significantly improves the restoration of high-frequency details while suppressing noise interference, thereby enhancing the model's performance in complex modal scenes. A supervised learning strategy is employed to optimize network parameters through reconstruction quality optimization constraints and perceptual quality optimization constraints. Training parameters are saved, and denoising and de-mosaicing reconstruction results are generated based on the input high-noise video sequence, completing inference and performance evaluation. This addresses the problems of existing technologies that treat all input frames equally, making it difficult to balance reconstruction quality and inter-frame consistency, and the inability of existing technologies to effectively balance denoising capability and reconstruction effect when handling high-frequency textures and complex motions. This method effectively improves the robustness and reconstruction effect of video denoising and de-mosaicing models in high-noise and complex scenes.

[0081] Figure 2 A schematic diagram of a video denoising and de-mosaicing system based on dual-channel spatiotemporal enhancement provided in this application embodiment includes a noisy RAW video data processing subsystem M10, a video denoising and de-mosaicing subsystem M20, and a supervised optimization and result evaluation subsystem M30:

[0082] Among them, the noisy RAW video data processing subsystem M10 is used to segment the RAW video to be input into frames according to the definition and acquisition method of video denoising and de-mosaicing, to obtain a continuous multi-frame video sequence, and to crop and process the video frames into TIFF format. While maintaining the high dynamic range characteristics of the RAW domain of the data to be input, it reduces unnecessary information, improves the usability of the data and reduces the storage cost.

[0083] The M20 video denoising and de-mosaicing subsystem comprises a spatiotemporal feature extraction branch and a RAW domain feature scanning branch. The spatiotemporal feature extraction branch generates spatiotemporal feature codes based on the input video data. Using an attention mechanism network and transposed convolution, it models the spatiotemporal feature codes to extract global spatiotemporal dynamic features of the video sequence. The RAW domain feature scanning branch outputs RAW domain feature representations through Retinex decomposition, a noise removal module, a multi-directional RAW domain scanning mechanism, and a frequency domain channel attention module, mining information from the frames to be reconstructed. The dual-channel spatiotemporally enhanced video denoising and de-mosaicing model concatenates the spatiotemporal feature representations and RAW domain feature representations along the channel dimension and outputs the reconstruction result through a decoding and reconstruction module.

[0084] The supervised optimization and result evaluation subsystem M30 is used to construct a loss function to optimize the training of the aforementioned network, further save the trained network parameters, and generate the final reconstruction result. The generated reconstruction result is objectively evaluated using built-in evaluation metrics such as PSNR and SSIM.

[0085] The connections between the above-mentioned components are as follows:

[0086] The output of the noisy RAW video data processing subsystem is connected to the input of the video denoising and de-mosaicing subsystem, and the output of the video denoising and de-mosaicing subsystem is connected to the input of the supervised optimization and result evaluation subsystem. The explanations and descriptions of the video denoising and de-mosaicing method based on dual-channel spatiotemporal enhancement in the foregoing embodiments also apply to the video denoising and de-mosaicing system based on dual-channel spatiotemporal enhancement in this embodiment, and will not be repeated here.

[0087] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video denoising and de-mosaic method based on dual-channel spatiotemporal enhancement, characterized in that, Includes the following steps: Step 1: Based on the definition and acquisition method of video denoising and de-mosaicing, collect and process Bayer format RAW video data with high noise to provide basic data for model training; Step 2: Construct a spatiotemporal feature extraction branch. Using a spatiotemporal coding mechanism and an attention module, construct a spatiotemporal feature extraction module, and use transposed convolution to sample the feature map to the target resolution to extract the global spatiotemporal dynamic features of the video sequence. Step 3: Construct a RAW domain feature scanning branch, and use Retinex decomposition, noise removal module, RAW domain scanning mechanism and frequency domain channel attention module to refine the model of the target frame and extract rich RAW domain features of the frame to be reconstructed. Step 4: Construct a dual-channel spatiotemporal enhancement video denoising and de-mosaic model. The spatiotemporal feature representations output from the spatiotemporal feature extraction branch and the RAW domain feature representations output from the RAW domain feature scanning branch are concatenated along the channel dimension, and the final reconstruction result is output through the constructed decoder. Step 5: Save the optimized network parameters, generate denoising and de-mosaic reconstruction results based on the input high-noise video sequence, and combine quality evaluation indicators (such as peak signal-to-noise ratio, structural similarity index, etc.) to infer and evaluate the model performance.

2. The method as described in claim 1, characterized in that, The spatiotemporal feature extraction branch in step 2 includes the following process: Step 2.1: Based on the input video data, after generating spatiotemporal features, spatial and temporal location codes are injected and further integrated into a unified spatiotemporal representation. Step 2.2: Based on an attention mechanism network and transposed convolution, a multi-head attention mechanism is used to capture long-range spatial dependencies, and transposed convolution is further used to sample the spatiotemporal feature representation to the target resolution. Simultaneously, 3D convolution technology is introduced to enhance the modeling capability of local spatiotemporal features.

3. The method as described in claim 1, characterized in that, The construction of the RAW domain feature scanning branch in step 3 includes the following process: Step 3.1: For the input video sequence, apply Retinex theory to decompose the reconstructed target frame. Step 3.2: Multiply the illumination component element-wise with the input frame, and then perform denoising operations together with the reflection component through a domain adaptive fusion module and a simple denoising module. Step 3.3: The RAWMamba module processes the target frame... The denoising results are then subjected to multi-directional scanning and feature aggregation. Step 3.4: A frequency domain channel attention module is introduced as a core component in the domain adaptive fusion module. This module combines frequency adaptive dilated convolution and channel attention to further model and dynamically optimize the feature map in the frequency domain, balancing high-frequency detail preservation with low-frequency background suppression. Step 3.5: High-quality, denoised, and optimized RAW domain feature maps are output, providing a strong feature foundation for subsequent final reconstruction.

4. The method as described in claim 3, characterized in that, The specific formula for the frequency domain channel attention module is as follows: Among them, Pool It is a global average pooling operation. , These are the weight matrices of two frequency-adaptive dilated convolutional fully connected layers, ReLU. It is a corrected linear unit. This represents the Sigmoid activation function.

5. A video denoising and de-mosaic system based on dual-channel spatiotemporal enhancement, characterized in that, include: Noisy RAW video data processing subsystem: Based on the definition and acquisition method of video denoising and de-mosaicing, the RAW video to be input is segmented into frames to obtain a continuous multi-frame video sequence. The video frames are then cropped and processed into TIFF format. While maintaining the high dynamic range characteristics of the RAW domain of the input data, unnecessary information is reduced, data usability is improved and storage costs are reduced. Video denoising and de-mosaic subsystem: Generates cross-modal binocular image data based on depth control and edge-aware restoration; Supervised optimization and result evaluation subsystem: Saves training parameters and generates reconstruction results, and completes performance evaluation.

Images