Raw domain moire removing method based on Mama
By constructing a Mamba-based U-Net-type multi-scale encoder-decoder architecture and introducing a selective scanning mechanism, the problem of unutilized moiré characteristics in the Raw domain was solved, achieving high-quality demoiré effects and effective suppression of complex moiré patterns.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-08
AI Technical Summary
Existing techniques for removing moiré patterns in the sRGB domain are complex, the moiré pattern characteristics in the Raw domain are not fully utilized, and traditional methods have limited receptive fields or high computational complexity when modeling long-range periodic artifacts.
A U-Net-type multi-scale encoder-decoder architecture is adopted, with both the encoder and decoder built based on Mamba modules. By combining a selective scanning mechanism of temporal and spatial dimensions and training the model through the deep learning PyTorch framework, the ability to model and suppress moiré patterns is improved.
It achieves high-quality demoiring effects, significantly improves the ability to suppress complex moiré patterns, and maintains linear computational complexity, making it suitable for high-resolution image processing.
Smart Images
Figure CN121998871A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image signal processing technology, and in particular to a raw domain demoiring method based on Mamba. Background Technology
[0002] Moiré patterns are a common artifact in digital imaging systems, especially noticeable when photographing objects with fine textures or periodic patterns. This phenomenon is typically caused by the interaction between the sensor pixel array and the detailed structure of the subject, resulting in irregular ripples or stripes in the image, severely impacting visual quality and subsequent processing. Early demoiré patterns networks primarily focused on the sRGB domain. Researchers have proposed various demoiré pattern removal methods based on the multi-scale characteristics and frequency domain recognizability of sRGB domain moiré patterns.
[0003] Moiré patterns are generated by frequency aliasing between the camera's color filter array (CFA) and the display grid. In the sRGB domain, moiré patterns are further affected by image signal processing (ISP), where demosaicing interpolates pixels, complicating the moiré characteristics. Unlike the sRGB domain, Raw domain moiré patterns are generated solely by frequency aliasing between the camera's photosensitive grid array and the display's diode grid array. Without ISP nonlinear transformations, there is no cross-contamination from moiré patterns in other channels, making Raw domain moiré patterns simpler and purer. VDRawmoire proposed using the different degrees of moiré contamination in Raw domain color channels to remove moiré patterns. Mamba's potential in low-level vision tasks has not yet been fully explored, but its architectural characteristics naturally align with the core requirements of such tasks. Mamba effectively utilizes the consistency and differences in raw photosensitive information between channels, exhibiting stronger long-range modeling capabilities than CNNs while avoiding the high computational overhead of Transformers. Therefore, it has significant advantages and broad application prospects in the low-level vision field.
[0004] To address the aforementioned issues, this invention proposes a raw domain demoiring method based on Mamba, achieving optimal demoiring results. Summary of the Invention
[0005] The purpose of this invention is to propose a raw domain demoiring method based on Mamba to solve the problems mentioned in the background art and achieve a higher quality demoiring effect.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] The Mamba-based method for removing moiré patterns from raw domains includes the following steps: S1. Constructing a Mamba-based demoiring model: A U-Net-type multi-scale encoder-decoder architecture is adopted. The encoder extracts multi-scale Raw domain moiré features, and the decoder fuses multi-level contextual information through skip connections, finally outputting a high-quality demoiring image. Both the encoder and decoder are built based on the Mamba module. S2. Design of the spatiotemporal scanning module: Under the Mamba framework, a selective scanning mechanism combining the time and spatial dimensions is introduced, which enables the constructed demoiring model to adaptively focus on the spatiotemporal regions where moiré patterns are significant, thereby improving the modeling and suppression capabilities of moiré patterns. S3. Training the model: Use the deep learning PyTorch framework to train the constructed moiré pattern removal model, and repeatedly traverse the VDRawmoire dataset until the model converges. S4. Output: Input the Raw domain moiré images from the VDRawmoire dataset into the trained model to obtain the RGB images after removing the moiré patterns.
[0008] Preferably, S1 specifically includes the following: S1.1, Given the moiré pattern image as input. I M The shallow feature extraction convolutional layer converts the features into embedded dimensional features, while simultaneously recording the spatial dimension information of the input image for subsequent recovery. The function is represented as: ; S1.2. The obtained features are fed into the image patch embedding module to flatten the two-dimensional feature map into a one-dimensional sequence. Then, a random discarding operation is applied to the sequence to enhance the generalization ability of the model. S1.3 The feature sequentially passes through multiple residual state space groups. Each residual state space group contains a spatially selective scan module and a temporally selective scan module at a specified depth. The function is expressed as: ; S1.4 After state-space processing, the features are fused with the original input through residual connections. Then, channel attention is used for inter-channel information exchange and importance recalibration. Finally, normalization and patching inverse embedding layers convert the 1D sequence back to a 2D feature map, and the recovered result is obtained through residual connections. The function is expressed as:
[0009] in, For residual weights; DP Indicates that the module is randomly discarded; SS 2 D _ T Indicates a time-selective scanning module; SS 2 D _ S Indicates a spatially selective scanning module; S1.5. The feature input channel attention module, which has been processed by the spatiotemporal scanning module, is used to enhance the response of important channels. Then, it is added to the original input through residual connection to obtain the final recovery result of the current level.
[0010] Preferably, S2 specifically includes the following: S2.1 Design a spatial dimension scanning strategy, using different serialization strategies to serialize and scan the feature maps, so that the network learns the spatial texture and structural dependencies of the feature maps; S2.2 Design a time-dimensional scanning strategy to scan the channel time series of each pixel in the feature map and capture the correlation features between the channels and the time series through sequence modeling.
[0011] Preferably, the spatial dimension scanning strategy includes the following: S2.1.1 The input features are expanded to the internal dimensions through a linear projection layer and segmented into input features. X and gating signals Z The function is represented as:
[0012] First through the linear layer Linder e Expand dimensions, then use Chunk segmentation gate control signal Z and input features X ; S2.1.2 Input Features X Local feature enhancement is performed using depthwise separable convolution and applied. SiLU The activation function, specifically, is expressed as follows: ; S2.1.3. Use a specific unfolding method to arrange the features in four directions, including the original order. x transpose x T Reverse sequence and transposed and flipped sequences ; S2.1.4. Using four linear projection layers corresponding to four arrangement directions, generate the parameters of the continuous state equation: time step. State transition matrix A Input projection matrix B Output projection matrix C Jump connection weights D ; S2.1.5, State Transition Matrix A By generating a 1 to d state The basic sequence is broadcast to each channel. d inner Next, after taking the natural logarithm and converting it into a learnable parameter, initialization is completed. The function is represented as:
[0013] in, Index for the scan direction; Indexed by feature dimension; Index for state dimension; A log These are trainable parameters in the logarithmic space, which are autonomously optimized during training. Input projection matrix B Output projection matrix C The output is projected linearly. X L Mapped to In passing torch . split Perform dimensional splitting, splitting into time steps. Input projection matrix B Output projection matrix C The function is represented as:
[0014]
[0015]
[0016]
[0017] in, express No. Position in each direction Features; It is the corresponding submatrix of the linear projection layer; It is the corresponding submatrix of the linear projection layer; It is the corresponding submatrix of the linear projection layer; It is the bias vector; It is the linear projection layer matrix after dimension compression; The activation time step is guaranteed to be positive. S2.1.6 Apply selective scan operation, combined with time step Input projection matrix B Output projection matrix C Jump connection weights DThis completes the discretized state-space transformation, specifically represented by the following function: ; S2.1.7, Merge the output results from the four directions. After layer normalization, it is compared with the gate signal. Z Feature selection is achieved through element-wise multiplication, and finally, the output projection layer maps back to the original dimension. The specific function is as follows: .
[0018] Preferably, the time-dimensional scanning strategy specifically includes the following: Input features X Flatten along the time and channel dimensions, and then reverse the flattened dimension to form two scanning directions; the remaining processing steps are the same as the scanning strategy for the spatial dimension.
[0019] Preferably, S3 specifically includes the following: S3.1 The reconstruction loss adopts L1 loss, which is expressed as follows:
[0020] In the formula, I G Indicates Ground Truth; I DM Indicates the removal of moiré patterns; S3.2, Further introduce perceptual loss and color loss To improve visual quality and color fidelity; mixing loss Represented as:
[0021] in, and This represents the weight parameters, all of which are set to 1.
[0022] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention proposes a U-Net type multi-scale architecture, in which the encoder and decoder are both built based on the Mamba module, and can finally output high-quality demoiré images.
[0023] (2) This invention proposes a selective scanning mechanism that combines time and space dimensions under the Mamba framework, which can effectively improve the modeling and suppression of moiré patterns. Attached Figure Description
[0024] Figure 1This is a diagram illustrating the overall model framework of the Mamba-based Raw domain demoiring method proposed in this invention. Figure 2 This is the spatiotemporal selective scanning implementation process proposed in Embodiment 1 of the present invention. Detailed Implementation
[0025] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0026] Example 1: Please see Figure 1 This invention proposes a method for removing moiré patterns from Raw domains based on Mamba, comprising the following steps: S1. Construct a Mamba-based demoiring model: Traditional demoirée methods based on CNNs or Transformers face challenges such as limited receptive fields or high computational complexity when modeling long-range periodic artifacts. In contrast, the Mamba architecture, with its selective state-space mechanism, can efficiently capture widely existing long-range structured moirée patterns in images with linear complexity, significantly improving the suppression of complex moirées. This invention employs a U-Net-type multi-scale encoder-decoder architecture, where both the encoder and decoder are built based on Mamba modules. The encoder extracts multi-scale Raw domain moirée features step by step, while the decoder fuses multi-level contextual information through skip connections, ultimately outputting a high-quality, detail-preserving demoirée image.
[0027] First, given the input moiré image The initial embedded features are obtained by embedding features through a shallow convolutional layer. Subsequently, the feature is fed into the image patch embedding module, which flattens the two-dimensional feature map into a one-dimensional sequence and applies a random dropout operation to enhance the model's generalization ability. The feature is then sequentially passed through multiple stacked residual state space groups, each containing several cascaded spatially selective and temporally selective scanning modules. These modules perform multi-directional sequence modeling along the spatial and temporal-channel dimensions, respectively, achieving long-range dependency capture and dynamic context awareness. Within each state space group, the feature is updated as follows:
[0028] in DP represents DropPath regularization, where residual weights are either learnable or fixed. After all state space processing is completed, the output features are further processed by a channel attention module to adaptively recalibrate the importance of each channel. Subsequently, through layer normalization and patch in-embedding operations, the one-dimensional sequence is reconstructed into a two-dimensional feature map with the original resolution. Finally, the initial input is fused through residual connections to obtain a high-quality demoiré recovery result for the current layer.
[0029] S2, Design Spatiotemporal Scan Module: First, a channel-by-channel spatial selective scanning strategy is designed to effectively capture the directionality and periodicity of moiré patterns by modeling the long-range spatial structure in the image through multi-directional serialization. Secondly, a pixel-by-pixel time-channel selective scanning strategy is designed, which treats the channel dimension of each pixel as a sequence, models cross-channel correlation, enhances color consistency, and fuses information from adjacent frames.
[0030] The details of each module are as follows: (I) Spatial Dimension Scanning Strategy In raw images, moiré patterns typically exhibit strong directionality and periodicity, and their modeling relies on a joint perception of long-range structure and local texture. To address this, this invention proposes a multi-directional spatial scanning strategy. By serializing two-dimensional feature maps along different spatial paths, the Mamba state-space model can effectively capture structural dependencies in the horizontal, vertical, and diagonal directions.
[0031] Specifically, given input features First, through a linear projection layer Extending this to the internal dimension and splitting it into two parts along the channel dimension: the main input feature With gate signal ,Right now
[0032] Subsequently, Local context enhancement is performed through depthwise separable convolution, and through Activation functions yield locally enhanced features .
[0033] Please see Figure 2 Channel-by-channel spatial dimension scanning, It is unfolded into a one-dimensional sequence according to four spatial orders: the original order Transpose order Time reverse series and the transposed and flipped sequence These four arrangements cover the main structural directions of the image.
[0034] For each direction, separate linear projection layers are used to generate the dynamic parameters required for the state-space equations: time step. Input projection matrix Output projection matrix At the same time, they share the statically initialized state transition matrix. Weights of jump connections .in, The initialization is based on learnable parameters in the logarithmic space. Its initial value is given by the sequence Construct, and satisfy:
[0035] in For direction index, For channel indexing, Index for the state dimension.
[0036] Dynamic parameters Then it is adaptively generated from the input features:
[0037]
[0038]
[0039]
[0040] in Ensure that the time step is always positive and that each weight matrix comes from the projection layer in the corresponding direction.
[0041] Subsequently, selective scan operations are performed in each of the four directions to complete the discretized state space transformation:
[0042] Finally, the four-directional outputs are remapped back to two-dimensional space and merged:
[0043] Where LN represents layer normalization, To perform element-wise multiplication, a gate signal is used to implement it. Feature selection for drivers.
[0044] (ii) Time-channel dimension scanning strategy In addition to spatial structure, the moiré patterns in Raw images also exhibit strong correlations across channels. To model this cross-channel dependency, this invention further designs a time-channel dimension scanning strategy.
[0045] This strategy reuses the entire workflow of the spatial scanning module (including feature projection, local enhancement, parameter generation, selective scanning, and gated fusion), making only adjustments to the serialization method: Please refer to [link to relevant documentation]. Figure 2 Pixel-by-pixel time-channel dimension scanning, input features Flattening along the joint "time-channel" dimension forms a one-dimensional sequence; simultaneously, its reverse sequence is constructed, constituting two scanning directions. Since the channel order itself contains semantics, bidirectional scanning can effectively capture the forward and backward channel dependencies.
[0046] The remaining steps, including dynamic parameter generation, state space evolution, jump connections, and gating modulation, are consistent with the spatial scanning strategy, ensuring architectural uniformity and high efficiency.
[0047] Through the aforementioned spatial and temporal-channel dual-dimensional scanning mechanism, this invention enables Mamba to simultaneously model long-range spatial periodic artifacts and cross-channel structural consistency in the Raw domain, significantly improving the ability to suppress complex moiré patterns while maintaining linear computational complexity, making it suitable for high-resolution image processing.
[0048] S3, Training the Model: The model is trained using the deep learning PyTorch framework, and the VDRawmoire dataset is repeatedly traversed until the model converges.
[0049] The reconstruction loss uses L1 loss:
[0050] In the formula Indicates Ground Truth, This represents the result of removing moiré patterns. To further improve visual quality and color fidelity, perceptual loss is introduced. and color loss .
[0051] Mixed loss Represented as:
[0052] Weight parameters and Set all to 1.
[0053] During training, the batch size was 1, and the input moiré raw image patch size was 192 × 192. The Adam optimizer was used. All experiments were performed using PyTorch on an Nvidia GeForce RTX 3090 GPU.
[0054] S4. Output results: Input the VDRawmoire dataset into the trained model to obtain the RGB image after removing moiré patterns.
[0055] Example 2: Based on Example 1 but with a difference, this invention selects five state-of-the-art comparative methods trained on the VDRawMoire dataset, including three general methods: RDNet, EDVR, and VRT, and two demoiring methods: VDMoire and VDRawMoire. All methods are evaluated based on the VDRawMoire dataset, and the results are shown in Table 1.
[0056] Table 1
[0057] As shown in Table 1, on the VDRawMoire dataset, the method of this invention significantly outperforms the second-best method in PSNR, SSIM, and LPIPS.
[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for removing moiré patterns from raw domains based on Mamba, characterized in that, Specifically, the following steps are included: S1. Constructing a Mamba-based demoiring model: A U-Net-type multi-scale encoder-decoder architecture is adopted. The encoder extracts multi-scale Raw domain moiré features, and the decoder fuses multi-level contextual information through skip connections, finally outputting a high-quality demoiring image. Both the encoder and decoder are built based on the Mamba module. S2. Design of the spatiotemporal scanning module: Under the Mamba framework, a selective scanning mechanism combining the time and spatial dimensions is introduced, which enables the constructed demoiring model to adaptively focus on the spatiotemporal regions where moiré patterns are significant, thereby improving the modeling and suppression capabilities of moiré patterns. S3. Training the model: Use the deep learning PyTorch framework to train the constructed moiré pattern removal model, and repeatedly traverse the VDRawmoire dataset until the model converges. S4. Output: Input the Raw domain moiré images from the VDRawmoire dataset into the trained model to obtain the RGB images after removing the moiré patterns.
2. The method for removing moiré patterns from Raw domains based on Mamba according to claim 1, characterized in that, S1 specifically includes the following: S1.1, Given the moiré pattern image as input. I M The shallow feature extraction convolutional layer converts the features into embedded dimensional features, while simultaneously recording the spatial size information of the input image for subsequent recovery. The function is represented as: ； S1.
2. The obtained features are fed into the image patch embedding module to flatten the two-dimensional feature map into a one-dimensional sequence. Then, a random discarding operation is applied to the sequence to enhance the generalization ability of the model. S1.3 The feature sequentially passes through multiple residual state space groups. Each residual state space group contains a spatially selective scan module and a temporally selective scan module at a specified depth. The function is expressed as: ； S1.4 After state-space processing, the features are fused with the original input through residual connections. Then, channel attention is used for inter-channel information exchange and importance recalibration. Finally, normalization and patching inverse embedding layers convert the 1D sequence back to a 2D feature map, and the recovered result is obtained through residual connections. The function is expressed as: in, For residual weights; DP Indicates that the module is randomly discarded; SS 2 D _ T Indicates a time-selective scanning module; SS 2 D _ S Indicates a spatially selective scanning module; S1.
5. The feature input channel attention module, which has been processed by the spatiotemporal scanning module, is used to enhance the response of important channels. Then, it is added to the original input through residual connection to obtain the final recovery result of the current level.
3. The method for removing moiré patterns from Raw domains based on Mamba according to claim 1, characterized in that, S2 specifically includes the following: S2.1 Design a spatial dimension scanning strategy, using different serialization strategies to serialize and scan the feature maps, so that the network learns the spatial texture and structural dependencies of the feature maps; S2.2 Design a time-dimensional scanning strategy to scan the channel time series of each pixel in the feature map and capture the correlation features between the channels and the time series through sequence modeling.
4. The method for removing moiré patterns from Raw domains based on Mamba according to claim 3, characterized in that, The spatial dimension scanning strategy includes the following: S2.1.1 The input features are expanded to the internal dimensions through a linear projection layer and segmented into input features. X and gating signals Z The function is represented as: First through the linear layer Linder e Expand dimensions, then use Chunk Split door control signal Z and input features X ; S2.1.2 Input Features X Local feature enhancement is performed using depthwise separable convolution and applied. SiLU The activation function, specifically, is expressed as follows: ； S2.1.
3. Use a specific unfolding method to unfold the features in four directions, including the original order. x transpose x T Reversing the sequence and transposed and flipped sequences ; S2.1.
4. Using four linear projection layers corresponding to four arrangement directions, generate the parameters of the continuous state equation: time step. State transition matrix A Input projection matrix B Output projection matrix C Jump connection weights D ; S2.1.5, State Transition Matrix A By generating a 1 to d state The basic sequence is broadcast to each channel. d inner Next, after taking the natural logarithm and converting it into a learnable parameter, initialization is completed. The function is represented as: in, Index for the scan direction; Indexed by feature dimension; Index for state dimension; A log These are trainable parameters in the logarithmic space, which are autonomously optimized during training. Input projection matrix B Output projection matrix C Input is projected linearly X L Mapped to In passing torch . split Perform dimensional splitting, splitting into time steps. Input projection matrix B Output projection matrix C The function is represented as: in, express No. Position in each direction Features; It is the corresponding submatrix of the linear projection layer; It is the corresponding submatrix of the linear projection layer; It is the corresponding submatrix of the linear projection layer; It is the bias vector; It is the linear projection layer matrix after dimension compression; The activation guarantee time step is positive. S2.1.6 Apply selective scan operation, combined with time step Input projection matrix B Output projection matrix C Jump connection weights D This completes the discretized state-space transformation, specifically represented by the following function: ； S2.1.7, Merge the output results from the four directions. After layer normalization, it is compared with the gate signal. Z Feature selection is achieved through element-wise multiplication, and finally, the output projection layer maps back to the original dimension. The specific function is as follows: 。 5. The method for removing moiré patterns from Raw domains based on Mamba according to claim 4, characterized in that, The time-dimensional scanning strategy specifically includes the following: Input features X Flatten along the time and channel dimensions, and then reverse the flattened dimension to form two scanning directions; the remaining processing steps are the same as the scanning strategy for the spatial dimension.
6. The method for removing moiré patterns from Raw domains based on Mamba according to claim 1, characterized in that, S3 specifically includes the following: S3.1 The reconstruction loss adopts L1 loss, which is expressed as follows: In the formula, I G Indicates Ground Truth; I DM This indicates the result of removing moiré patterns; S3.2, Further introduce perceptual loss and color loss To improve visual quality and color fidelity; mixing loss Represented as: in, and This represents the weighting parameter.