Multimodal adaptive low-illumination raw image enhancement method and system
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2026-05-09
- Publication Date
- 2026-08-07
AI Technical Summary
为解决传统状态空间模型在极暗场景中易破坏RAW拜耳阵列二维周期性,以及纯视觉网络缺乏语义认知导致细节丢失且过度依赖人工文本提示的问题,本发明在原始传感器域开展建模,通过模拟文本条件的潜在表示进行语义调制,并结合混合移位扫描机制,兼顾了全局长距离依赖建模与局部结构保真度,从而获得颜色更一致、无伪影且细节更完整的增强结果
[0023] Compared with the prior art, the present invention has the following advantages:
Smart Images

Figure CN122530002A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing and computer vision technology, and relates to improving the quality of low-light imaging, particularly for RAW data output from digital image sensors, under the constraints of multimodal adaptation and multi-scale fusion. Specifically, it proposes a frequency domain multi-scale enhancement method and system for low-light RAW images, which can improve denoising robustness, edge sharpness, and color consistency in scenes with strong noise, underexposure, color cast, and detail degradation. It is applicable to on-device and cloud inference environments such as mobile phones, cameras, security systems, automotive, medical, and industrial inspection. Background Technology
[0002] Under extreme low-light conditions, RAW data acquired by image sensors not only suffers from severe signal-to-noise ratio imbalance, but its delicate intrinsic optical structure is also easily overwhelmed by complex photon shot noise and sensor readout noise, presenting extremely high recovery difficulties. Traditional convolutional neural networks (CNNs) are limited by local receptive fields, making it difficult to achieve global illumination and color consistency correction; while visual Transformers possess global modeling capabilities, their quadratic computational complexity faces severe computational bottlenecks on high-resolution RAW data. Although the state-space model (SSM) introduced in recent years has achieved long-distance modeling with linear complexity, its conventional one-dimensional spatial scanning operation brutally disrupts the inherent two-dimensional periodic arrangement of the RAW format Bayer array, leading to the breakage of spatial correlation between channels, and consequently causing severe color shifts and spatial artifacts. Furthermore, existing pure visual enhancement architectures lack advanced semantic cognition, making it difficult to accurately distinguish between intrinsic texture and random noise in extreme noise, easily resulting in oversmoothing; while existing multimodal methods heavily rely on manual text prompts or large online text encoders during the inference stage, resulting in a severe modal gap and making them unsuitable for deployment in automated image signal processing (ISP) pipelines. Therefore, there is an urgent need for a state-space modeling method that combines "unannounced multimodal semantic guidance" and "RAW perceptual spatial scanning" to give the model autonomous scene degradation perception capabilities while maintaining the Bayer array structure prior and linear complexity, thereby achieving efficient, color-consistent and detail-preserving image reconstruction in extremely dark scenes. Summary of the Invention
[0003] This invention relates to RAW image enhancement under low-light conditions, proposing an end-to-end method and system that combines unaided multimodal semantic guidance with RAW perceptual spatial scanning. To address the problems of traditional state-space models easily disrupting the two-dimensional periodicity of the RAW Bayer array in extremely dark scenes, and the lack of semantic cognition in pure visual networks leading to detail loss and over-reliance on human text prompts, this invention performs modeling in the original sensor domain, performs semantic modulation through latent representations simulating text conditions, and combines a hybrid shift scanning mechanism to balance global long-range dependency modeling with local structural fidelity, thereby obtaining enhancement results with more consistent colors, no artifacts, and more complete details.
[0004] The present invention is achieved using the following technical solution.
[0005] The multimodal adaptive low-light RAW image enhancement method includes the following steps in its overall process:
[0006] Step 1: Input and Preprocessing.
[0007] Basic preprocessing is performed on the input low-light RAW image, including black level correction and size alignment. Fixed biases for each channel are removed to bring the signal back to the zero-based linear domain, and strict size alignment is performed to prevent subsequent operations from disrupting intra-channel correlations.
[0008] Step 2: Retinex decomposition and prior constraints.
[0009] A decomposition strategy based on the Retinex concept is introduced to decouple the severely degraded RAW observation signal into an illumination map and its corresponding reflection features. This physically constrained initialization process provides a deterministic prior basis for subsequent enhancement trajectories and effectively reduces the interference caused by illumination fluctuations on structure discrimination under extremely dark conditions.
[0010] Step 3: Semantic representation generation and state space modeling.
[0011] This invention proposes a promptless teacher-student distillation strategy and a text student module. During the inference phase, it directly simulates and extracts latent representations of textual conditions from degenerate visual features, generating a global semantic embedding. The core feature transformation is accomplished by a series of semantically guided Mamba blocks. Before entering the state space transformation, visual features undergo adaptive layer normalization, and their affine modulation process is implemented through the following formula:
[0012]
[0013] in, This represents the visual features after normalization and global semantic modulation. Represents the original visual features of the input; This represents the global semantic embedding generated by the text student module; and These represent the affine scaling parameter and offset parameter generated by the global semantic embedding dynamic regression, respectively; This represents the standard layer normalization operation. This process aligns the local feature distribution with the global scene prior and modulates the readout matrix of the state transition equation, giving the model the ability to distinguish between inherent optical structures and complex noise distributions.
[0014] Step 4: Hybrid shift scanning and spatial structure fidelity.
[0015] This invention proposes an attention-based SSM module with a built-in hybrid shift scanning mechanism that divides the input tensor into two channel groups. The first group performs orthogonal serpentine scanning, alternating between horizontal and vertical traversal; the second group performs nested serpentine scanning, deploying a spatial cyclic shift aligned with the Bayer pattern before sequence traversal. The shift operation is defined as follows:
[0016]
[0017] in, The two-dimensional spatial feature tensor representing the input; and These represent the spatial coordinates of the height and width dimensions of the feature map, respectively. Indicates the total width of the feature map; This indicates the shift ratio parameter, which is precisely calibrated to 0.25 for common 4-channel RAW package formats; This represents the modulo operation. This design effectively offsets the disruption caused by one-dimensional flattening without increasing computational complexity, explicitly preserving the inherent two-by-two intra-channel correlation and global receptive field of the RAW format.
[0018] Step 5: Cross-layer feature filtering and result output.
[0019] By utilizing a semantic skip-gate mechanism, channel-level gating weights are generated through global semantic embedding to selectively modulate encoder features. This mechanism accurately filters noise-dominated channels while preserving key structural frequencies, preventing low-level noise distributions from directly propagating to the decoder. Feature downsampling and upsampling are achieved through discrete wavelet transform and inverse transform, mapping back to RAW representation, ultimately obtaining an enhanced high-fidelity RAW image.
[0020] Furthermore, the method of this invention employs a joint optimization strategy of reconstruction and semantic distillation during the training phase. The overall optimization objective is formed by linearly weighting the spatial reconstruction loss of RAW and RGB domains and the text and visual semantic distillation loss, guiding the network to maintain the structural consistency of the real scene and the naturalness of global illumination while achieving strong denoising.
[0021] Therefore, in terms of system implementation, this invention provides a complete pipeline consisting of a multimodal data access unit, a Retinex decomposition unit, a text student unit, a feature extraction and dual adaptive processing unit, a RAW domain fidelity reconstruction unit, and a model management and engineering deployment unit. Through the above design, this invention achieves synergistic optimization of multimodal semantic guidance and spatial topology awareness in the RAW domain, demonstrating excellent applicability in various imaging hardware systems and low-light vision applications.
[0022] Beneficial effects
[0023] Compared with the prior art, the present invention has the following advantages:
[0024] 1. This invention constructs an enhanced architecture based on a state-space model in the RAW domain and innovatively proposes a hybrid shift scanning mechanism. Unlike traditional visual networks or standard one-dimensional sequence scanning, this invention, while establishing a global long-range receptive field, explicitly preserves the inherent two-by-two channel intra-correlation and two-dimensional spatial prior of the RAW format through a combination of orthogonal scanning and Bayer-calibrated spatial cyclic shifting. This design not only physically avoids the signal distortion and error accumulation caused by color interpolation and nonlinear mapping in traditional image signal processing pipelines, but also effectively eliminates spatial artifacts and color shifts caused by conventional state-space models when processing RAW data, achieving efficient inference with linear complexity and excellent color and structure fidelity in extremely dark scenes.
[0025] 2. The unhinted multimodal semantic guidance design of this invention endows pure visual networks with advanced scene degradation cognition capabilities. Through a teacher-student distillation strategy, the system aligns the visual latent space with degradation diagnostic priors extracted by a high-level visual language model during the training phase. This allows the system to autonomously simulate accurate semantic embeddings directly from degraded visual input without relying on any manual text prompts or large online text encoders during the inference phase. This mechanism successfully bridges the gap in the practical application of multimodal technologies, perfectly adapting to the deployment requirements of automated image signal processing pipelines. Simultaneously, it enables the model to accurately distinguish between inherent optical textures and complex random noise even at extremely low signal-to-noise ratios, effectively overcoming the oversmoothing problem easily caused by traditional blind restoration networks, and significantly improving the sharpness and realism of detail restoration.
[0026] 3. The semantic modulation and skip gating strategy of this invention achieves dynamic and precise noise filtering in cross-layer feature fusion. By injecting global semantic embedding into the adaptive layer normalization and the readout matrix of the state transition equation, the system can dynamically adjust the response weights of the state space equation according to the degradation features of the specific scene. In addition, the semantic skip gate selectively attenuates skip connection features using the generated gating array, effectively blocking the direct propagation of shallow strong noise distribution to the decoder. This conditional and dynamic feature filtering mechanism enables the model to maintain excellent robustness and generalization ability when facing different illumination ratios, extremely low exposure conditions, or even extreme data distribution shifts caused by cross-camera sensor, and stably outputs enhanced images with clear edges and natural brightness. Attached Figure Description
[0027] Figure 1 This is a flowchart of the method of the present invention.
[0028] Figure 2 This is a schematic diagram of the model structure described in the method of the present invention.
[0029] Figure 3 This is a schematic diagram of the system composition of the present invention. Detailed Implementation
[0030] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below.
[0031] Example
[0032] like Figure 1 As shown, a text-guided low-light RAW image enhancement method includes the following steps:
[0033] Step 1: Input and Preprocessing. After acquiring the low-light RAW image to be enhanced, black level correction is first performed to remove the fixed bias of each channel, bringing the signal back to the linear domain with zero as the reference. Then, the original pixels are linearly normalized according to the sensor's full threshold, preserving the linear response relationship for subsequent feature estimation and noise modeling. For common Bayer arrays or Quad Bayer arrangements, the system performs strict size alignment during the preprocessing stage to ensure that subsequent strided convolutions or shift operations do not destroy the inherent intra-channel correlations. Simultaneously, textual descriptions corresponding to the scene (e.g., "A very dark indoor photo with severe noise") are acquired, and their high-level semantic features are extracted using a pre-trained CLIP text encoder to provide teacher signals for subsequent teacher-student distillation.
[0034] Step 2: Retinex Decomposition and Prior Constraints. The preprocessed RAW is mapped to the feature space, and the illumination map and corresponding reflection features are estimated via the Retinex decomposition module. The illumination map describes a slowly changing distribution of light and dark areas, requiring piecewise smoothness in space; the reflection features carry texture and structural information, encouraging a sparse distribution to highlight realistic details. This decomposition, based on physical optical constraints, decouples the severely degraded observation signal, providing a stable prior foundation for subsequent text condition injection and state space modeling, effectively reducing interference from illumination fluctuations on structure discrimination under extremely dark conditions.
[0035] Step 3: Semantic Representation Generation and State Space Modeling. Moving into the feature transformation stage, the system abandons traditional convolutional or self-attention mechanisms, employing semantically guided Mamba blocks for long-range context aggregation. First, the text student module directly extracts latent representations from the visual features output in Step 2, generating simulated text conditional semantic embeddings.
[0036]
[0037] It is a text vector obtained from the teacher-student distillation network; Represents the original visual features of the input; It is a teacher-student distillation network. During the training phase, the embedding is distilled and aligned with the teacher embedding generated from real text using cosine similarity loss; during the inference phase, it relies entirely on... It provides scene degradation awareness. This semantic embedding first injects an adaptive layer normalization module to dynamically regress affine scaling and offset parameters, globally modulating the visual features. Subsequently, the features enter the attention-based SSM module, utilizing its internal semantic modulation mechanism according to... The readout matrix of the state transition equation is dynamically adjusted to guide the model in distinguishing between inherent optical structures and complex noise distributions.
[0038] Step 4: Hybrid Shift Scanning with Spatial Structure Preservation. Within the attention SSM module, to overcome the disruption of the two-dimensional periodicity of RAW data by standard one-dimensional sequence scanning, a hybrid shift scan operation is performed. The input feature tensor is divided into two channel groups: the first group performs orthogonal serpentine scanning, alternating between horizontal and vertical traversals to establish an unbiased two-dimensional global receptive field; the second group performs nested serpentine scanning, deploying spatial cyclic shifts before sequence traversal.
[0039]
[0040] in, The two-dimensional spatial feature tensor representing the input; and These represent the spatial coordinates of the height and width dimensions of the feature map, respectively. Indicates the total width of the feature map; This indicates the shift ratio parameter, which is precisely calibrated to 0.25 for common 4-channel RAW package formats; This represents the modulo operation. This design effectively offsets the disruption caused by one-dimensional flattening without increasing computational complexity, explicitly preserving the inherent two-by-two intra-channel correlation and global receptive field of the RAW format. The shift ratio is... The precision calibration is 0.25 (for a 4-channel RAW container format). This shift operation forces the state-space model to be strictly aligned and preserves the inherent 2x2 intra-channel correlation of the RAW format when calculating global dependencies. After summing the two sets of scan results, efficient feature extraction is achieved with linear complexity, balancing global illumination consistency and local texture fidelity.
[0041] Step 5: Cross-layer feature filtering and result output. A semantic skip gate mechanism is deployed at the skip connections in the U-Net architecture. This mechanism utilizes global semantic embeddings to compute a channel-level gating array via a multilayer perceptron:
[0042]
[0043] It is a processed text vector used for injection into the model; and These are the network layers and activation layers, respectively. Then, the encoder features are analyzed. Element-wise multiplication modulation is performed to obtain the filtered features. This gating operation strictly adheres to semantic priors, selectively attenuating noise-dominant feature channels to prevent low-level noise distributions from directly propagating to the decoder. The decoded features are then upsampled using inverse discrete wavelet transform and mapped back to their RAW representation, resulting in an enhanced high-fidelity RAW image. For easy comparison, the enhancement result can be simultaneously output as an RGB preview image via a lightweight rendering pipeline.
[0044] The decomposition in step 2, the semantic generation in step 3, and the spatial scanning in step 4 are tightly integrated in the implementation. Retinex decomposition provides a clear physical signal path; the text student module endows the network with the ability to "recognize degradation," transforming state-space modeling from "blind restoration" to "conditional recovery"; and the hybrid shift scanning ensures that this powerful global modeling capability does not come at the expense of the underlying physical arrangement. The entire process can robustly extract weak textures at extremely low signal-to-noise ratios while completely eliminating the dependence on external text encoders.
[0045] like Figure 2As shown, the low-light RAW image enhancement network proposed in this embodiment of the invention adopts an end-to-end structure of "U-shaped multi-scale, semantic dual-path injection, and dual-domain supervision". The leftmost part of the network consists of the low-light RAW input and the text prompt (used only during training). The RAW input is first decoupled by the Retinex decomposition module, and then enters the encoder based on discrete wavelet transform downsampling. At each level of the network, the core computational unit is a semantically guided Mamba block.
[0046] The diagram details the internal structure of the semantically guided Mamba block. It clearly shows the global semantic embeddings generated by the text-based student module. As a key control signal, it is simultaneously injected into the adaptive layer normalization module and the attention SSM module. Inside the attention SSM module, the feature stream is split into two, which are sent to the orthogonal serpentine scanning path and the nested serpentine scanning path, respectively. Then, these two sequence features carrying rich 2D spatial priors enter the semantic modulation SSM module to complete the controlled long-distance dependency calculation.
[0047] The skip connection section in the diagram illustrates the working mechanism of the semantic skip gate. Before being concatenated with the decoder features, features from the same scale from the encoder must first pass through the semantic skip gate. This selection is also controlled by $z_{sem}$, ensuring that only pure high-frequency information relevant to the scene structure participates in the final reconstruction. The right side of the network represents the decoding and output ends. Features upsampled by the inverse discrete wavelet transform ultimately reconstruct the enhanced RAW image and the rendered RGB image, respectively. During training, the RAW domain reconstruction loss and the RGB domain consistency loss work synergistically, while the text distillation loss constrains the optimization direction of the text student module; during inference, only the purely visual forward propagation path is retained, achieving efficient end-to-end enhancement.
[0048] The following is about Figure 3 The system composition and workflow shown are further explained in detail below. To facilitate implementation and engineering deployment, this invention divides the system into the following core functional modules. Each module is tightly coupled with tensor flow and semantic control flow to ensure high-fidelity recovery in extremely dark scenes.
[0049] The system first includes a multimodal data access and preprocessing module. This module is responsible for reading the low-light RAW files output by the sensors, parsing them, and performing black level subtraction and linear normalization. In training mode, this module is additionally responsible for loading scene text descriptions and calling the frozen visual language large model to extract teacher semantic vectors and establish a reference language latent space; in inference mode, it only processes visual data.
[0050] Following this is the Retinex Physics Prior Decomposition module. This module receives normalized RAW files and approximates the illumination distribution and reflection texture using a lightweight convolutional network. The output piecewise smoothed illumination map and sparse reflection features are packaged and fed into the subsequent backbone network, providing physical constraints for the model to anchor structural boundaries in extreme noise.
[0051] The Text Student Module (TSM) and the Semantic Routing Module constitute the system's "cognitive hub." This module contains a small feature extraction and mapping network that directly "guesses" the corresponding latent text representation from the current degraded visual input. During training, the Semantic Routing Module injects downstream semantic vectors (teacher or student) with probabilistic switching; however, in actual deployment, the Semantic Routing Module is hardcoded to only receive the output of the Student Module, thus completely severing external dependencies on the text modality.
[0052] The semantic feature modulation and extraction module is the main computational flow of the system. This module consists of semantically guided Mamba blocks distributed across various layers of the U-shaped architecture. Internally, the module first utilizes adaptive layer normalization to dynamically adjust the mean and variance of the visual features based on the global semantic embedding. Subsequently, the features enter the core hybrid shift scanning unit. The hybrid shift execution unit is specifically designed for Bayer or Quad Bayer arrays. Instead of using traditional row or column flattening, the module performs orthogonal serpentine scanning with Bayer strides (e.g., ...) on the segmented feature groups. The nested serpentine scan of the unit allows the subsequent one-dimensional state-space equations to "sense" the true physical adjacency of pixels in the two-dimensional plane, avoiding interpolation-like color distortion.
[0053] The Semantic Modulation (SSM) processing unit receives the scanned sequence. Internally, the module maps the global semantic embedding through a learnable projection matrix, dynamically generating fine-tuning amounts for the readout matrix of the state equation. This design enables the state-space model to exhibit strong low-pass filtering characteristics in flat, textureless regions to suppress shot noise, while exhibiting band-pass or full-pass characteristics in regions where semantic cues indicate structure to preserve edge sharpness.
[0054] A semantic cross-layer gating and fusion module is deployed at the connection between the encoder and decoder. This module receives shallow features from the encoder and generates a channel mask matrix using global semantic embedding. Through channel-by-channel soft thresholding, the module effectively intercepts high-frequency noise features, passing only the clean structural residuals to the decoder for concatenation and fusion with deeper features.
[0055] Finally, the RAW domain fidelity reconstruction and rendering module maps the processed deep features back to full-resolution space using discrete wavelet inverse transform. The system not only outputs enhanced, linearly responsive high dynamic range RAW files for further color correction by professional ISP pipelines, but also incorporates a lightweight, differentiable ISP proxy module that outputs real-time RGB preview images with white balance and gamma correction to support dual-domain loss calculation and visual manual evaluation. These modules are optimized end-to-end within a unified training framework and can be compiled into an efficient static computation graph during deployment, perfectly meeting the low-light image processing needs of edge devices.
[0056] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A text-guided low-light RAW image enhancement method, characterized in that, Includes the following steps: Step 1: Data preprocessing and decomposition. Obtain low-light RAW images and decouple them into illumination maps and corresponding reflection features via the Retinex decomposition module (RDM). Step 2: Semantic representation generation, utilizing the Text Student Module (TSM) to extract latent representations from degraded visual features to generate simulated textual conditional semantic embeddings for autonomous reasoning; Step 3: Semantic-guided state space modeling. The features obtained in Step 1 are input into an encoder-decoder architecture containing multiple semantically guided Mamba blocks (SGMBs), and the features are modulated in adaptive layer normalization (ALN) through the semantic embedding. Step 4: RAW perception long-distance dependency modeling. In the attention SSM module (ASM) of the SGMB, the hybrid shift scan (HSS) mechanism is used to process features, calculate global dependencies while preserving the inherent 2D spatial priors. Step 5: Image reconstruction and output. The semantic skip gate (SSG) is used to adjust the skip connection features. After upsampling via inverse discrete wavelet transform (IWT), the high-fidelity enhanced RAW and sRGB images are reconstructed and output.
2. The method as described in claim 1, characterized in that, The hybrid shift scanning mechanism in step 4 divides the input tensor into two channel groups: the first group performs orthogonal serpentine scanning, alternating between horizontal and vertical traversal to aggregate continuous two-dimensional spatial texture; the second group performs nested serpentine scanning, deploying spatial cyclic shifts calibrated with Bayer mode before sequence traversal to accurately locate the receptive field and maintain the inherent 2x2 intra-channel correlation of the RAW format.
3. The method as described in claim 1, characterized in that, The attention SSM module includes a semantic modulation SSM, which uses the readout matrix of the semantic embedding dynamic modulation state transition equation to filter unconstrained noise distributions by a learnable projection matrix that limits the modulation amplitude, while preserving the inherent structural representation.
4. The method as described in claim 1, characterized in that, In step 2, the text student module adopts an unannounced teacher-student distillation strategy during the training phase. Specifically, the pre-trained visual language model extracts the teacher representation from the real text description and aligns the latent space of the text student module with the teacher's high-level semantic priors through cosine similarity distillation.
5. The method as described in claim 4, characterized in that, The training phase employs a random semantic routing mechanism, where global semantic embeddings injected into the semantically guided Mamba block and semantic jump gate are sampled from the teacher representation with half the probability, and from the student representation otherwise. In the inference phase, the student representation is strictly used by default to isolate auxiliary text dependencies.
6. The method as described in claim 1, characterized in that, The semantic skip gate in step 5 processes the global semantic embedding through a multilayer perceptron to calculate the channel-level gate array and performs element-wise multiplication modulation on the encoder representation. This selectively attenuates the noise-dominated feature channels under strict semantic prior conditions while preserving the basic structural frequencies.
7. A text-guided low-light RAW image enhancement system, characterized in that, The system comprises a Retinex decomposition module, a text student module, a feature extraction and reconstruction module, and a semantic skip gate module, connected in sequence. The Retinex decomposition module decouples the degraded RAW input into an illumination map and reflection features. The text student module generates a global semantic embedding by instinctively simulating the latent representation of text conditions from the degraded input. The feature extraction and reconstruction module consists of a U-shaped network architecture based on discrete wavelet transform and multiple semantically guided Mamba blocks. The semantically guided Mamba blocks have a built-in attention SSM module, which uses hybrid shift scanning to aggregate orthogonal and nested serpentine scanning to preserve two-dimensional spatial priors. The semantic skip gate module filters skip connection features based on the global semantic embedding.
8. The system as described in claim 7, characterized in that, The optimization objective function of the system includes a linearly aggregated spatial reconstruction loss and a semantic distillation constraint, wherein the spatial reconstruction loss calculates the L1 norm in both the RAW and standard RGB domains, and the semantic distillation constraint penalizes the angular divergence between the visual derived representation and the text-derived embedding.
9. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores a program that can run on the processor, and the program executes the steps of any of the preceding methods when it runs.