An inverse isp method based on spiral diffusion model and camera perception adaptation

Abstract

The application discloses an inverse ISP method based on a spiral diffusion model and camera perception adaptation, and relates to the technical field of computer photography and image signal processing. The method comprises the following steps: S1, constructing an inverse ISP training sample, each training sample comprising an RGB image, a target RAW image corresponding to the RGB image and a camera label; S2, constructing an inverse ISP network based on a spiral diffusion model, introducing a time-varying weight map related to pixel intensity at different time steps in the diffusion process; S3, setting a camera perception low-rank adaptation module in the inverse ISP network, comprising a plurality of low-rank adaptation branches corresponding to different camera labels, and selecting a corresponding low-rank adaptation branch to participate in network calculation according to the input camera label; S4, in the training stage, training the inverse ISP network based on the forward probability distribution of the spiral diffusion model; in the sampling stage, inputting the RGB image and the camera label into the trained inverse ISP network, and obtaining the target RAW image based on the reverse iterative sampling process of the spiral diffusion model.

Description

Technical Field

[0001] This invention relates to the fields of image signal processing, computational photography, and computer vision, and particularly to an inverse ISP method based on a spiral diffusion model and camera perception adaptation, used to reconstruct the corresponding RAW image from an input RGB image. Background Technology

[0002] RAW images are the direct output of image sensors. Compared to RGB images after image signal processing, RAW images retain richer linear radiometric response information, higher dynamic range, and more realistic noise distribution characteristics. Therefore, they have important application value in tasks such as low-light enhancement, image denoising, and target detection.

[0003] In actual imaging processes, RAW images typically undergo image signal processing steps such as desacrifice, white balance, color correction, tone mapping, and compression encoding to obtain RGB images suitable for display and storage. These processing steps alter or even destroy crucial physical information within the RAW image. Therefore, how to recover a RAW image from an RGB image—the inverse ISP problem—has become an important research direction in the field of image processing in recent years.

[0004] Existing RGB to RAW conversion methods typically employ end-to-end mapping, cycle consistency constraints, or generative models for modeling. While these methods can achieve RGB to RAW inversion to some extent, they still suffer from the following problems:

[0005] First, existing methods typically employ a uniform reconstruction strategy, neglecting the differences in reconstruction difficulty between RGB-to-RAW regions with varying pixel intensities. For low-brightness areas, the residual between RGB and RAW is small and relatively stable, so high-fidelity restoration should be emphasized. However, for high-brightness or overexposed areas, the mapping uncertainty between RGB and RAW increases significantly due to factors such as nonlinear tone mapping and truncation saturation. If a uniform perturbation intensity and a uniform generation mechanism are still used, it is often difficult to balance the restoration of details in dark areas with the flexible generation of bright areas.

[0006] Secondly, existing methods typically lack effective camera-specific adaptation mechanisms in multi-camera scenarios. Different cameras exhibit significant device-dependent mapping relationships in the RGB-to-RAW conversion of the same scene due to variations in sensor response curves, white balance strategies, ISP processes, and color styles. Simply training with mixed multi-camera data can easily confuse the model with the imaging characteristics of different cameras, leading to a decrease in conversion accuracy.

[0007] Therefore, the inverse ISP method, which can adaptively adjust the diffusion process based on pixels and achieve camera-specific adaptation based on the input camera label, is a feasible solution to improve conversion accuracy. Summary of the Invention

[0008] The purpose of this invention is to provide an inverse ISP method based on a spiral diffusion model and camera perception adaptation to solve the following problems existing in current RGB to RAW conversion methods: (1) Using a unified reconstruction mechanism for different pixel intensity areas makes it difficult to simultaneously take into account both dark and bright areas; (2) In multi-camera scenarios, the lack of device-related adaptation leads to mutual interference of different camera characteristics.

[0009] The objective of this invention is achieved through the following technical solution: An inverse ISP method based on a spiral diffusion model and camera perception adaptation includes the following steps: S1. Construct inverse ISP training samples, each training sample including an RGB image, a target RAW image corresponding to the RGB image, and a camera label; S2. Construct an inverse ISP network based on a spiral diffusion model. The spiral diffusion model introduces a time-varying weight map related to pixel intensity at different time steps in the diffusion process to adjust the noise perturbation intensity of different pixel positions at each time step. S3. Set up a camera-aware low-rank adaptation module in the reverse ISP network. The camera-aware low-rank adaptation module includes multiple low-rank adaptation branches corresponding to different camera labels, and selects the corresponding low-rank adaptation branch to participate in network calculation according to the input camera label. S4. During the training phase, the inverse ISP network is trained based on the forward probability distribution of the spiral diffusion model. During the sampling phase, the RGB image and camera label are input into the trained inverse ISP network, and the target RAW image is obtained based on the reverse iterative sampling process of the spiral diffusion model.

[0010] Preferably, the input RGB image is set as follows: The target RAW image is The residual term is:

[0011] in, Residual information between RGB and RAW images is used to characterize the signal changes introduced during the RAW to RGB conversion process; The spiral diffusion model constructs a time-varying weight map related to pixel intensity:

[0012] in, Represents a time-varying weighted graph; Indicates a time step; This represents the shift coefficient that changes with the time step. Differential perturbations are applied to different brightness regions, with smaller perturbation intensities for low-intensity regions and larger perturbation intensities for high-intensity regions, so that the diffusion process matches the reconstruction difficulty of different regions in the RGB to RAW conversion.

[0013] Preferably, the time-varying weight map changes dynamically with the diffusion time step and is introduced into the residual diffusion model to construct the forward probability distribution of the spiral diffusion model:

[0014] in, and Let these represent the mean and variance, respectively, and their calculation formulas are as follows:

[0015]

[0016]

[0017] in, This represents the coefficient related to the weight.

[0018] Preferably, the camera-perception low-rank adaptation module updates the shared network weights through low-rank decomposition; for the i-th camera, its corresponding weight matrix is ​​represented as:

[0019] in, For shared weight matrix; and This is the low-rank adaptation parameter matrix corresponding to the i-th camera.

[0020] Preferably, the inverse ISP network adopts an encoder-decoder structure that includes an attention module, and a camera-aware low-rank adaptation module is embedded in the attention module for camera-related modulation of the network weights.

[0021] Compared with existing technologies, this invention provides an inverse ISP method based on a spiral diffusion model and camera perception adaptation, which has the following beneficial effects: (1) By introducing a time-varying weight map related to pixel intensity at different time steps in the diffusion process, the present invention enables low brightness areas to maintain high reconstruction fidelity, while giving high brightness and overexposed areas greater generation flexibility, thereby better adapting to signal-related uncertainties in RGB to RAW conversion.

[0022] (2) The spiral diffusion model proposed in this invention can couple the diffusion process with the signal conversion process from RGB to RAW, making the evolution of the intermediate state more in line with the characteristics of the inverse ISP task, which is conducive to improving reconstruction stability and output quality.

[0023] (3) By setting up a camera perception low-rank adaptation module, the present invention enables the unified model to learn and call specific camera parameters based on the input camera labels, effectively alleviating the feature interference problem in multi-camera joint training and improving the inverse ISP conversion performance in cross-camera scenarios.

[0024] (4) Based on maintaining backbone network sharing, this invention can achieve device-related modeling with only a small number of low-rank adaptation parameters, which has good parameter efficiency and scalability, and is convenient for subsequent adaptation to new cameras. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the effects of forward noise diffusion and reverse noise reduction diffusion on high dynamic range images according to the present invention.

[0026] Figure 2 This is a schematic diagram of the high dynamic range image reconstruction process of multi-exposure images based on the diffusion model of the present invention.

[0027] Figure 3 This is a schematic diagram of the multi-exposure image feature guidance network based on spatial attention mechanism and the window-based pixel-level interactive attention mechanism in this invention.

[0028] Figure 4 This invention is compared with other high dynamic range image reconstruction methods. Detailed Implementation

[0029] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0030] Example 1: Please see Figure 1 This embodiment provides an inverse ISP method based on a spiral diffusion model and camera perception adaptation, including the following steps: S1. Construct inverse ISP training samples. Each training sample includes an RGB image, a target RAW image corresponding to the RGB image, and a camera label. S2. Construct an inverse ISP network based on a spiral diffusion model. The spiral diffusion model introduces time-varying weight maps related to pixel intensity at different time steps in the diffusion process to adjust the noise perturbation intensity at different pixel locations at each time step; please refer to [link to relevant documentation]. Figure 2-3 Specifically, it includes the following: S2.1, Basic Variable Definition: Let the input RGB image be denoted as... The target RAW image is denoted as The residual between the two is denoted as:

[0031] in, It represents the residual information between the RGB image and the RAW image, and is used to characterize the signal changes introduced during the RAW to RGB conversion process.

[0032] In this embodiment, the spiral diffusion model is established from RAW images. to RGB image The diffusion and transfer process is used to construct a sampling process for reverse recovery of RAW images from RGB images.

[0033] S2.2 Time-varying weight map design: Unlike existing diffusion models that use uniform Gaussian noise at all pixel locations, this embodiment considers that the reconstruction difficulty of RGB to RAW conversion is closely related to pixel intensity. Therefore, a set of time-varying weight maps related to pixel intensity is introduced at different time steps in the diffusion process. At time step Below, the time-varying weighted graph is defined as:

[0034] in, The shift coefficient is a monotonically changing factor that satisfies the following condition: and followed As it increases, it gradually approaches 1.

[0035] From the above formula, it can be seen that the time-varying weighted graph Consistent with the noise-free intermediate state during the diffusion process, it can be regarded as a RAW image. To RGB image Intermediate representation during gradual transition. As the time step changes, the time-varying weight map also changes synchronously, so that the noise injection intensity can be matched with the signal conversion process.

[0036] In a preferred embodiment, considering To address the issues of value range and numerical stability, the weight graph is normalized and a lower bound bias term is introduced to obtain the corrected weight graph. :

[0037] in, A preset bias coefficient is used to ensure that the weight graph is strictly greater than zero, avoiding numerical instability during subsequent probability distribution calculations. In a preferred embodiment, We set the value to 0.1. Through the above design, low-brightness areas correspond to smaller noise perturbations, while high-brightness areas or overexposed areas correspond to larger noise perturbations. This allows the model to focus more on high-fidelity restoration in dark areas and to have greater generation flexibility in bright areas.

[0038] S2.3 Forward Probability Distribution: In this embodiment, the forward transition probability distribution of the spiral diffusion model is expressed as:

[0039] in, Indicates a Gaussian distribution. , This is the overall scaling factor. This is the identity matrix. From the above recursive relationship, the time step can be further obtained. The following edge distribution:

[0040] Therefore, in this invention, the noise variance during the diffusion process is no longer spatially uniform, but is controlled by the weight map corresponding to the pixel intensity at the current position, thus forming a spatially and temporally related noise scheduling mechanism.

[0041] S2.4, Reverse Probability Distribution: Based on the above forward distribution, the corresponding reverse transition probability distribution can be derived:

[0042] The mean and variance terms are respectively:

[0043]

[0044]

[0045] As can be seen from the above expression, the mean term in the backsampling distribution is determined by two parts: One part corresponds to the back projection result of the current noise state, and the other part corresponds to the forward approximation result of the clear target structure. The two are connected by coefficients related to the weight map. Pixel-by-pixel fusion is performed. Compared with existing diffusion models with uniform coefficients, this invention can achieve an adaptive balance between noise and sharp states at different pixel locations, thus making the sampling trajectory more consistent with the characteristics of the inverse ISP task.

[0046] S3. Configure a camera-aware low-rank adaptation module in the reverse ISP network. This module includes multiple low-rank adaptation branches corresponding to different camera labels, and selects the appropriate low-rank adaptation branch to participate in network computation based on the input camera label. (See also...) Figure 4Specifically, it includes the following: S3.1 Structural Design of Camera-Sensing Low-Rank Adaptation Module: Let a certain trainable weight matrix in the inverse ISP network be denoted as... Then, for the i-th camera, the corresponding camera-perception low-rank adaptation update can be expressed as:

[0047] in, , Therefore, for each camera, only the following needs to be added: Only one parameter is needed to achieve camera-specific adaptation, thereby significantly reducing the number of new parameters while ensuring adaptation capability; S3.2 Camera Label-Driven Branch Selection Mechanism: During training and inference, camera labels are used to select the corresponding low-rank adaptation branches. For the current input sample, only the adaptation branch corresponding to its camera label is activated to participate in network computation. Specifically, during the training phase, the shared weights in the backbone network... Updates are performed based on all training data to learn the inverse ISP features common to multiple cameras; while the low-rank update term corresponds to each camera. It is updated only when corresponding camera sample input is received, thereby learning the imaging features specific to that camera. In a preferred embodiment, to ensure training stability, the low-rank matrix is ​​initialized as follows: the matrix... Initialize as a zero matrix, then set the matrix as zero. Initialize using a Gaussian random distribution; S3.3 Module Embedding Location: In a preferred embodiment, the inverse ISP network adopts a U-Net structure including convolutional modules and attention modules, and combines it with SwingTransformer blocks to construct the backbone network. A camera-aware low-rank adaptation module is embedded in the attention module to perform camera-related modulation on the linear projection weights in the attention layer. Further, the camera-aware low-rank adaptation module preferably operates on the query projection matrix, key projection matrix, value projection matrix, and output projection matrix in the SwingTransformer, to enhance the network's ability to represent different camera ISP characteristics while maintaining the uniformity of the backbone structure. In a preferred embodiment, the rank of the low-rank decomposition... Take 8.

[0048] S4. During the training phase, the inverse ISP network is trained based on the forward probability distribution of the spiral diffusion model. During the sampling phase, the RGB image and camera label are input into the trained inverse ISP network, and the target RAW image is obtained through a reverse iterative sampling process based on the spiral diffusion model. This includes the following: S4.1 During the training phase, a set of triples is randomly sampled from the training sample set. ,in Represents the target RAW image. This represents the corresponding RGB image. This indicates the camera label. Subsequently, random sampling time steps... And sample standard Gaussian noise According to the residual term And a time-varying weighted graph, constructing noise samples at the current time step:

[0049] in, This is the weight map after bias correction.

[0050] Then, the noise samples , corresponding RGB image Time step and camera tags Inputting both images into the inverse ISP network yields the predicted RAW image for the current time step.

[0051] in, The parameter is A noise reduction network.

[0052] In this embodiment, the network directly predicts the target RAW image, rather than predicting noise. The network parameters are then updated based on the difference between the predicted RAW image and the target RAW image.

[0053] S4.2 Loss Function Design: In a preferred embodiment, the loss function adopts a combined form of mean squared error loss, L1 loss, and logarithmic L1 loss, and its expression can be expressed as:

[0054] Furthermore, it can be expressed as:

[0055] in, A tiny constant used to prevent abnormal values ​​in logarithmic operations.

[0056] By using the aforementioned combined loss, the overall reconstruction error, pixel-level absolute deviation, and dark area relative error can be taken into account simultaneously, thereby improving the quality of RAW reconstruction.

[0057] (3) Sampling process During the sampling phase, input the RGB image to be converted. and its camera tag First, an initial state is constructed. Specifically, Gaussian noise is sampled. And obtain the initial weight map based on the input RGB image:

[0058] Thus, the initial state is obtained:

[0059] Then, iterate in reverse from time step T to time step 1. At each time step... , the current state Input RGB image Current time step and camera tags Input the trained inverse ISP network to obtain the predicted RAW image for the current time step:

[0060] Update the residuals based on the prediction results:

[0061] And further update the weight graph at the current time step:

[0062]

[0063] Then, the backsampling coefficients are calculated based on the updated weight graph:

[0064] This leads to the mean and variance of the backsampling distribution:

[0065]

[0066] Finally, the state of the previous time step is obtained by sampling based on the mean and variance:

[0067] Repeat the above process until all time steps are completed to obtain the final target RAW image.

[0068] Example 2: Based on Example 1 but with a difference, quantitative experiments were conducted on four camera-related test sets, including FiveK Canon, FiveK Nikon, NOD Nikon, and NOD Sony. Experiments were performed in both independent and joint training settings, using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as evaluation metrics. In the experiments, the method of this invention was compared with existing RGB-to-RAW conversion methods, including CycleISP, InvISP, ReRAW, and RAW-Difusion. The joint training setting further verified the performance changes after introducing a camera-aware low-rank adaptation module; the relevant results are shown in Tables 1 and 2.

[0069] Table 1: Performance comparison of different methods under independent training settings, with PSNR (↑) and SSIM (↑) as the evaluation metrics.

[0070] Note: The optimal result is marked in bold.

[0071] Table 2: Performance comparison of different methods under joint training settings, with PSNR (↑) and SSIM (↑) as the evaluation metrics.

[0072] Note: The optimal result is marked in bold.

[0073] As shown in Table 1, under the independent training setting, the spiral diffusion model proposed in this invention achieved the best quantitative results on all four test sets. This indicates that the spiral diffusion model based on pixel intensity-related time-varying weight map proposed in this invention can more effectively characterize the signal-related uncertainty in RGB to RAW conversion compared with the existing diffusion method that uses unified noise modeling, thereby improving the accuracy of inverse ISP reconstruction.

[0074] Furthermore, under the joint training setting, the overall performance is somewhat lower than that of the independent training setting because a single model needs to adapt to multiple camera sensors and their corresponding ISP characteristics simultaneously. However, after introducing the camera-aware low-rank adaptation module proposed in this invention into the joint training model, the PSNR and SSIM metrics on each test set are significantly improved, and the results are close to those of the corresponding independent training settings. This indicates that the camera-aware adaptation mechanism proposed in this invention can effectively modulate the model based on the input camera labels, thereby enhancing the model's adaptability to different camera imaging characteristics and significantly improving the inverse ISP conversion performance in multi-camera scenarios.

[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An inverse ISP method based on a spiral diffusion model and camera perception adaptation, characterized in that, Includes the following steps: S1. Construct inverse ISP training samples, each training sample including an RGB image, a target RAW image corresponding to the RGB image, and a camera label; S2. Construct an inverse ISP network based on a spiral diffusion model. The spiral diffusion model introduces a time-varying weight map related to pixel intensity at different time steps in the diffusion process to adjust the noise perturbation intensity of different pixel positions at each time step. S3. Set up a camera-aware low-rank adaptation module in the reverse ISP network. The camera-aware low-rank adaptation module includes multiple low-rank adaptation branches corresponding to different camera labels, and selects the corresponding low-rank adaptation branch to participate in network calculation according to the input camera label. S4. During the training phase, the inverse ISP network is trained based on the forward probability distribution of the spiral diffusion model. During the sampling phase, the RGB image and camera label are input into the trained inverse ISP network, and the target RAW image is obtained through a reverse iterative sampling process based on the spiral diffusion model.

2. The method according to claim 1, characterized in that, Let the input RGB image be The target RAW image is The residual term is: in, Residual information between RGB and RAW images is used to characterize the signal changes introduced during the RAW to RGB conversion process; The spiral diffusion model constructs a time-varying weight map related to pixel intensity: in, Represents a time-varying weighted graph; Indicates a time step; This represents the shift coefficient that changes with the time step. Differential perturbations are applied to different brightness regions, with smaller perturbation intensities for low-intensity regions and larger perturbation intensities for high-intensity regions, so that the diffusion process matches the reconstruction difficulty of different regions in the RGB to RAW conversion.

3. The method according to claim 2, characterized in that, The time-varying weight map changes dynamically with the diffusion time step and is incorporated into the residual diffusion model to construct the forward probability distribution of the spiral diffusion model: in, Indicates a Gaussian distribution. , This is the overall scaling factor. It is the identity matrix; The reverse probability distribution of the spiral diffusion model is as follows: in, and Let these represent the mean and variance, respectively, and their calculation formulas are as follows: in, This represents the coefficient related to the weight.

4. The inverse ISP method according to claim 1, characterized in that, The camera-sensing low-rank adaptation module updates the shared network weights through low-rank decomposition; for the i-th camera, its corresponding weight matrix is ​​represented as: in, For shared weight matrix; and This is the low-rank adaptation parameter matrix corresponding to the i-th camera.

5. The inverse ISP method according to claim 1, characterized in that, The inverse ISP network adopts an encoder-decoder structure that includes an attention module. A camera-aware low-rank adaptation module is embedded in the attention module to perform camera-related modulation on the network weights.

Images