A combined super-resolution and low-light enhancement method based on RAW and sRGB images

Abstract

This invention relates to the field of image processing technology, and provides a method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images. The method includes: degrading a high-quality RAW image to obtain a low-quality RAW image and a low-quality sRGB image; performing maximum entropy downsampling on the high-quality RAW image to generate a fully sampled linear RAW image, and then generating a high-quality sRGB image through ISP pipeline rendering; constructing a joint super-resolution and low-light enhancement processing network; and training the joint super-resolution and low-light enhancement processing network using the low-quality RAW image, the low-quality sRGB image, and the high-quality sRGB image. This invention solves the problems of missing training datasets and mismatch between reference images and low-light, low-resolution input images in joint image super-resolution and low-light enhancement processing, and overcomes the limitations of traditional processing methods that use only a single sRGB or RAW image, thereby achieving higher image quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for combined super-resolution and low-light enhancement processing of RAW and sRGB images. Background Technology

[0002] Super-resolution (SR) technology enables digital zoom, allowing distant objects in images captured with short focal length lenses to be clearly displayed. However, in low-light environments, distant objects in short focal length images suffer not only from resolution degradation but also from a series of complex degradations, such as noise contamination, blurring, and color shift. Most existing super-resolution algorithms are trained on datasets under normal lighting conditions, making it difficult to adapt directly to low-light environments. This makes it challenging to simultaneously enhance illumination and suppress noise while improving resolution.

[0003] To address the aforementioned issues, a potential approach is to directly cascade Low-Light Image Enhancement (LLIE) and the SR algorithm. Since LLIE enhances illumination while suppressing noise, it typically tends to produce a smoother enhanced image. This significantly impacts the cascaded SR algorithm, leading to further accumulation and spread of errors.

[0004] Furthermore, most existing LLIE and SR methods are performed directly on sRGB (standard Red Green Blue) images. On one hand, because sRGB images have undergone image signal processing (ISP) pipeline processing, each channel only has 8 bits of information. On the other hand, because the ISP pipeline processing includes non-linear transformations such as tone mapping, gamma transformation, and JPEG compression, the pixel values ​​of sRGB images are no longer linearly related to light intensity. This makes it impossible to obtain optimal results when performing joint SR and LLIE processing on sRGB images. Additionally, existing super-resolution datasets are acquired using digital cameras with zoom lenses, such as SR-RAW. Due to depth-of-field and distortion issues caused by lens focal length changes, misalignment between the input and reference images is inevitable. This necessitates additional consideration of image registration for datasets acquired in this way.

[0005] RAW image data, recorded directly from a camera, is typically 12 to 14 bits, offering more detail and a higher dynamic range, and exhibiting a linear relationship with scene brightness. This makes RAW images more suitable for low-level image processing compared to sRGB images. While many SR datasets based on RAW images exist, these datasets cannot be used to train LLIE algorithm models. Furthermore, RAW images do not contain any information from the ISP pipeline, and key parameters of crucial processes within the camera ISP pipeline, such as color lookup and tone mapping, are usually strictly confidential by each camera manufacturer. This means that using only RAW images for joint SR and LLIE processing cannot accurately render the color style corresponding to the target camera model.

[0006] In summary, existing technologies lack methods for generating datasets that simultaneously contain low-light, low-resolution RAW and sRGB images with perfectly aligned inputs and outputs, and also cannot effectively fuse complementary information from RAW and sRGB images. Therefore, there is still room for improvement in the quality of reconstructed images.

[0007] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method for super-resolution and low-light enhancement of RAW and sRGB images.

[0009] The present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images, comprising:

[0011] In step 201, the high-quality RAW image X is processed. ori Degradation processing is performed to obtain a low-quality RAW image X. raw and low-quality sRGB images X srgb ; wherein, the high-quality RAW image X ori The low-quality RAW image X is a high-resolution RAW image with normal lighting. raw The image is a low-light, low-resolution RAW image, specifically a low-quality sRGB image X. srgb This is a low-light, low-resolution sRGB image.

[0012] In step 202, the high-quality RAW image X is processed. ori Maximum entropy downsampling is performed to generate a fully sampled linear RAW image, which is then rendered using an ISP pipeline to produce a high-quality sRGB image. srgbThe high-quality sRGB image is a high-resolution sRGB image under normal lighting conditions.

[0013] In step 203, a joint processing network for super-resolution and low-light enhancement is constructed; the low-quality RAW image X is then used. raw The low-quality sRGB image X srgb and the high-quality sRGB image Y srgb Generate a dataset; use the dataset to train the super-resolution and low-light enhancement joint processing network to obtain a trained super-resolution and low-light enhancement joint processing network for image enhancement.

[0014] The high-quality RAW image X ori Degradation processing is performed to obtain a low-quality RAW image X. raw and low-quality sRGB images X srgb Specifically, it includes:

[0015] For the high-quality RAW image X ori The illumination degradation process was performed sequentially using normalization, inverse white balance, simulated low illumination, and denormalization to obtain the RAW image X after illumination degradation. dark ;

[0016] X-rays of RAW images after illumination degradation dark Perform a resolution degradation operation to obtain a low-quality RAW image X. raw ;

[0017] For the low-quality RAW image X raw Perform ISP pipeline rendering to obtain a low-quality sRGB image X. srgb .

[0018] The simulated low light is achieved by setting an exposure value, and the pixel value of the image obtained after simulating low light is y = x·2. EV Where x is the normalized and inverse white balance image pixel value, and EV is the exposure value.

[0019] The RAW image X after illumination degradation dark Perform a resolution degradation operation to obtain a low-quality RAW image X. raw Specifically, it includes:

[0020] The maximum entropy downsampling method was used to process the RAW image after illumination degradation. dark Sampling is performed to obtain a fully sampled linear RAW image X. lin ;

[0021] For fully sampled linear RAW images X linAdd out-of-focus blur and motion blur sequentially, then perform downsampling and Bayer sampling, and finally add noise to obtain a low-quality RAW image X. raw =f Bayer (f down (X lin *k def *k mot ))+n;

[0022] Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) indicates a downsampling operation; k def and k mot These represent the defocus blur kernel and the dynamic blur kernel, respectively; * represents the convolution operation; and n represents heteroscedastic Gaussian noise.

[0023] The added noise specifically refers to the addition of shooting noise and reading noise.

[0024] Where I represents a noise-free image, The image after noise has been added is represented by N(·), which represents the normal distribution function, where 0 represents the mean, β1I+β2 represents the variance, and β1 and β2 are the shot noise parameter and read noise parameter, respectively.

[0025] The super-resolution and low-light enhancement joint processing network includes an initial enhancement sub-network and a wavelet enhancement sub-network based on discrete wavelet transform;

[0026] The initial enhancement subnetwork is used for low-quality RAW images X. raw Feature extraction is performed to obtain the initial features F in the RAW domain. raw And for low-quality sRGB images X srgb Feature extraction is performed to obtain the initial sRGB domain features F. srgb ;

[0027] The wavelet enhancement subnetwork uses discrete wavelet transform to transform the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Transforming to the frequency domain, a frequency fusion module is used to fuse and enhance the two types of features, followed by enhancement using a feature enhancement module. Next, a sub-pixel layer is used for upsampling to improve spatial resolution, and finally, an inverse discrete wavelet transform is used to convert the image into an image enhancement result.

[0028] The initial enhancement subnetwork includes a RAW image initial enhancement module and an sRGB image initial enhancement module;

[0029] The RAW image initial enhancement module will X-ray the low-quality RAW image. rawPacked into a 4-channel RGGB format, resulting in X. pack ; Where H and W represent the height and width of the input RAW image, respectively;

[0030] X pack The RAW features are fed into the feature enhancement module to complete feature extraction and enhancement. Then, the extracted and enhanced RAW features are upsampled by a factor of 2 to obtain the initial features in the RAW domain.

[0031] The sRGB image initial enhancement module will transform the low-quality sRGB image X srgb The data is fed into the feature enhancement module to obtain the initial enhanced sRGB features.

[0032] The wavelet enhancement subnetwork uses discrete wavelet transform to transform the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Transforming to the frequency domain, the frequency fusion module is used to fuse and enhance the two types of features, specifically including:

[0033] The wavelet enhancement subnetwork applies the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Each component was transformed to the frequency domain using wavelet transform, yielding four frequency components. and Where i = 0 represents the low-frequency component LL, and i = 1, 2, 3 represent the three high-frequency components LH, HL and HH respectively;

[0034] Same frequency and The signal is fed into the frequency fusion module to complete the feature fusion and enhancement.

[0035] The same frequency and The signal is fed into the frequency fusion module to complete the feature fusion and enhancement, specifically including:

[0036] For the same frequency and The layers are then normalized sequentially, followed by a 1×1 convolutional layer and a 3×3 separable convolutional layer to obtain the key matrix K, value matrix V, and query matrix Q. The shapes of the key matrix K, value matrix V, and query matrix Q are then adjusted as follows: To conform to the rules of matrix multiplication;

[0037] Apply matrix multiplication to the shape-adjusted key matrix K and the shape-adjusted query matrix Q, and obtain the attention map using the Softmax function. Right now Where τ is a learnable parameter;

[0038] Shape-adjusted value matrix V and attention map Multiply and then combine the results with... The scaling factor α and offset factor β of the affine transformation are generated by adding them together through two independent 1×1 convolutional layers;

[0039] The frequency components of sRGB are modulated using scaling factor α and offset factor β. Obtain the fused wavelet features Where * represents element-wise multiplication, and Conv1(·) represents a 1×1 convolutional layer.

[0040] The enhancement process involves using a feature enhancement module, followed by upsampling at the sub-pixel level to improve spatial resolution, and finally, inverse discrete wavelet transform to convert the image into an enhanced result. Specifically, it includes:

[0041] Will and The sums are then fed into a feature enhancement module for feature enhancement. The enhanced output is then upsampled using a 3×3 convolution combined with a sub-pixel layer to achieve resolution enhancement.

[0042] After the four frequency components i = 0, 1, 2, and 3 have been fused, enhanced, and upsampled, an inverse wavelet transform is applied to convert them to the spatial domain. A 3×3 convolutional layer is then applied to the output of the inverse transform to finally output the image enhancement result.

[0043] This invention solves the problems of missing training datasets and mismatch between reference images and low-light, low-resolution input images in joint image super-resolution and illumination enhancement processing. It also overcomes the limitations of traditional processing methods that process with a single sRGB or RAW image, thereby achieving higher image quality. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0045] Figure 1 This is a flowchart illustrating a method for joint processing of super-resolution and low-light enhancement based on RAW and sRGB images provided in an embodiment of the present invention.

[0046] Figure 2This is a schematic diagram of the synthetic image processing pipeline in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the super-resolution and low-light enhancement joint processing network in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the initial enhancement module of RAW image in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the feature enhancement module in a joint processing method for super-resolution and low-light enhancement based on RAW and sRGB images provided in an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram of the residual channel attention group in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided in an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of the initial enhancement module of the sRGB image in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided in an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram of the frequency fusion module in a joint processing method for super-resolution and low-light enhancement based on RAW and sRGB images provided in an embodiment of the present invention.

[0053] Figure 9 This is a flowchart illustrating a method for joint processing of super-resolution and low-light enhancement based on RAW and sRGB images provided in an embodiment of the present invention.

[0054] Figure 10 This is a schematic diagram of a low-light, low-resolution sRGB input image, provided by an embodiment of the present invention, for a joint processing method of super-resolution and low-light enhancement based on RAW and sRGB images.

[0055] Figure 11 This is a schematic diagram illustrating the application effects of various existing methods in a joint processing method for super-resolution and low-light enhancement of RAW and sRGB images provided in an embodiment of the present invention.

[0056] Figure 12This is a schematic diagram illustrating the application effect of a combined super-resolution and low-light enhancement processing method based on RAW and sRGB images provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0059] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0060] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0061] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0062] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] Example 1:

[0064] To reduce error accumulation and diffusion, joint processing of Super-Resolution (SR) and Low-Illumination Image Enhancement (LLIE) can be performed. While joint processing methods exist for LLIE and other image restoration problems, such as deblurring, dehazing, or deraining, joint processing methods for SR and LLIE are rare. Existing technologies lack methods for generating datasets that simultaneously contain low-light, low-resolution RAW and sRGB images with perfectly aligned input and output, and also cannot effectively fuse complementary information from RAW and sRGB images. Therefore, there is still room for improvement in the quality of the reconstructed image. To address this issue, Embodiment 1 of this invention provides a joint processing method for super-resolution and low-light enhancement based on RAW and sRGB images, such as... Figure 1 As shown, it includes:

[0065] In step 201, the high-quality RAW image X is processed. ori Degradation processing is performed to obtain a low-quality RAW image X. raw and low-quality sRGB images X srgb ; wherein, the high-quality RAW image X ori The low-quality RAW image X is a high-resolution RAW image with normal lighting. raw The image is a low-light, low-resolution RAW image, specifically a low-quality sRGB image X. srgb This is a low-light, low-resolution sRGB image.

[0066] In step 202, the high-quality RAW image X is processed. ori Maximum entropy downsampling is performed to generate a fully sampled linear RAW image, which is then rendered using an ISP pipeline to produce a high-quality sRGB image. srgb Wherein, the high-quality sRGB image is a high-resolution sRGB image under normal lighting conditions; the high-quality sRGB image Y srgb The image is used as a reference for supervised learning. In one alternative implementation, the ISP pipeline uses Adobe Camera Raw, which can simulate some of the complex operations in a real camera's ISP pipeline, such as tone mapping and color lookup.

[0067] In step 203, a joint processing network for super-resolution and low-light enhancement is constructed; the low-quality RAW image X is then used. raw The low-quality sRGB image X srgb and the high-quality sRGB image Y srgb A dataset is generated; the super-resolution and low-light enhancement joint processing network is trained using the dataset to obtain a trained super-resolution and low-light enhancement joint processing network for image enhancement. This dataset is also known as the DarkSR dataset. Specifically, training the super-resolution and low-light enhancement joint processing network using the dataset involves: X-raying low-quality RAW images from the dataset... raw and low-quality sRGB images X srgb The input is fed into the super-resolution and low-light enhancement joint processing network to obtain the output of the super-resolution and low-light enhancement joint processing network, i.e., the image enhancement result. Based on the image enhancement results and high-quality sRGB images Y in the dataset srgb Calculate the loss function, and use the loss function to iteratively train the super-resolution and low-light enhancement joint processing network until the network converges. In one optional implementation, the loss function may be: the image enhancement result. With the high-quality sRGB image Y srgb The L1 norm of the difference.

[0068] It should be noted that in this embodiment, high resolution and low resolution do not refer to a specific resolution, but are relative to each other. For example, in a practical application scenario, there is an image A with a resolution of 480×640, an image B with a resolution of 720×1048, and an image C with a resolution of 4K. For images A and B, image A has a low resolution and image B has a high resolution; for images B and C, image B has a low resolution and image C has a high resolution. Similarly, in this embodiment, normal lighting and low lighting do not refer to a specific type of lighting, but are relative terms. For example, in one optional implementation, normal lighting can be the lighting conditions for taking pictures outdoors during the day, while low lighting can be the lighting conditions for taking pictures at night or in low-light environments (such as a dark indoor space). Taking a picture A with an illuminance of 0.2 lux, a picture B with an illuminance of 3 lux, and a picture C with an illuminance of 5 lux as an example, then for pictures A and B, the lighting conditions for picture A are low lighting, and for picture B, the lighting conditions for picture C are normal lighting. Similarly, for pictures B and C, the lighting conditions for picture B are low lighting, and for picture C, the lighting conditions for picture C are normal lighting. In practical use, a daytime photo of the same scene can be used as a normal lighting image, and a nighttime photo of the same scene can be used as a low lighting image. The high and low resolution of the image are determined by the photographer setting the appropriate resolution during shooting.

[0069] This embodiment proposes a joint super-resolution and low-light enhancement processing method based on RAW and sRGB images. By simulating the illumination and resolution degradation process and combining it with a simulated ISP pipeline, high-resolution RAW images under normal illumination are degraded into low-light, low-resolution RAW images and sRGB images. At the same time, the corresponding normal-light and high-resolution sRGB images are used as reference images to construct a training dataset. Based on the constructed training dataset, a joint super-resolution and low-light enhancement processing network is established, which takes low-light, low-resolution RAW and sRGB images as inputs and performs end-to-end fully supervised training. This solves the problems of missing training datasets and mismatch between reference images and low-light, low-resolution input images in joint image super-resolution and illumination enhancement processing. It also overcomes the limitations of traditional processing methods that process only a single sRGB or RAW image, thus achieving higher image quality.

[0070] Steps 201 and 202 can be achieved by establishing, for example... Figure 2 The image processing pipeline shown is implemented as described above, specifically the processing of high-quality RAW images X. ori Degradation processing is performed to obtain a low-quality RAW image X. rawand low-quality sRGB images X srgb Specifically, it includes:

[0071] For the high-quality RAW image X ori The illumination degradation process was performed sequentially using normalization, inverse white balance, simulated low illumination, and denormalization to obtain the RAW image X after illumination degradation. dark ;in, The normalization process involves mapping pixel values ​​to values ​​between 0 and 1. The inverse white balance is used to remove the influence of the light source. The simulated low-light effect is achieved by setting the exposure value. The pixel value of the image obtained after simulating low light is y = x·2. EV Where x is the normalized and inverse white balance image pixel value, and EV is the exposure value. Reducing the illumination means that EV is a number less than 0. In actual use, EV can be randomly selected from the range [-2.2, -1.8].

[0072] X-rays of RAW images after illumination degradation dark Perform a resolution degradation operation to obtain a low-quality RAW image X. raw ; for the low-quality RAW image X raw Perform ISP pipeline rendering to obtain a low-quality sRGB image X. srgb .

[0073] In one alternative implementation, the X-ray image after illumination degradation... dark Perform a resolution degradation operation to obtain a low-quality RAW image X. raw Specifically, it includes:

[0074] The maximum entropy downsampling method was used to process the RAW image after illumination degradation. dark Sampling is performed to obtain a fully sampled linear RAW image X. lin ;X lin This is a linear RAW image with 3 channels fully sampled. This avoids the errors caused by the "remove mosaic first and then Bayer sampling" method.

[0075] For fully sampled linear RAW images X lin Add out-of-focus blur and motion blur sequentially, then perform downsampling and Bayer sampling, and finally add noise to obtain a low-quality RAW image X. raw =f Bayer (f down (X lin *k def *k mot))+n; The downsampling can be 2x downsampling, and the sequential addition of out-of-focus blur and dynamic blur is to simulate the blur caused by increased focusing and exposure time when shooting in low-light environments.

[0076] Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) indicates a downsampling operation. In practical use, f down (·) can represent a 2x downsampling operation; k def and k mot represents the defocus blur kernel and the motion blur kernel, respectively; * represents the convolution operation; n represents heteroscedastic Gaussian noise. The size of the defocus blur kernel is a random integer in the interval [3, 5], and the size of the motion blur kernel is a random integer in the set {3, 5, 7}.

[0077] Specifically, the added noise includes shot noise and read noise, specifically:

[0078] Where I represents a noise-free image, The image with added noise is represented by N(·), which represents a normal distribution function, where 0 represents the mean, β1I+β2 represents the variance, and β1 and β2 are the shot noise parameter and read noise parameter, respectively. β1 and β2 are manually determined by a person skilled in the art through noise measurement of real nighttime images, and encapsulated to conform to the noise characteristics of short exposure and high ISO images at night. Specifically, the noise is heteroscedastic Gaussian noise. The shot and read noise parameters are manually set by measuring real nighttime image pairs to conform to the noise characteristics of high ISO and short exposure times at night. Encapsulating β1 and β2 as ISO noise is a prior art technique. The value of the ISO noise is a random integer in the range [400, 1600].

[0079] In practical applications, the super-resolution and low-light enhancement joint processing network includes an initial enhancement sub-network and a wavelet enhancement sub-network based on discrete wavelet transform, such as... Figure 3 As shown.

[0080] The initial enhancement subnetwork is used for low-quality RAW images X. raw Feature extraction is performed to obtain the initial features F in the RAW domain. raw And for low-quality sRGB images X srgb Feature extraction is performed to obtain the initial sRGB domain features F. srgb .

[0081] The wavelet enhancement subnetwork uses discrete wavelet transform to transform the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgbTransforming to the frequency domain, a frequency fusion module is used to fuse and enhance the two types of features, followed by enhancement using a feature enhancement module. Next, a sub-pixel layer is used for upsampling to improve spatial resolution, and finally, an inverse discrete wavelet transform is used to convert the image into an image enhancement result. In practical use, the upsampling here can be 2 times the upsampling. The frequency fusion module is also called the FFM module, and the feature enhancement module is also called the FEM module.

[0082] The initial enhancement subnetwork includes a RAW image initial enhancement module and an sRGB image initial enhancement module; the RAW image initial enhancement module is also referred to as the upper branch, and the sRGB image initial enhancement module is also referred to as the lower branch.

[0083] like Figure 4 As shown, the RAW image initial enhancement module includes, from input to output, a packing module, a 3x3 convolutional layer, a feature enhancement module, and another 3x3 convolutional layer. The RAW image initial enhancement module processes low-quality RAW images X... raw Packed into a 4-channel RGGB format, resulting in X. pack ; Where H and W represent the input RAW image (i.e., the low-quality RAW image X) respectively. raw The height and width of the low-quality RAW image, H, are the values ​​of the low-quality RAW image X. raw High, W represents low-quality RAW image X raw The width.

[0084] X pack The RAW features are fed into the feature enhancement module to complete feature extraction and enhancement. Then, the extracted and enhanced RAW features are upsampled by a factor of 2 to obtain the initial features in the RAW domain. like Figure 5 As shown, the feature enhancement module includes two cascaded residual channel attention groups (RCAGs), such as... Figure 6 As shown, a residual channel attention block consists of four residual channel attention blocks (RCABs). RCABs are an existing neural network technique and will not be elaborated upon here. The 2x upsampling is performed to complete the subsequent fusion and enhancement with sRGB features. Here, C is the number of feature channels. In an optional implementation, C = 32.

[0085] like Figure 7 As shown, the sRGB image initial enhancement module includes a 3x3 convolutional layer and a feature enhancement module sequentially from input to output. The sRGB image initial enhancement module transforms the low-quality sRGB image X... srgb The data is fed into the feature enhancement module to obtain the initial enhanced sRGB features.

[0086] In some embodiments, the frequency fusion module is as follows: Figure 8 As shown, the wavelet enhancement subnetwork uses discrete wavelet transform to enhance the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain ssrgb Transforming to the frequency domain, the frequency fusion module is used to fuse and enhance the two types of features, specifically including:

[0087] The wavelet enhancement subnetwork applies the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Each component was transformed to the frequency domain using wavelet transform, yielding four frequency components. and Where i = 0 represents the low-frequency component LL, and i = 1, 2, 3 represent the three high-frequency components LH, HL, and HH, respectively; for the initial feature F in the RAW domain raw Transform to the frequency domain using wavelet transform to obtain Initial features F in the sRGB domain srgb Transform to the frequency domain using wavelet transform to obtain

[0088] Same frequency and The data is fed into the frequency fusion module to complete feature fusion and enhancement. Since low-frequency components represent global information of the image, such as brightness and color, while high-frequency components represent structural and detail information, enhancing different frequency components individually helps to efficiently achieve brightness and detail enhancement. Furthermore, because RAW and sRGB images have a high degree of correlation at the same spatial location, the same frequency components of RAW and sRGB images can be fused together. and Fusion facilitates the efficient capture of complementary information. The frequency fusion module consists of affine transformation and cross-attention; cross-attention can effectively capture… and The long-range dependencies are used to construct the scaling factor α and offset factor β of the affine transformation, which can break through the limited receptive field of the original affine transformation.

[0089] The same frequency and The signal is fed into the frequency fusion module to complete the feature fusion and enhancement, specifically including:

[0090] For the same frequency and Layer normalization, 1×1 convolutional layer, and 3×3 separable convolutional layer are executed sequentially to obtain key matrix K, value matrix V, and query matrix Q; among them, RAW features generate key matrix K and value matrix V, and sRGB features generate query matrix Q.

[0091] Adjust the shapes of the key matrix K, value matrix V, and query matrix Q to: To comply with the rules of matrix multiplication.

[0092] Apply matrix multiplication to the shape-adjusted key matrix K and the shape-adjusted query matrix Q, and obtain the attention map using the Softmax function. Right now Where τ is a learnable parameter.

[0093] Shape-adjusted value matrix V and attention map Multiply them and then multiply the result (i.e., the result of the multiplication) with... The scaling factor α and offset factor β of the affine transformation are generated by adding them together through two independent 1×1 convolutional layers.

[0094] The frequency components of sRGB are modulated using scaling factor α and offset factor β. Obtain the fused wavelet features Where ⊙ represents element-wise multiplication, and Conv1(·) represents a 1×1 convolutional layer. This can be understood as follows: First, for the frequency components after discrete wavelet transform of the initial RAW and sRGB features, calculate the cross-attention between the corresponding frequency components; second, use the features after cross-attention to calculate a scale parameter and an offset parameter; finally, apply the scale parameter and offset parameter to the wavelet frequency components of the sRGB features to achieve the fusion and enhancement of RAW and sRGB features in the wavelet transform domain.

[0095] The enhancement process involves using a feature enhancement module, followed by upsampling at a sub-pixel layer to improve spatial resolution, and finally, inverse discrete wavelet transform to convert the image into an enhanced result. Specifically, it includes:

[0096] Will and The sums are then fed into a feature enhancement module for feature enhancement. The enhanced output is then upsampled using a 3×3 convolution combined with a sub-pixel layer to achieve resolution enhancement. In practice, the upsampling here can be 2 times the upsampling.

[0097] After the four frequency components i = 0, 1, 2, and 3 have been fused, enhanced, and upsampled, an inverse wavelet transform is applied to convert them to the spatial domain. A 3×3 convolutional layer is then applied to the output of the inverse transform to finally output the image enhancement result.

[0098] This embodiment designs a degradation pipeline from high-resolution RAW images under normal lighting to generate image datasets for low-light, low-resolution images, used for training and evaluating algorithm performance. On the other hand, it designs a joint super-resolution and low-light enhancement processing network. Its main feature is that it simultaneously takes low-light, low-resolution RAW and sRGB images as input. After extracting features from both types of image inputs, wavelet transform is used to convert the images to the frequency domain, and an FFM module is used to fuse the two types of wavelet domain features. By fully utilizing the complementary information between the two types of images, high-quality image enhancement is achieved with relatively low computational complexity.

[0099] Example 2:

[0100] Based on the method described in Embodiment 1, this invention combines specific application scenarios and uses technical descriptions in relevant scenarios to illustrate the implementation process of the features of this invention in those scenarios.

[0101] This embodiment uses Adobe Camera Raw software as the ISP pipeline to simulate the ISP pipelines for 20 camera models from four camera brands, including Canon, Sony, Leica, and Nikon, generating corresponding sRGB image styles to enrich the types and models of cameras in the training and test sets. For example, this embodiment describes a joint super-resolution and low-light enhancement processing method based on RAW and sRGB images. Figure 9 As shown, it specifically includes:

[0102] In step 301, the high-resolution RAW image with normal exposure (i.e., normal lighting) is... Degradation processing to generate low-resolution RAW images with low illumination. and sRGB images Where H and W represent the height and width of the image or feature, respectively. Specifically: For a normally exposed high-resolution RAW image X... ori Low-light, low-resolution RAW images are synthesized through low-light degradation (including normalization, inverse white balance, simulated low light, and denormalization), maximum entropy downsampling, resolution degradation (including out-of-focus blur, motion blur, 2x downsampling, Bayer sampling, and noise addition), and ISP pipeline rendering. raw and sRGB image X srgb .

[0103] Step 301 specifically includes:

[0104] (1) For the input high-resolution RAW image with normal exposure X oriFirst, illumination degradation is performed. Illumination degradation mainly utilizes the linear relationship between the pixel values ​​of the RAW image and the light intensity of the actual scene to linearly adjust the pixel values ​​of the RAW image. X ori Normalization involves mapping pixel values ​​to values ​​between 0 and 1. Inverse white balance is used to remove the influence of light sources, and negative EV values ​​are set to simulate low-light effects. Denormalization is then performed to obtain a high-resolution RAW image with degraded illumination. In this invention, the EV value is randomly selected from the interval [-1.8, -2.2], that is:

[0105] y = x·2 EV (1)

[0106] Where x is the normalized and inverse white balance image pixel value, EV is the set exposure value, and y is the image pixel value after the illumination is reduced.

[0107] (2) For the output result X of (1) under low light degradation dark To perform resolution degradation: first, maximum entropy downsampling is used to obtain a 3-channel fully sampled linear RAW image. This avoids the errors associated with the "de-mosaicing followed by Bayer sampling" method. Next, out-of-focus blur and motion blur are added sequentially to simulate the blurring caused by increased focus and exposure time in low-light environments. Then, 2x downsampling and Bayer sampling are performed. Finally, noise is added to obtain a RAW image X with degraded lighting and resolution. raw As shown in equation (2), it is from X lin Degenerate into X raw The process:

[0108] X raw =f Bayer (f down (X lin *k def *k mot ))+n (2)

[0109] Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) indicates a 2x downsampling operation; k def and k mot represents the defocus blur and dynamic blur kernels respectively; * represents the convolution operation; n represents heteroscedastic Gaussian noise, as shown in equation (3), and the noise is added as follows:

[0110]

[0111] Where I represents a noise-free image, This represents the image after noise has been added, where β1 and β2 are the shot noise and read noise parameters, respectively. β1 and β2 are manually determined by measuring noise in real nighttime images, and the image is encapsulated to conform to the noise characteristics of short exposure, high ISO images at night. Since encapsulating β1 and β2 as ISO noise is existing technology, it will not be elaborated further.

[0112] (3) The degraded RAW image is rendered using an ISP pipeline to generate a corresponding low-light, low-resolution sRGB image X. srgb Maximum entropy downsampling is an existing technology and will not be elaborated here. In this invention, the blur kernel size for defocus blur is a random integer in the interval [3, 5], and the blur kernel size for motion blur is a random integer in the set {3, 5, 7}. The value of ISO noise is a random integer in the interval [400, 1600]. The ISP pipeline selected is Adobe Camera Raw, which can simulate some complex operations in the ISP pipeline of a real camera, such as tone mapping and color lookup.

[0113] In step 302, in order to perform supervised learning, it is also necessary to generate the corresponding high-resolution sRGB image under normal lighting. As a reference image. Specifically, for the input, a normally exposed high-resolution RAW image X ori Maximum entropy downsampling and ISP pipeline rendering are performed sequentially to generate a high-resolution sRGB image Y with normal lighting. srgb .

[0114] In step 303, the low-resolution RAW image with low illumination generated in steps 301 and 302 is X-rayed. raw and low-resolution sRGB images in low light X srgb And normal lighting, high-resolution sRGB images Y srgb The DarkSR dataset was constructed, and a dual-input super-resolution and low-light enhancement joint processing network was built, which simultaneously accepts RAW and sRGB images as input and outputs high-resolution sRGB images with enhanced lighting. (i.e., image enhancement results). For example... Figure 3 As shown, the dual-input network contains two subnetworks: an initial enhancement subnetwork and a wavelet enhancement subnetwork.

[0115] Step 303 specifically includes:

[0116] (1) In the initial enhanced subnetwork, such as Figure 3 As shown, it is divided into two branches, with the upper branch performing initial feature enhancement on the RAW image and the lower branch performing initial feature enhancement on the sRGB image. Figure 4As shown, in the initial enhancement subnetwork of the RAW image in the upper branch, for the input RAW image X raw First, it is packaged into a 4-channel RGGB format. Where C is the number of channels in the feature. Further, X... pack The RAW features are fed into the Feature Enhancement (FEM) module to complete feature extraction and enhancement. To facilitate subsequent fusion and enhancement with sRGB features, the RAW features are upsampled by a factor of 2.

[0117] like Figure 7 As shown, in the initial sRGB image enhancement subnetwork of the lower branch, the input sRGB image is directly fed into the FEM to obtain the initial enhanced sRGB features. like Figure 5 As shown, FEM is composed of RCAG, as Figure 6 As shown, RCAG is composed of RCAB, which is an existing neural network technology and will not be described further. In this invention, C = 32.

[0118] (2) In the wavelet enhancement subnetwork, Haar wavelets are used to perform discrete wavelet transform on the initially enhanced RAW and sRGB image features to obtain wavelet domain features of the RAW and sRGB images. Since RAW and sRGB features with the same frequency components are highly correlated, FFM is used to fuse and enhance the same frequency components of the two types of features to achieve complementary advantages of RAW and sRGB features. The frequency fusion module combines attention mechanism and affine transformation. Affine transformation can effectively fuse features, but its performance is limited by the limited receptive field of ordinary convolution. First, cross-attention is used to capture the long-distance dependence between the two types of features, and then the obtained attention map is used to generate the scale factor and offset factor of affine transformation to effectively fuse RAW and sRGB features. The frequency-fused features and RAW wavelet features are residually connected and fed into FEM for further feature enhancement. Then, the enhanced output is improved by passing a 3×3 convolutional layer and a sub-pixel layer to improve the resolution of the wavelet features. After all frequency components have been enhanced and the resolution has been improved, the three-channel image after illumination enhancement and resolution improvement is reconstructed using the inverse wavelet transform function. Finally, to further reduce potential artifacts, a 3×3 convolutional layer is used to optimize the three-channel image reconstructed from the sub-pixel layer, resulting in the final output image. For example... Figure 3 As shown, for the RAW and sRGB features F generated in (1) raw and F srgb Each component was transformed to the frequency domain using wavelet transform, yielding four frequency components. and Where i = 0 represents the low-frequency component LL, and i = 1, 2, and 3 represent the three high-frequency components LH, HL, and HH, respectively. Since the low-frequency components represent global information of the image, such as brightness and color, while the high-frequency components represent structural and detail information, individually enhancing different frequency components helps to efficiently achieve brightness and detail enhancement. Furthermore, because RAW and sRGB images have a high degree of correlation at the same spatial location, for the same frequency components of RAW and sRGB... and Fusion helps to efficiently capture complementary information; therefore, combining signals of the same frequency... and The data is fed into the FFM module for efficient feature fusion and enhancement. The FFM module consists of affine transformation and cross-attention; cross-attention effectively captures... and The long-range dependencies are used to construct the scaling factor α and offset factor β of the affine transformation, which can break through the limited receptive field of the original affine transformation.

[0119] Specifically: such as Figure 8 As shown, the same frequency components of RAW and sRGB and Layer normalization, a 1×1 convolutional layer, and a 3×3 separable convolutional layer are executed sequentially. RAW features generate a key matrix K and a value matrix V, while sRGB features generate a query matrix Q, and their shapes are adjusted accordingly. This conforms to the matrix multiplication rule. Furthermore, matrix multiplication is applied to K and Q, and the attention map is obtained using the Softmax function. As shown in equation (4):

[0120]

[0121] Where τ is a learnable parameter.

[0122] Shape-adjusted value matrix V and attention map Multiply and then combine the results with... The addition generates scaling factors α and offset factors β for the affine transformation through two independent 1×1 convolutional layers. Furthermore, α and β are used to modulate the frequency components of sRGB. As shown in equation (5), the final FFM output is the fused wavelet feature.

[0123]

[0124] Where ⊙ represents element-wise multiplication, and Conv1(·) represents a 1×1 convolutional layer.

[0125] The output of FFM and The sums are then fed into the FEM module for final feature enhancement. The enhanced output is then upsampled by a factor of 2 using a 3×3 convolution combined with a sub-pixel layer to achieve resolution enhancement. After the four frequency components i = 0, 1, 2, and 3 have been fused, enhanced, and upsampled, an inverse wavelet transform is used to convert the frequency domain to the spatial domain. To reduce potential artifacts, a 3×3 convolutional layer is applied to the output of the inverse transform, ultimately outputting a high-resolution sRGB image with enhanced illumination.

[0126] In summary, through the above steps, on the one hand, a degradation pipeline is designed from high-resolution RAW images under normal lighting to generate image datasets for low-light, low-resolution images, used for training and evaluating algorithm performance; on the other hand, a joint super-resolution and low-light enhancement processing network is designed. Its main feature is that it simultaneously takes low-light, low-resolution RAW and sRGB images as input. After extracting features from both types of image inputs, wavelet transform is used to convert the images to the frequency domain, and an FFM module is used to fuse the two types of wavelet domain features. By fully utilizing the complementary information between the two types of images, high-quality image enhancement is achieved with relatively low computational complexity.

[0127] To illustrate the advantages of this embodiment, experiments were conducted using Python within the PyTorch framework. The method described in this embodiment was used to perform joint super-resolution and low-light enhancement processing on low-light, low-resolution RAW and sRGB images. This was compared with existing LLIE methods based on RAW image input (SID), SR methods (JDnDmSR), LLIE methods based on sRGB image input (CUE), and SR methods (SRFormer, abbreviated as SR-F), as well as SR methods based on dual inputs of RAW and sRGB images (RawSR and PRNet). Since existing LLIE methods cannot improve resolution, this embodiment also cascades the SR method SR-F after these methods to improve the output resolution.

[0128] Specifically, the performance of this invention is evaluated using the DarkSR dataset generated using the dataset synthesis method provided in this invention. DarkSR contains 1395 and 153 pairs of low-light, low-resolution RAW and sRGB images / normal-light, high-resolution sRGB images for training and testing, respectively. To train the proposed super-resolution and low-light enhancement joint processing network, the training process uses the L1 loss function, which measures the pixel-wise difference between the network's reconstructed output and the ground truth. RAW and corresponding sRGB images are randomly cropped into 256×256 image patches according to the RGGB Bayer pattern and then input into the network. No data augmentation is used. The batch size is set to 6, and the AdamW optimizer is used, with optimizer parameters β1 = 0.9, β2 = 0.99, and weight decay ω = 0.05. A warm-up strategy is used, with the learning rate linearly increased initially to 1.5×10⁻⁶. -4 Then, a cosine annealing strategy is used to decay the temperature to 6×10. -6 The entire training process lasted 500 epochs, and the results are shown in the table below:

[0129]

[0130] The table above presents the quantitative comparison results of algorithms with different input types on the DarkSR dataset. The comparison metrics are: Color Peak Signal to Noise Ratio (CPSNR), Structural Similarity Index (SSIM), CIELAB Delta E, Number of Parameters, Floating Point Operations (FLOPs), and Runtime. FLOPs and Runtime were measured on images with a resolution of 256×256. All experimental results were obtained by running on an NVIDIA GTX 3090 GPU. The optimal values ​​for each metric are represented in bold in the table.

[0131] As can be seen from the results in the table above, the method of this invention outperforms existing algorithms in most metrics. Compared with the suboptimal PRNet algorithm, the method of this invention requires 0.41M fewer parameters, and the FLOPs of the method of this invention are approximately half that of the PRNet algorithm.

[0132] Figure 10Given a low-light, low-resolution sRGB image on DarkSR that has undergone 2x Bicubic upsampling, various algorithms in existing technologies (including: SID-cascaded SR-F, JDnDmSR, CUE-cascaded SR-F, SR-F, RawSR, and PRNet methods) are used to process it. Figure 10 The enhanced image is as follows Figure 11 As shown, the method described in this embodiment is used to... Figure 10 The enhanced image is as follows Figure 12 As shown, where, Figure 10 The bottom left corner of the image shows the RGB curve of the image; similarly, Figure 11 The bottom left corner of each image in the table shows the RGB curve of that image. Figure 12 The lower left corner of the image shows the RGB curve of the image. It can be seen that, compared with other methods, the method of the present invention can maintain more texture structure and details, produce less distortion, and at the same time, the color reproduction effect of the image is better.

[0133] In summary, the method described in this invention, on the one hand, designs a degradation pipeline from normal-illumination, high-resolution RAW images to generate image datasets for low-illumination, low-resolution images, used for training and evaluating algorithm performance; on the other hand, it designs a joint super-resolution and low-illumination enhancement processing network, whose main feature is that it simultaneously takes low-illumination, low-resolution RAW and sRGB images as inputs. After extracting features from both types of image inputs, it uses wavelet transform to convert the images to the frequency domain, and uses an FFM module to fuse the two types of wavelet domain features. By fully utilizing the complementary information between the two types of images, high-quality image reconstruction (i.e., image enhancement) is achieved with relatively low computational complexity.

[0134] In summary, the technical solutions conceived by this invention, compared with the prior art, have the following advantages:

[0135] Beneficial effects:

[0136] The main advantages of this invention are: First, it can effectively simulate the processes of illumination degradation and resolution degradation, thereby constructing a dataset that can be used for end-to-end training, solving the problems of missing low-illumination, low-resolution training datasets or misalignment between low-quality input images and reference images; Second, the super-resolution and low-illumination enhancement joint processing network of this invention simultaneously uses low-resolution, low-illumination RAW and sRGB images as input, and uses wavelet transform and frequency fusion modules to fuse and enhance them, combining deep learning and wavelet transform. In the frequency domain after wavelet transform, it can effectively process detailed information in multiple directions of the image, and because... The resolution of each frequency component of wavelet transform is relatively small, so the use of attention mechanism will not generate a lot of computation. At the same time, since there is a strong correlation between the frequency components of RAW and sRGB images on the same spatial coordinates, this invention designs a fusion enhancement module for the same frequency components of RAW and sRGB images to achieve information complementarity between RAW and sRGB images. By combining the advantages of deep learning and wavelet transform, this invention achieves better performance than existing methods while maintaining a relatively small model size and low computational complexity, due to the efficient and accurate application of complementary information between RAW and sRGB images.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images, characterized in that, include: In step 201, the high-quality RAW image X is processed. ori Degradation processing is performed to obtain a low-quality RAW image X. raw and low-quality sRGB images X srgb ; wherein, the high-quality RAW image X ori The low-quality RAW image X is a high-resolution RAW image with normal lighting. raw The image is a low-light, low-resolution RAW image, specifically a low-quality sRGB image X. srgb This is a low-light, low-resolution sRGB image. In step 202, the high-quality RAW image X is processed. ori Maximum entropy downsampling is performed to generate a fully sampled linear RAW image, which is then rendered using an ISP pipeline to produce a high-quality sRGB image. srgb The high-quality sRGB image is a high-resolution sRGB image under normal lighting conditions. In step 203, a joint processing network for super-resolution and low-light enhancement is constructed; the low-quality RAW image X is then used. raw The low-quality sRGB image X srgb and the high-quality sRGB image Y srgb Generate a dataset; use the dataset to train the super-resolution and low-light enhancement joint processing network to obtain a trained super-resolution and low-light enhancement joint processing network for image enhancement. The super-resolution and low-light enhancement joint processing network includes an initial enhancement sub-network and a wavelet enhancement sub-network based on discrete wavelet transform; The initial enhancement subnetwork is used for low-quality RAW images X. raw Feature extraction is performed to obtain the initial features F in the RAW domain. raw And for low-quality sRGB images X srgb Feature extraction is performed to obtain the initial sRGB domain features F. srgb ; The wavelet enhancement subnetwork uses discrete wavelet transform to transform the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Transforming to the frequency domain, a frequency fusion module is used to fuse and enhance the two types of features, followed by enhancement using a feature enhancement module. Next, a sub-pixel layer is used for upsampling to improve spatial resolution, and finally, an inverse discrete wavelet transform is used to convert the image into an image enhancement result.

2. The method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 1, characterized in that, The high-quality RAW image X ori Degradation processing is performed to obtain a low-quality RAW image X. raw and low-quality sRGB images X srgb Specifically, it includes: For the high-quality RAW image X ori The illumination degradation process was performed sequentially using normalization, inverse white balance, simulated low illumination, and denormalization to obtain the RAW image X after illumination degradation. dark ; X-rays of RAW images after illumination degradation dark Perform a resolution degradation operation to obtain a low-quality RAW image X. raw ; For the low-quality RAW image X raw Perform ISP pipeline rendering to obtain a low-quality sRGB image X. srgb .

3. The method for combined super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 2, characterized in that, The simulated low light is achieved by setting an exposure value, and the pixel value of the image obtained after simulating low light is y = x·2. EV Where x is the normalized and inverse white balance image pixel value, and EV is the exposure value.

4. The method for combined super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 2, characterized in that, The RAW image X after illumination degradation dark Perform a resolution degradation operation to obtain a low-quality RAW image X. raw Specifically, it includes: The maximum entropy downsampling method was used to process the RAW image after illumination degradation. dark Sampling is performed to obtain a fully sampled linear RAW image X. lin ; For fully sampled linear RAW images X lin Add out-of-focus blur and motion blur sequentially, then perform downsampling and Bayer sampling, and finally add noise to obtain a low-quality RAW image X. raw =f Bayer (f down (X lin *k def *k mot ))+n; Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) indicates a downsampling operation; k def and k mot These represent the defocus blur kernel and the dynamic blur kernel, respectively; * represents the convolution operation; and n represents heteroscedastic Gaussian noise.

5. The method for combined super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 4, characterized in that, The added noise specifically includes shot noise and read noise. Where I represents a noise-free image, The image after noise has been added is represented by N(·), which represents the normal distribution function, where 0 represents the mean, β1I+β2 represents the variance, and β1 and β2 are the shot noise parameter and read noise parameter, respectively.

6. The method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 1, characterized in that, The initial enhancement subnetwork includes a RAW image initial enhancement module and an sRGB image initial enhancement module; The RAW image initial enhancement module will X-ray the low-quality RAW image. raw Packed into a 4-channel RGGB format, resulting in X. pack ; Where H is the low-quality RAW image X raw High, W represents low-quality RAW image X raw The width; X pack The RAW features are fed into the feature enhancement module to complete feature extraction and enhancement. Then, the extracted and enhanced RAW features are upsampled by a factor of 2 to obtain the initial features in the RAW domain. Where C is the number of channels of the feature; The sRGB image initial enhancement module will transform the low-quality sRGB image X srgb The data is fed into the feature enhancement module to obtain the initial enhanced sRGB features.

7. The method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 1, characterized in that, The wavelet enhancement subnetwork uses discrete wavelet transform to transform the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Transforming to the frequency domain, the frequency fusion module is used to fuse and enhance the two types of features, specifically including: The wavelet enhancement subnetwork applies the initial features F in the RAW domain. raw and the initial feature F in the sRGB domain srgb Each component was transformed to the frequency domain using wavelet transform, yielding four frequency components. and Where i = 0 represents the low-frequency component LL, and i = 1, 2, 3 represent the three high-frequency components LH, HL and HH respectively; Same frequency and The signal is fed into the frequency fusion module to complete the feature fusion and enhancement.

8. The method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 7, characterized in that, The same frequency and The signal is fed into the frequency fusion module to complete the feature fusion and enhancement, specifically including: For the same frequency and By sequentially performing layer normalization, 1×1 convolutional layer, and 3×3 separable convolutional layer, the key matrix K, value matrix V, and query matrix Q are obtained. Adjust the shapes of the key matrix K, value matrix V, and query matrix Q to: To conform to the rules of matrix multiplication; Apply matrix multiplication to the shape-adjusted key matrix K and the shape-adjusted query matrix Q, and obtain the attention map using the Softmax function. Right now Where τ is a learnable parameter; Shape-adjusted value matrix V and attention map Multiply, and then combine the result with... The scaling factor α and offset factor β of the affine transformation are generated by adding them together through two independent 1×1 convolutional layers; The frequency components of sRGB are modulated using scaling factor α and offset factor β. Obtain the fused wavelet features Where ⊙ represents element-wise multiplication, and Conv1(·) represents a 1×1 convolutional layer.

9. The method for joint super-resolution and low-light enhancement processing based on RAW and sRGB images according to claim 1, characterized in that, The enhancement process involves using a feature enhancement module, followed by upsampling at the sub-pixel level to improve spatial resolution, and finally, inverse discrete wavelet transform to convert the image into an enhanced result. Specifically, it includes: Will and The sums are then fed into a feature enhancement module for feature enhancement. The enhanced output is then upsampled using a 3×3 convolution combined with a sub-pixel layer to achieve resolution enhancement. After the four frequency components i = 0, 1, 2, and 3 have been fused, enhanced, and upsampled, an inverse wavelet transform is applied to convert them to the spatial domain. A 3×3 convolutional layer is then applied to the output of the inverse transform to finally output the image enhancement result.

Images