Super-resolution and low-illumination enhancement combined processing method based on RAW and sRGB images

Abstract

The invention relates to the technical field of image processing, and provides a super-resolution and low-illumination enhancement combined processing method based on RAW and sRGB images. The method comprises the following steps: performing degradation processing on a high-quality RAW image to obtain a low-quality RAW image and a low-quality sRGB image; performing maximum entropy down-sampling processing on the high-quality RAW image to generate a full-sampled linear RAW image, and generating a high-quality sRGB image through ISP pipeline rendering; constructing a super-resolution and low-illumination enhancement combined processing network; and training the super-resolution and low-illumination enhancement combined processing network by using the low-quality RAW image, the low-quality sRGB image and the high-quality sRGB image. According to the method, the problems that a training data set of image super-resolution and illumination enhancement combined processing is missing and a reference image is not matched with a low-illumination and low-resolution input image are solved, and the limitation of a traditional processing method when a single sRGB image or a RAW image is used for processing can be overcome, so that higher image quality can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images. Background Art

[0002] Super Resolution (SR) technology can be used to achieve digital zoom, so that distant objects in images taken with a short-focus lens can be clearly displayed. However, in low-light environments, distant objects in short-focus images will experience a series of complex degradations in addition to resolution degradation, such as noise pollution, blur, color shift, etc. However, most existing super-resolution algorithms are trained on datasets under normal lighting and are difficult to directly adapt to low-light environments. This makes it challenging to enhance lighting and suppress noise while improving resolution.

[0003] To solve the above problems, a potential method is to directly cascade low-light image enhancement (LLIE) and SR algorithm. Since LLIE suppresses noise while enhancing illumination, it usually tends to obtain smoother enhanced images, which significantly affects the cascaded SR algorithm, causing errors to accumulate and spread further.

[0004] Moreover, most of the existing LLIE and SR methods are directly performed on sRGB (standard Red Green Blue) images. On the one hand, since sRGB images have been processed by the camera's Image Signal Processing (ISP) pipeline, each channel has only 8 bits of information; on the other hand, since the processing operations of the ISP pipeline include nonlinear transformations, such as tone mapping and gamma changes and JPEG (Joint Photographic Experts Group) compression, the pixel values ​​of sRGB images are no longer linearly related to light intensity, which makes it impossible to obtain the best results by performing SR and LLIE joint processing on sRGB images. On the other hand, the existing super-resolution datasets are collected by using a digital camera with a zoom lens, such as SR-RAW. Due to the depth of field and distortion caused by the change in lens focal length, the input and reference images are inevitably misaligned, which makes it necessary to consider the image registration problem for the datasets collected in this way.

[0005] The RAW image data originally recorded by the camera is usually 12 to 14 bits, with more detail information and a higher dynamic range, and is linearly related to the brightness of the scene. This makes RAW images more suitable for low-level image processing than sRGB images. Although there are many SR datasets based on RAW images, these datasets cannot be used to train LLIE algorithm models. On the other hand, RAW images do not contain any information about the ISP pipeline, and some key processing within the camera ISP pipeline, such as color lookup, tone mapping and other key parameters of the process are usually strictly confidential by each camera manufacturer, which results in the inability to accurately render the color style corresponding to the target camera model by using only RAW images for SR and LLIE joint processing.

[0006] In summary, existing technologies lack a method to generate a dataset that contains both low-light, low-resolution RAW and sRGB images, with input and output fully aligned, and cannot effectively fuse the complementary information of RAW and sRGB images. Therefore, there is room for further improvement in the quality of reconstructed images.

[0007] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images.

[0009] The present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images, comprising:

[0011] In step 201, a high-quality RAW image X is ori Degraded processing is performed to obtain a low-quality RAW image X raw and low quality sRGB image X srgb ; Wherein, the high-quality RAW image X ori is a high-resolution RAW image under normal lighting, and the low-quality RAW image X raw For low-light, low-resolution RAW images, the low-quality sRGB image X srgb It is a low-light, low-resolution sRGB image;

[0012] In step 202, the high-quality RAW image X ori Perform maximum entropy downsampling to generate a fully sampled linear RAW image, and generate a high-quality sRGB image Y through ISP pipeline rendering srgb; Wherein, the high-quality sRGB image is a sRGB image with normal illumination and high resolution;

[0013] In step 203, a super-resolution and low-light enhancement joint processing network is constructed; the low-quality RAW image X is used raw , the low quality sRGB image X srgb and the high quality sRGB image Y srgb , generate a data set; use the data set to train the super-resolution and low-light enhancement joint processing network to obtain a trained super-resolution and low-light enhancement joint processing network for image enhancement.

[0014] The high quality RAW images ori Degraded processing is performed to obtain a low-quality RAW image X raw and low quality sRGB image X srgb , including:

[0015] For the high quality RAW image X ori Normalization, inverse white balance, simulated low light, and denormalization are used in sequence to perform light degradation processing to obtain the RAW image X after light degradation. dark ;

[0016] For RAW images after light degradation dark Perform resolution degradation to obtain low-quality RAW image X raw ;

[0017] For the low quality RAW image X raw Perform ISP pipeline rendering to obtain a low-quality sRGB image X srgb .

[0018] The simulation of low light is achieved by setting the exposure value. The pixel value of the image obtained after the simulation of low light is y=x·2 EV ; Where x is the normalized and inverse white balanced image pixel value, and EV is the exposure value.

[0019] The RAW image X after the illumination degradation dark Perform resolution degradation to obtain low-quality RAW image X raw , including:

[0020] Use maximum entropy downsampling to degrade the RAW image X dark Sampling is performed to obtain a fully sampled linear RAW image X lin ;

[0021] For fully sampled linear RAW images X linAdd out-of-focus blur and motion blur in sequence, then perform downsampling and Bayer sampling, and finally add noise to get a low-quality RAW image X raw =f Bayer (f down (X lin *k def *k mot ))+n;

[0022] Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) represents the downsampling operation; k def and k mot denote the defocus blur kernel and the dynamic blur kernel respectively; * denotes the convolution operation; n denotes heteroscedastic Gaussian noise.

[0023] The added noise specifically includes adding shooting noise and reading noise, specifically:

[0024] Where I represents the noise-free image, represents the image after adding noise, N(·) represents the normal distribution function, where 0 represents the mean, β1I+β2 represents the variance, and β1 and β2 are the shot noise parameter and read noise parameter, respectively.

[0025] The super-resolution and low-light enhancement joint processing network includes an initial enhancement sub-network and a wavelet enhancement sub-network based on discrete wavelet transform;

[0026] The initial enhancement sub-network is used for low-quality RAW images X raw Perform feature extraction to obtain the initial feature F in the RAW domain raw , and for low-quality sRGB images X srgb Perform feature extraction to obtain the initial feature F in the sRGB domain srgb ;

[0027] The wavelet enhancement sub-network uses discrete wavelet transform to transform the RAW domain initial feature F raw and the sRGB domain initial feature F srgb Transform to the frequency domain, use the frequency fusion module to fuse and enhance the two types of features, then use the feature enhancement module to enhance them, then use the sub-pixel layer to upsample to improve the spatial resolution, and finally use the inverse discrete wavelet transform to convert to image enhancement results

[0028] The initial enhancement sub-network includes a RAW image initial enhancement module and an sRGB image initial enhancement module;

[0029] The RAW image initial enhancement module converts low quality RAW images into rawPacked into RGGB 4-channel format, get X pack ; Where H and W represent the height and width of the input RAW image respectively;

[0030] X pack The feature enhancement module is used to extract and enhance the features, and then the extracted and enhanced RAW features are upsampled by 2 times to obtain the initial features in the RAW domain.

[0031] The sRGB image initial enhancement module converts the low-quality sRGB image X srgb Send it to the feature enhancement module to obtain the initial enhanced sRGB features

[0032] The wavelet enhancement sub-network uses discrete wavelet transform to transform the RAW domain initial feature F raw and the sRGB domain initial feature F srgb Transform to the frequency domain and use the frequency fusion module to fuse and enhance the two types of features, including:

[0033] The wavelet enhancement sub-network is used to transform the initial feature F in the RAW domain raw and the sRGB domain initial feature F srgb Use wavelet transform to transform into frequency domain respectively, and get 4 frequency components respectively and Where i=0 represents the low-frequency component LL, i=1, 2, 3 represent the three high-frequency components LH, HL and HH respectively;

[0034] The same frequency and The features are sent to the frequency fusion module to complete feature fusion enhancement.

[0035] The same frequency and The frequency fusion module is sent to complete the fusion enhancement of features, including:

[0036] For the same frequency and Perform layer normalization, 1×1 convolution layer, and 3×3 separable convolution layer in sequence to obtain the key matrix K, value matrix V, and query matrix Q; adjust the shapes of the key matrix K, value matrix V, and query matrix Q to To comply with the matrix multiplication rules;

[0037] Apply matrix multiplication to the reshaped key matrix K and the reshaped query matrix Q and get the attention map through the Softmax function Right now Among them, τ is a learnable parameter;

[0038] The reshaped value matrix V and the attention map Multiply them and add the result to Add together and generate the scaling factor α and offset factor β of the affine transformation through two independent 1×1 convolutional layers;

[0039] Modulate the frequency components of sRGB using scaling factor α and offset factor β Get the fused wavelet features Among them, * represents element-wise multiplication, and Conv1(·) represents a 1×1 convolutional layer.

[0040] The feature enhancement module is used for enhancement, and then the sub-pixel layer is used for upsampling to improve the spatial resolution, and finally the discrete wavelet inverse transform is used to convert it into an image enhancement result. Specifically include:

[0041] Will and The output is added and further sent to the feature enhancement module for feature enhancement. The enhanced output is upsampled using 3×3 convolution and sub-pixel layer to enhance the resolution.

[0042] After the four frequency components i=0, 1, 2, and 3 have completed fusion enhancement and upsampling, they are transformed into the spatial domain using the inverse wavelet transform. A 3×3 convolutional layer is used on the output of the inverse transform to finally output the image enhancement result.

[0043] The present invention solves the problems of missing training data sets and mismatch between reference images and low-light, low-resolution input images in the joint processing of image super-resolution and illumination enhancement, and can overcome the limitations of traditional processing methods when processing with a single sRGB image or RAW image, thereby obtaining higher image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of a method for joint processing of super-resolution and low-light enhancement based on RAW and sRGB images provided by an embodiment of the present invention;

[0046] Figure 2is a schematic diagram of a synthetic image processing pipeline in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of a super-resolution and low-light enhancement joint processing network in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a RAW image initial enhancement module in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0049] Figure 5 is a schematic diagram of a feature enhancement module in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0050] Figure 6 is a schematic diagram of a residual channel attention group in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0051] Figure 7 is a schematic diagram of an sRGB image initial enhancement module in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0052] Figure 8 is a schematic diagram of a frequency fusion module in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0053] Fig. 9 It is a flowchart of a method for joint processing of super-resolution and low-light enhancement based on RAW and sRGB images provided by an embodiment of the present invention;

[0054] Fig.10 is a schematic diagram of a low-light, low-resolution sRGB input image according to a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0055] Fig.11 It is a schematic diagram of the application effects of various methods in the prior art in a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention;

[0056] Fig.12It is a schematic diagram of the application effect of a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0059] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0060] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0061] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0062] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] Embodiment 1:

[0064] In order to reduce error accumulation and diffusion, SR and LLIE can be jointly processed. Although there are joint processing methods for LLIE and other image restoration problems, such as deblurring, defogging, or deraining, there are few joint processing methods for SR and LLIE. The existing technology lacks a method for generating a data set that contains both low-light and low-resolution RAW and sRGB images, and whose input and output are fully aligned. It also cannot effectively fuse the complementary information of RAW and sRGB images. Therefore, there is room for further improvement in the quality of the reconstructed image. In order to solve this problem, embodiment 1 of the present invention provides a joint processing method for super-resolution and low-light enhancement based on RAW and sRGB images, such as Figure 1 As shown, including:

[0065] In step 201, a high-quality RAW image X is ori Degraded processing is performed to obtain a low-quality RAW image X raw and low quality sRGB image X srgb ; Wherein, the high-quality RAW image X ori is a high-resolution RAW image under normal lighting, and the low-quality RAW image X raw For low-light, low-resolution RAW images, the low-quality sRGB image X srgb It is a low-light, low-resolution sRGB image.

[0066] In step 202, the high-quality RAW image X ori Perform maximum entropy downsampling to generate a fully sampled linear RAW image, and generate a high-quality sRGB image Y through ISP pipeline rendering srgb ; Wherein, the high-quality sRGB image is a sRGB image with normal illumination and high resolution; the high-quality sRGB image Y srgb Used as a reference image for supervised learning. In an optional implementation, the ISP pipeline is Adobe Camera RAW, which can simulate some complex operations in the ISP pipeline of a real camera, such as tone mapping, color lookup, etc.

[0067] In step 203, a super-resolution and low-light enhancement joint processing network is constructed; the low-quality RAW image X is used raw , the low quality sRGB image X srgb and the high quality sRGB image Y srgb , generate a data set; use the data set to train the super-resolution and low-light enhancement joint processing network to obtain a trained super-resolution and low-light enhancement joint processing network for image enhancement. The data set is also called the DarkSR data set. The use of the data set to train the super-resolution and low-light enhancement joint processing network is specifically: low-quality RAW images X in the data set are trained raw and low quality sRGB image X srgb Input to the super-resolution and low-light enhancement joint processing network to obtain the output of the super-resolution and low-light enhancement joint processing network, that is, the image enhancement result According to the image enhancement results and the high-quality sRGB image Y in the dataset srgb Calculate the loss function, and use the loss function to iteratively train the super-resolution and low-light enhancement joint processing network until the super-resolution and low-light enhancement joint processing network converges. In an optional implementation, the loss function may be: With the high quality sRGB image Y srgb The L1 norm of the difference.

[0068] It should be noted here that the high resolution and low resolution in this embodiment do not refer to a specific resolution, but are relative to each other. For example, in an actual application scenario, there is a picture A with a resolution of 480×640, a picture B with a resolution of 720×1048, and a picture C with a resolution of 4K. For pictures A and B, the resolution of picture A is low resolution, and the resolution of picture B is high resolution; for pictures B and C, the resolution of picture B is low resolution, and the resolution of picture C is high resolution. Similarly, the normal light and low light in this embodiment do not refer to a specific light, but are relative to each other. For example, in an optional implementation, the normal light may be the light for taking pictures outdoors during the day, and the low light may be the light for taking pictures at night or in an insufficiently lit environment (such as a dark room). For example, there is a photo A with an illumination of 0.2 lux, a photo B with an illumination of 3 lux, and a photo C with an illumination of 5 lux. For photos A and B, the illumination of photo A is low light, and the illumination of photo B is normal light. For photos B and C, the illumination of photo B is low light, and the illumination of photo C is normal light. In actual use, photos taken during the day of the same scene can be used as images with normal light, and photos taken at night of the same scene can be used as images with low light. The high resolution and low resolution of the image are obtained by the photographer by setting the corresponding resolution when shooting.

[0069] This embodiment proposes a super-resolution and low-light enhancement joint processing method based on RAW and sRGB images. By simulating the illumination and resolution degradation process and combining the simulation ISP pipeline, a high-resolution RAW image with normal illumination is degraded into a low-light, low-resolution RAW image and an sRGB image. At the same time, the corresponding normal illumination and high-resolution sRGB are used as reference images to construct a training data set. According to the constructed training data set, a super-resolution and low-light enhancement joint processing network is established with the low-light, low-resolution RAW and sRGB images as inputs, and end-to-end fully supervised training is performed, thereby solving the problems of missing training data sets and mismatch between reference images and low-light, low-resolution input images in the joint processing of image super-resolution and illumination enhancement, and overcoming the limitations of traditional processing methods when processing with a single sRGB image or RAW image, thereby obtaining higher image quality.

[0070] Wherein, the steps 201 and 202 may be achieved by establishing Figure 2 The synthetic image processing pipeline shown is implemented, that is, the high-quality RAW image X ori Degraded processing is performed to obtain a low-quality RAW image X rawand low quality sRGB image X srgb , including:

[0071] For the high quality RAW image X ori Normalization, inverse white balance, simulated low light, and denormalization are used in sequence to perform light degradation processing to obtain the RAW image X after light degradation. dark ;in, Normalization is to map the pixel value to a value between 0 and 1. The purpose of inverse white balance is to remove the influence of the light source. The simulation of low light is achieved by setting the exposure value. The pixel value of the image obtained after the simulation of low light is y=x·2 EV ; Where x is the normalized and inverse white balanced image pixel value, EV is the exposure value, and reducing the illumination means that EV is a number less than 0. In actual use, EV can be randomly selected from the interval [-2.2, -1.8].

[0072] For RAW images after light degradation dark Perform resolution degradation to obtain low-quality RAW image X raw ; For the low quality RAW image X raw Perform ISP pipeline rendering to obtain a low-quality sRGB image X srgb .

[0073] In an optional implementation, the RAW image X after illumination degradation dark Perform resolution degradation to obtain low-quality RAW image X raw , including:

[0074] Use maximum entropy downsampling to degrade the RAW image X dark Sampling is performed to obtain a fully sampled linear RAW image X lin ;X lin It is a 3-channel fully sampled linear RAW image. This avoids the errors caused by the method of "demosaicing first and then Bayer sampling".

[0075] For fully sampled linear RAW images X lin Add out-of-focus blur and motion blur in sequence, then perform downsampling and Bayer sampling, and finally add noise to get a low-quality RAW image X raw =f Bayer (f down (X lin *k def *k mot))+n; the downsampling may be 2 times downsampling, and the sequential addition of defocus blur and motion blur is to simulate the blur caused by increased focus and exposure time when shooting in a low-light environment.

[0076] Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) represents the downsampling operation. In actual use, f down (·) can represent a 2x downsampling operation; k def and k mot denote the defocus blur kernel and the dynamic blur kernel respectively; * denotes a convolution operation; n denotes heteroscedastic Gaussian noise. The blur kernel size of the defocus blur is a random integer in the interval [3, 5], and the blur kernel size of the motion blur is a random integer in the set {3, 5, 7}.

[0077] The added noise specifically includes adding shot noise and read noise, specifically:

[0078] Where I represents the noise-free image, Represents the image after adding noise, N(·) represents the normal distribution function, where 0 represents the mean, β1I+β2 represents the variance, β1 and β2 are the shot noise parameter and the read noise parameter, respectively. A technician in this field measures the noise of a real night image, and manually determines β1 and β2, which are packaged to meet the image noise characteristics of short exposure and high ISO at night. Specifically: the noise is heteroscedastic Gaussian noise, and its shot and read noise parameters are manually set by measuring a real night image pair to meet the image noise characteristics of high ISO and short exposure time at night. Encapsulating β1 and β2 as ISO noise is a prior art. The value of ISO noise is a random integer in the interval [400, 1600].

[0079] In actual use, the super-resolution and low-light enhancement joint processing network includes an initial enhancement sub-network and a wavelet enhancement sub-network based on discrete wavelet transform, such as Figure 3 shown.

[0080] The initial enhancement sub-network is used for low-quality RAW images X raw Perform feature extraction to obtain the initial feature F in the RAW domain raw , and for low-quality sRGB images X srgb Perform feature extraction to obtain the initial feature F in the sRGB domain srgb .

[0081] The wavelet enhancement sub-network uses discrete wavelet transform to transform the RAW domain initial feature F raw and the sRGB domain initial feature F srgbTransform to the frequency domain, use the frequency fusion module to fuse and enhance the two types of features, then use the feature enhancement module to enhance them, then use the sub-pixel layer to upsample to improve the spatial resolution, and finally use the inverse discrete wavelet transform to convert to image enhancement results The upsampling here can be 2 times upsampling in actual use. The frequency fusion module is also called FFM module, and the feature enhancement module is also called FEM module.

[0082] Among them, the initial enhancement subnetwork includes a RAW image initial enhancement module and an sRGB image initial enhancement module; the RAW image initial enhancement module is also called an upper branch, and the sRGB image initial enhancement module is also called a lower branch.

[0083] like Figure 4 As shown, the RAW image initial enhancement module includes a packing module, a 3x3 convolution layer, a feature enhancement module and a 3x3 convolution layer in sequence from input to output. The RAW image initial enhancement module converts the low-quality RAW image X raw Packed into RGGB 4-channel format, get X pack ; Where H and W represent the input RAW image (i.e., the low-quality RAW image X raw ), that is, H is the height and width of the low-quality RAW image X raw High, W is a low quality RAW image X raw width.

[0084] X pack The feature enhancement module is used to extract and enhance the features, and then the extracted and enhanced RAW features are upsampled by 2 times to obtain the initial features in the RAW domain. like Figure 5 As shown, the feature enhancement module includes two cascaded residual channel attention groups (RCAGs), such as Figure 6 As shown, one residual channel attention group is composed of four residual channel attention blocks (RCABs). RCAB is an existing neural network technology and is not described in detail here. The 2x upsampling is to complete the subsequent fusion enhancement with sRGB features. Wherein, C is the number of channels of the feature. In an optional embodiment, C=32.

[0085] like Figure 7 As shown, the sRGB image initial enhancement module includes a 3x3 convolution layer and a feature enhancement module from input to output. The sRGB image initial enhancement module converts the low-quality sRGB image X srgb Send it to the feature enhancement module to obtain the initial enhanced sRGB features

[0086] In some embodiments, the frequency fusion module is as follows: Figure 8 As shown, the wavelet enhancement sub-network uses discrete wavelet transform to transform the RAW domain initial feature F raw and the sRGB domain initial feature F ssrgb Transform to the frequency domain and use the frequency fusion module to fuse and enhance the two types of features, including:

[0087] The wavelet enhancement sub-network is used to transform the initial feature F in the RAW domain raw and the sRGB domain initial feature F srgb Use wavelet transform to transform into frequency domain respectively, and get 4 frequency components respectively and Where i = 0 represents the low-frequency component LL, i = 1, 2, 3 represent the three high-frequency components LH, HL and HH respectively; for the RAW domain initial feature F raw Using wavelet transform to frequency domain, we get For the initial feature F in sRGB domain srgb Using wavelet transform to frequency domain, we get

[0088] The same frequency and The low-frequency components represent the global information of the image, such as brightness and color, and the high-frequency components represent the structure and detail information of the image. Therefore, by enhancing different frequency components separately, it is helpful to efficiently enhance brightness and detail. At the same time, since RAW and sRGB images have a high correlation at the same spatial position, for the same frequency components of RAW and sRGB and The fusion helps to efficiently capture complementary information. The frequency fusion module consists of affine transformation and cross attention. The cross attention can effectively capture and The long-range dependency of the original affine transformation is used to construct the scale factor α and offset factor β of the affine transformation, which can break through the limited receptive field of the original affine transformation.

[0089] The same frequency and The frequency fusion module is sent to complete the fusion enhancement of features, including:

[0090] For the same frequency and Layer normalization, 1×1 convolution layer, and 3×3 separable convolution layer are performed in sequence to obtain the key matrix K, value matrix V, and query matrix Q; among them, RAW features generate the key matrix K and value matrix V, and sRGB features generate the query matrix Q.

[0091] Adjust the shape of the key matrix K, value matrix V and query matrix Q to to comply with the matrix multiplication rules.

[0092] Apply matrix multiplication to the reshaped key matrix K and the reshaped query matrix Q and get the attention map through the Softmax function Right now Among them, τ is a learnable parameter.

[0093] The reshaped value matrix V and the attention map Multiply them and compare the result (i.e. the result of multiplication) with Added together, the scaling factor α and offset factor β of the affine transformation are generated through two independent 1×1 convolutional layers.

[0094] Modulate the frequency components of sRGB using scaling factor α and offset factor β Get the fused wavelet features Among them, ⊙ represents element-by-element multiplication, and Conv1(·) represents a 1×1 convolutional layer. It can be understood as follows: first, the cross attention between the corresponding frequency components of the discrete wavelet transformed frequency components of the RAW and sRGB initial features is calculated; secondly, the features after the cross attention are used to calculate a scale parameter and an offset parameter; finally, the scale parameter and offset parameter are used to act on the wavelet frequency components of the sRGB features to achieve fusion enhancement of the RAW and sRGB features in the wavelet transform domain.

[0095] The feature enhancement module is used for enhancement, and then the sub-pixel layer is used for upsampling to improve the spatial resolution, and finally the discrete wavelet inverse transform is used to convert it into an image enhancement result. Specifically include:

[0096] Will and The output is added and further sent to the feature enhancement module for feature enhancement. The enhanced output is upsampled using 3×3 convolution and sub-pixel layer to enhance the resolution. The upsampling here can be 2 times upsampling in actual use.

[0097] After the four frequency components i=0, 1, 2, and 3 have completed fusion enhancement and upsampling, they are transformed into the spatial domain using the inverse wavelet transform. A 3×3 convolutional layer is used on the output of the inverse transform to finally output the image enhancement result.

[0098] This embodiment designs a degradation pipeline from a normal light high-resolution RAW image to generate a low-light, low-resolution image dataset for training and evaluating algorithm performance; on the other hand, a super-resolution and low-light enhancement joint processing network is designed, the main feature of which is that it uses low-light, low-resolution RAW and sRGB images as input at the same time, extracts the features of the two image inputs respectively, uses wavelet transform to transform the image to the frequency domain, and uses the FFM module to fuse the two types of wavelet domain features. By making full use of the complementary information between the two types of images, high-quality image enhancement is achieved with low computational complexity.

[0099] Embodiment 2:

[0100] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0101] In this embodiment, Adobe Camera RAW software is used as the ISP pipeline to simulate the ISP pipelines corresponding to 20 camera models including four camera brands including Canon, SONY, LEICA and Nikon, and generate corresponding sRGB image styles, thereby enriching the types and models of cameras in the training set and the test set. For example, the super-resolution and low-light enhancement joint processing method based on RAW and sRGB images described in this embodiment is as follows: Fig. 9 As shown, specifically including:

[0102] In step 301, a high-resolution RAW image with normal exposure (i.e., normal lighting) is Degradation processing to generate low-resolution RAW images in low light and sRGB images Where H and W represent the height and width of the image or feature respectively. Specifically: For a high-resolution RAW image X with normal exposure ori , synthesize low-light, low-resolution RAW images through low-light degradation (including normalization, inverse white balance, simulated low-light, denormalization), maximum entropy downsampling, resolution degradation (including defocus blur, motion blur, 2x downsampling, Bayer sampling, noise addition) and ISP pipeline rendering raw and sRGB image X srgb .

[0103] The step 301 specifically includes:

[0104] (1) For the input high-resolution RAW image X with normal exposure oriFirst, perform light degradation processing. Light degradation mainly uses the linear relationship between the pixel value of the RAW image and the light intensity of the actual scene to linearly adjust the pixel value of the RAW image. ori Normalization is the process of mapping pixel values ​​to values ​​between 0 and 1, removing the influence of light sources through inverse white balance, simulating low-light effects by setting negative EV values, and then denormalizing to obtain a high-resolution RAW image with light degradation. In the present invention, the EV value is randomly selected from the interval [-1.8, -2.2], that is:

[0105] y=x·2 EV (1)

[0106] Among them, x is the image pixel value after normalization and inverse white balance processing, EV is the set exposure value, and y is the image pixel value after the light is reduced.

[0107] (2) For the output result X of (1) with low or medium illumination degradation dark , perform resolution degradation operation: first use maximum entropy downsampling to obtain a linear RAW image with 3 channels fully sampled This avoids the error caused by the method of "de-mosaicing first and then Bayer sampling". Next, defocus blur and motion blur are added in sequence to simulate the blur caused by increased focus and exposure time when shooting in a low-light environment, followed by 2x downsampling and Bayer sampling, and finally noise is added to obtain a RAW image with degraded light and resolution. raw As shown in formula (2), from X lin Degenerates to X raw The process:

[0108] X raw =f Bayer (f down (X lin *k def *k mot ))+n (2)

[0109] Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) indicates a 2x downsampling operation; k def and k mot denote the defocus blur and dynamic blur kernels respectively; * denotes the convolution operation; n denotes heteroscedastic Gaussian noise, as shown in formula (3), and the noise is added as follows:

[0110]

[0111] Where I represents the noise-free image, It represents the image after adding noise, β1 and β2 are the shot noise and read noise parameters respectively. β1 and β2 are manually determined by measuring the noise of real night images, and are packaged to meet the image noise characteristics of short exposure and high ISO at night. Since packaging β1 and β2 as ISO noise is an existing technology, it will not be repeated here.

[0112] (3) The degraded RAW image is rendered through the ISP pipeline to generate the corresponding low-light, low-resolution sRGB image X srgb . Maximum entropy downsampling is a prior art and will not be described in detail here. In the present invention, the blur kernel size of the out-of-focus blur is a random integer in the interval [3, 5], and the blur kernel size of the motion blur is a random integer in the set {3, 5, 7}. The value of ISO noise is a random integer in the interval [400, 1600]. The ISP pipeline selected is Adobe Camera RAW, which can simulate some complex operations in the ISP pipeline of a real camera, such as tone mapping, color lookup, etc.

[0113] In step 302, in order to perform supervised learning, it is also necessary to generate a corresponding high-resolution normal-light sRGB image As a reference image. Among them, for the input high-resolution RAW image X ori Perform maximum entropy downsampling and ISP pipeline rendering in sequence to generate a normal lighting, high-resolution sRGB image Y srgb .

[0114] In step 303, the low-resolution RAW image X generated in steps 301 and 302 under low light conditions is converted into raw and low-resolution sRGB images in low light X srgb And normal lighting, high-resolution sRGB image Y srgb Construct the DarkSR dataset and build a dual-input super-resolution and low-light enhancement joint processing network, which accepts RAW and sRGB image inputs at the same time and outputs high-resolution sRGB images with enhanced lighting. (i.e. image enhancement result). Figure 3 As shown, the dual-input network contains two sub-networks, one is the initial enhancement sub-network and the other is the wavelet enhancement sub-network.

[0115] The step 303 specifically includes:

[0116] (1) In the initial enhancement subnetwork, such as Figure 3 As shown in , it is divided into two branches, the upper branch completes the initial feature enhancement of the RAW image, and the lower branch completes the initial feature enhancement of the sRGB image. Figure 4As shown, in the RAW image initial enhancement subnetwork of the upper branch, for the input RAW image X raw , first packaged into RGGB 4-channel format Among them, C is the number of channels of the feature. Further, X pack The RAW features are sent to the feature enhancement module (FEM) for feature extraction and enhancement. In order to complete the subsequent fusion enhancement with sRGB features, the RAW features are upsampled by 2 times to obtain

[0117] like Figure 7 As shown in Figure 2, in the sRGB image initial enhancement subnetwork of the lower branch, the input sRGB image is directly fed into FEM to obtain the initial enhanced sRGB features. like Figure 5 As shown, FEM consists of RCAG, such as Figure 6 As shown, RCAG is composed of RCAB, which is an existing neural network technology and will not be described in detail. In the present invention, C=32.

[0118] (2) In the wavelet enhancement subnetwork, the Haar wavelet is used to perform discrete wavelet transform on the RAW and sRGB image features after initial enhancement to obtain the wavelet domain features of the RAW and sRGB images. Since the RAW and sRGB features of the same frequency components are highly correlated, the FFM is used to fuse and enhance the same frequency components of the two types of features to achieve the complementary advantages of the RAW and sRGB features. The frequency fusion module combines the attention mechanism and the affine transformation. The affine transformation can effectively complete the fusion of features, but its performance is limited by the limited receptive field of ordinary convolution. First, the long-range dependency of the two types of features is captured by using cross attention, and then the scale factor and offset factor of the affine transformation are generated using the obtained attention map to effectively complete the fusion of RAW and sRGB features. The frequency fused features and the RAW wavelet features are residually connected and sent to the FEM for further feature enhancement. Then, the enhanced output is subjected to a 3×3 convolution layer and a sub-pixel layer to improve the resolution of the wavelet features. After all frequency components are enhanced and the resolution is improved, the inverse wavelet transform function is used to reconstruct the three-channel image with illumination enhancement and resolution improvement. Finally, in order to further reduce possible artifacts, the three-channel image reconstructed by the sub-pixel layer is optimized again through a 3×3 convolution layer to obtain the final output image. Figure 3 As shown, for the RAW and sRGB features F generated in (1) raw and F srgb , respectively, using wavelet transform to the frequency domain, each obtaining 4 frequency components and Where i=0 represents the low-frequency component LL, and i=1, 2, 3 represent the three high-frequency components LH, HL and HH respectively. Since the low-frequency components represent the global information of the image such as brightness and color, and the high-frequency components represent the structure and detail information of the image, it is helpful to efficiently enhance brightness and detail by enhancing different frequency components separately. At the same time, since RAW and sRGB images have a high correlation at the same spatial position, for the same frequency components of RAW and sRGB and Fusion helps to efficiently capture complementary information, so the same frequency and The FFM module is sent to the FFM module to efficiently complete the fusion enhancement of features. The FFM module consists of affine transformation and cross attention. The use of cross attention can effectively capture and The long-range dependency of the original affine transformation is used to construct the scale factor α and offset factor β of the affine transformation, which can break through the limited receptive field of the original affine transformation.

[0119] Specifically: Figure 8 As shown, the same frequency components of RAW and sRGB and The layer normalization, 1×1 convolution layer, and 3×3 separable convolution layer are performed in sequence. The RAW features generate the key matrix K and value matrix V, and the sRGB features generate the query matrix Q, and the shape is adjusted to To comply with the matrix multiplication rule. Further, apply matrix multiplication to K and Q and obtain the attention map through the Softmax function. As shown in formula (4):

[0120]

[0121] Where τ is a learnable parameter.

[0122] The reshaped value matrix V and the attention map Multiply them and add the result to Addition, through two independent 1×1 convolutional layers to generate the scaling factor α and offset factor β of the affine transformation, further α and β are used to modulate the frequency components of sRGB As shown in formula (5), the final FFM outputs the fused wavelet features

[0123]

[0124] where ⊙ represents element-wise multiplication and Conv1(·) represents a 1×1 convolutional layer.

[0125] The output of FFM and The output is added and further sent to the FEM module for final feature enhancement. The enhanced output is subjected to 3×3 convolution and 2x upsampling at the sub-pixel layer to enhance the resolution. After the four frequency components i=0, 1, 2, and 3 have been fused, enhanced, and upsampled, the inverse wavelet transform is used to transform from the frequency domain to the spatial domain. In order to reduce possible artifacts, a 3×3 convolutional layer is used on the output of the inverse transform, and finally a high-resolution sRGB image with illumination enhancement is output.

[0126] In summary, after the above steps, on the one hand, a degradation pipeline is designed from normal light high-resolution RAW images to generate low-light, low-resolution image datasets for training and evaluating algorithm performance; on the other hand, a super-resolution and low-light enhancement joint processing network is designed, the main feature of which is that it takes low-light, low-resolution RAW and sRGB images as input at the same time, extracts the features of the two image inputs respectively, uses wavelet transform to transform the image into the frequency domain, and uses the FFM module to fuse the two types of wavelet domain features. By making full use of the complementary information between the two types of images, high-quality image enhancement is achieved with low computational complexity.

[0127] To illustrate the advantages of this embodiment, this embodiment also uses python to conduct experiments under the pytorch framework, and uses the method described in this embodiment to perform super-resolution and low-light enhancement joint processing on low-light, low-resolution RAW and sRGB images, and compares with the existing LLIE method SID based on RAW image input, and the SR method JDnDmSR, and the LLIE method CUE and SR method SRFormer (abbreviated as SR-F) based on sRGB image input, and the SR methods RawSR and PRNet based on dual input of RAW and sRGB images. Since the existing LLIE methods cannot improve the resolution, this embodiment also cascades the SR method SR-F after these methods to improve the resolution of their output.

[0128] Specifically, the performance of the present invention is evaluated by taking the dataset DarkSR generated by the dataset synthesis method provided by the present invention as an example. DarkSR contains 1395 and 153 pairs of low-light, low-resolution RAW and sRGB images / normal light, high-resolution sRGB images for training and testing, respectively. In order to train the super-resolution and low-light enhancement joint processing network proposed in the present invention, the training process uses the L1 loss function, which measures the pixel-by-pixel difference between the reconstructed output of the network and the true value. The RAW and corresponding sRGB images are randomly cropped into image blocks of size 256×256 according to the RGGB Bayer pattern and then input into the network. No data enhancement is used. The batch size is set to 6, and the AdamW optimizer is used, where the optimizer parameters are: β1=0.9, β2=0.99, and the weight decay ω=0.05. A warm-up strategy is used, and the learning rate is first linearly increased to 1.5×10 -4 , and then decayed to 6×10 using the cosine annealing strategy. -6 The entire training process lasted for 500 epochs, and the following results were obtained:

[0129]

[0130] The above table shows the quantitative comparison results of different input type algorithms on the DarkSR dataset. The comparison indicators are: Color Peak Signal to Noise Ratio (CPSNR), Structural Similarity Index (SSIM), CIELAB color difference (Delta E), Parameters, Floating Point Operations (FLOPs), and Runtime. FLOPs and Runtime are measured on images with a resolution of 256×256. All experimental results are obtained by running on an NVIDIA GTX 3090 GPU. The bold form in the above table represents the optimal value of each indicator.

[0131] From the results in the above table, it can be seen that the method of the present invention is superior to the existing algorithms in most indicators. Compared with the suboptimal PRNet algorithm, the method of the present invention requires 0.41M fewer parameters, and the FLOPs of the method of the present invention is about 1 / 2 of that of the PRNet algorithm.

[0132] Fig.10For a low-light, low-resolution sRGB image on DarkSR that has been upsampled by 2 times Bicubic, various algorithms in the existing technology (including: SID cascade SR-F, JDnDmSR, CUE cascade SR-F, SR-F, RawSR, PRNet method) are used to Fig.10 The enhanced image is as follows Fig.11 As shown, the method described in this embodiment is used to Fig.10 The enhanced image is as follows Fig.12 As shown, Fig.10 The lower left corner of the image is the RGB curve of the image. Similarly, Fig.11 The lower left corner of each image is the RGB curve of the corresponding image. Fig.12 The lower left corner of the middle image is the RGB curve of the image. It can be seen that, compared with other methods, the method of the present invention can maintain more texture structures and details, produce less distortion, and have a better effect of restoring image colors.

[0133] In summary, the method described in the present invention, on the one hand, designs a degradation pipeline from a normal light, high-resolution RAW image to generate a low-light, low-resolution image dataset for training and evaluating algorithm performance; on the other hand, designs a super-resolution and low-light enhancement joint processing network, the main feature of which is that it uses low-light, low-resolution RAW and sRGB images as input at the same time, extracts the features of the two image inputs respectively, converts the images to the frequency domain using wavelet transform, and fuses the two types of wavelet domain features using FFM module. By making full use of the complementary information between the two types of images, high-quality image reconstruction (i.e., image enhancement) is achieved with low computational complexity.

[0134] In general, the above technical solutions conceived by the present invention have the following advantages compared with the prior art:

[0135] Beneficial effects:

[0136] The main advantages of the present invention are: first, it can effectively simulate the illumination degradation and resolution degradation process, so as to construct a data set that can be used for end-to-end training, and solve the problem of missing low-light and low-resolution training data sets, or the misalignment of input low-quality images and reference images; second, in the super-resolution and low-light enhancement joint processing network of the present invention, low-resolution and low-light RAW and sRGB images are used as input at the same time, and wavelet transform and frequency fusion module are used to fuse and enhance the two, combining deep learning with wavelet transform, and the frequency domain after wavelet transform can effectively process the detailed information of multiple directions of the image, and at the same time, due to The resolution of each frequency component of the wavelet transform is relatively small, and a large amount of calculation will not be generated due to the use of the attention mechanism; at the same time, since there is a strong correlation between the frequency components of the RAW image and the sRGB image at the same spatial coordinates, the present invention designs a fusion enhancement module for the same frequency components of the RAW and sRGB images to achieve information complementarity between the RAW and sRGB images; by combining the respective advantages of deep learning and wavelet transform, due to the efficient and accurate application of the complementary information of the RAW and sRGB images, the present invention achieves better performance than the existing methods while maintaining a relatively small model size and low computational complexity.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A super-resolution and low-light enhancement joint processing method based on RAW and sRGB images, characterized in that: include: In step 201, a high-quality RAW image X is ori Degraded processing is performed to obtain a low-quality RAW image X raw and low quality sRGB image X srgb ; Wherein, the high-quality RAW image X ori is a high-resolution RAW image under normal lighting, and the low-quality RAW image X raw For low-light, low-resolution RAW images, the low-quality sRGB image X srgb It is a low-light, low-resolution sRGB image; In step 202, the high-quality RAW image X is ori Perform maximum entropy downsampling to generate a fully sampled linear RAW image, and generate a high-quality sRGB image Y through ISP pipeline rendering srgb ; Wherein, the high-quality sRGB image is a sRGB image with normal illumination and high resolution; In step 203, a super-resolution and low-light enhancement joint processing network is constructed; the low-quality RAW image X is used raw , the low quality sRGB image X srgb and the high quality sRGB image Y srgb , generate a data set; use the data set to train the super-resolution and low-light enhancement joint processing network to obtain a trained super-resolution and low-light enhancement joint processing network for image enhancement.

2. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 1, characterized in that: The high quality RAW images ori Degraded processing is performed to obtain a low-quality RAW image X raw and low quality sRGB image X srgb , specifically including: For the high quality RAW image X ori Normalization, inverse white balance, simulated low light, and denormalization are used in sequence to perform light degradation processing to obtain the RAW image X after light degradation. dark ; For RAW images after light degradation dark Perform resolution degradation to obtain low-quality RAW image X raw ; For the low quality RAW image X raw Perform ISP pipeline rendering to obtain a low-quality sRGB image X srgb .

3. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 2, characterized in that: The simulation of low light is achieved by setting the exposure value. The pixel value of the image obtained after the simulation of low light is y=x·2 EV ; Where x is the normalized and inverse white balanced image pixel value, and EV is the exposure value.

4. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 2, characterized in that: The RAW image X after the illumination degradation dark Perform resolution degradation to obtain low-quality RAW image X raw , including: Use maximum entropy downsampling to degrade the RAW image X dark Sampling is performed to obtain a fully sampled linear RAW image X lin ; For fully sampled linear RAW images X lin Add out-of-focus blur and motion blur in sequence, then perform downsampling and Bayer sampling, and finally add noise to get a low-quality RAW image X raw =f Bayer (f down (X lin *k def *k mot ))+n; Among them, f Bayer (·) represents the RGGB Bayer sampling function; f down (·) represents the downsampling operation; k def and k mot denote the defocus blur kernel and the dynamic blur kernel respectively; * denotes the convolution operation; n denotes heteroscedastic Gaussian noise.

5. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 4, characterized in that: The added noise specifically includes adding shot noise and read noise, specifically: Where I represents the noise-free image, represents the image after adding noise, N(·) represents the normal distribution function, where 0 represents the mean, β1I+β2 represents the variance, and β1 and β2 are the shot noise parameter and read noise parameter, respectively.

6. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 1, characterized in that: The super-resolution and low-light enhancement joint processing network includes an initial enhancement sub-network and a wavelet enhancement sub-network based on discrete wavelet transform; The initial enhancement sub-network is used for low-quality RAW images X raw Perform feature extraction to obtain the initial feature F in the RAW domain raw , and for low-quality sRGB images X srgb Perform feature extraction to obtain the initial feature F in the sRGB domain srgb ; The wavelet enhancement sub-network uses discrete wavelet transform to transform the RAW domain initial feature F raw and the sRGB domain initial feature F srgb Transform to the frequency domain, use the frequency fusion module to fuse and enhance the two types of features, then use the feature enhancement module to enhance them, then use the sub-pixel layer to upsample to improve the spatial resolution, and finally use the inverse discrete wavelet transform to convert to image enhancement results 7. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 6, characterized in that: The initial enhancement sub-network includes a RAW image initial enhancement module and an sRGB image initial enhancement module; The RAW image initial enhancement module converts low quality RAW images into raw Packed into RGGB 4-channel format, get X pack ; Where H is the low-quality RAW image X raw High, W is a low quality RAW image X raw width; X pack The feature enhancement module is used to extract and enhance the features, and then the extracted and enhanced RAW features are upsampled by 2 times to obtain the initial features in the RAW domain. Among them, C is the number of channels of the feature; The sRGB image initial enhancement module converts the low-quality sRGB image X srgb Send it to the feature enhancement module to obtain the initial enhanced sRGB features 8. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 6, characterized in that: The wavelet enhancement sub-network uses discrete wavelet transform to transform the RAW domain initial feature F raw and the sRGB domain initial feature F srgb Transform to the frequency domain and use the frequency fusion module to fuse and enhance the two types of features, including: The wavelet enhancement sub-network is used to transform the initial feature F in the RAW domain raw and the sRGB domain initial feature F srgb Use wavelet transform to transform into frequency domain respectively, and get 4 frequency components respectively and Where i=0 represents the low-frequency component LL, i=1, 2, 3 represent the three high-frequency components LH, HL and HH respectively; The same frequency and The features are sent to the frequency fusion module to complete feature fusion enhancement.

9. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 8, characterized in that: The same frequency and The frequency fusion module is sent to complete the fusion enhancement of features, including: For the same frequency and Perform layer normalization, 1×1 convolution layer, and 3×3 separable convolution layer in sequence to obtain the key matrix K, value matrix V, and query matrix Q; Adjust the shape of the key matrix K, value matrix V and query matrix Q to To comply with the matrix multiplication rules; Apply matrix multiplication to the reshaped key matrix K and the reshaped query matrix Q and get the attention map through the Softmax function Right now Among them, τ is a learnable parameter; The reshaped value matrix V and the attention map Multiply them and add the result to Add together and generate the scaling factor α and offset factor β of the affine transformation through two independent 1×1 convolutional layers; Modulate the frequency components of sRGB using scaling factor α and offset factor β Get the fused wavelet features Among them, ⊙ represents element-wise multiplication and Conv1(·) represents a 1×1 convolutional layer.

10. The method for super-resolution and low-light enhancement based on RAW and sRGB images according to claim 6, characterized in that: The feature enhancement module is used for enhancement, and then the sub-pixel layer is used for upsampling to improve the spatial resolution, and finally the discrete wavelet inverse transform is used to convert it into an image enhancement result. Specifically include: Will and The output is added and further sent to the feature enhancement module for feature enhancement. The enhanced output is upsampled using 3×3 convolution and sub-pixel layer to enhance the resolution. After the four frequency components i=0, 1, 2, and 3 have completed fusion enhancement and upsampling, they are transformed into the spatial domain using the inverse wavelet transform. A 3×3 convolutional layer is used on the output of the inverse transform to finally output the image enhancement result.

Images