Low-illumination image reconstruction method and system based on deep learning
By using a technology combining deep learning methods and physical models under low light conditions, the problem of poor image reconstruction effect in the prior art under low light conditions is solved, and high-quality low-light image reconstruction and adaptive exposure adjustment are achieved.
Patent Information
- Application Number
- CN202411905297.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art is difficult to effectively reconstruct high-quality images under low light conditions, especially in real scenes. The existing methods cannot adequately simulate noise in real camera images, resulting in poor performance when applying real low light images.
Using a deep learning-based method, a two-way image conversion and processing process reconstruction network and an adaptive exposure synthesis network were designed. Combined with low-light sensor noise modeling based on physical models, realistic paired low-light image training samples were generated, and image reconstruction was carried out through the U-net network.
Adaptive adjustment of exposure levels under low light conditions is achieved, high-quality low-light images are generated, and the time and manpower are required to acquire real paired training samples.
Smart Images
Figure CN119722496A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and more specifically, to a low-light image reconstruction method and system based on deep learning. Background Art
[0002] In the field of image processing, image reconstruction technology faces major challenges, especially under low-light conditions. Images taken under low-light conditions are usually not bright enough, resulting in an overall dark image, making it difficult to discern the outlines and details of objects. In addition, low light conditions can easily introduce a lot of noise, which can interfere with image quality, making it more difficult to distinguish image details and reducing the usability and reliability of the image.
[0003] In recent years, deep learning-based low-light image reconstruction methods have made significant progress on paired training samples. However, it remains challenging to recover high-quality image results in low-light situations due to the high cost and cumbersome process of obtaining real paired training samples. Currently, low-light image reconstruction algorithms are mainly developed and evaluated on synthetic data, which are usually generated under the assumption of additive white Gaussian noise (AWGN). Although deep learning methods have achieved impressive results on these synthetic datasets, they perform poorly when applied to real low-light images. This is because the AWGN model cannot adequately simulate the noise in real camera images, which is not only signal-dependent but also affected by the camera imaging process.
[0004] The prior art discloses a method for detecting extremely dark light objects based on RAW images, which is used for low-light images collected by conventional image sensors, establishes a dark light synthesis pipeline based on the physical imaging process, builds a high-quality dark light object detection simulation dataset based on existing normal light object detection dataset resources, and trains an accurate dark light object detection network. The defect of this solution is that the dark light synthesis pipeline established according to the physical imaging process will cause the synthesis process to fail to fully simulate the noise of real camera images, and thus performs poorly in the application of real low-light images.
[0005] To this end, in combination with the above requirements and the defects of the prior art, the present application proposes a low-light image reconstruction method and system based on deep learning. Summary of the invention
[0006] The present invention provides a low-light image reconstruction method and system based on deep learning, designs a deep learning-based algorithm to realize the mutual conversion between RGB images and RAW images, and proposes the use of an adaptive exposure synthesis network (AESN) to simulate the exposure level of low-light images. In this process, low-light sensor noise modeling based on a physical model is incorporated to synthesize realistic paired low-light image training samples.
[0007] The primary purpose of the present invention is to solve the above technical problems. The technical solution of the present invention is as follows:
[0008] A first aspect of the present invention provides a low-light image reconstruction method based on deep learning, the method comprising the following steps:
[0009] S1. Build a bidirectional image conversion and processing flow reconstruction network and an adaptive exposure synthesis network, and collect image samples to train the two networks.
[0010] S2, connects the bidirectional image conversion and processing flow reconstruction network and the adaptive exposure synthesis network, uses the low-light sensor model to model the noise, and obtains a synthesized low-light image.
[0011] S3. Use synthetic low-light images to train the preset U-net network.
[0012] S4. Input the real low-light image into the trained U-net network for reconstruction to obtain the low-light reconstructed image.
[0013] Furthermore, in step S1, the bidirectional image conversion and processing flow reconstructs the network, and its image conversion and processing process is specifically as follows: the preset RGB image is passed through a convolution layer with a size of 3×3 to obtain the feature T0, and then through three residual attention fusion modules to obtain the deep feature T1, and then through a convolution layer with a size of 3×3 to obtain the demosaiced image I dem , and then use the Bayer sampling function to omit two color channels for each pixel to obtain a RAW image; the RAW image first uses the Bayer unification technology to unify the image channels into the RGGB mode, and the unified RAW image passes through a 3×3 convolution layer and two residual attention fusion modules to obtain feature T2. Feature T2 and the preset RGB image are input into the color correction module to obtain feature T3, and then pass through a residual attention fusion module, a 3×3 convolution layer, and pixel shuffling to obtain an RGB image.
[0014] Furthermore, when training the bidirectional image conversion and processing flow reconstruction network, it is necessary to separately train the conversion of RGB images into RAW images and the conversion of RAW images into RGB images. After the training is completed, they are combined to realize the conversion of RGB images into RAW images and then into RGB images, wherein the loss functions of the two networks both use absolute error loss:
[0015]
[0016] in, is the RGB and RAW images reconstructed by the network, I rgb , I rawIt is a true RGB, RAW image.
[0017] Furthermore, the residual attention fusion module takes the input feature T input After five channel spatial attention modules, it passes through a convolutional layer of size 3×3 and is added to the input feature T to obtain the output feature T RAFM ; The channel space attention module takes the input feature T input First, pass through a convolutional layer of size 3×3, a ReLU activation function, and a convolutional layer of size 3×3 to obtain the feature T a , and then pass through average pooling, a convolutional layer with a size of 1×1, a ReLU activation function, a convolutional layer with a size of 1×1, a Sigmoid activation function, and the feature T a Multiply to get feature T c , the feature T c After average pooling and maximum pooling respectively, they are connected and then passed through a 3×3 convolutional layer, a Sigmoid activation function, and a 1×1 convolutional layer to obtain the feature T. s , and the characteristic T c And the input features T input Add to get the output feature T CSAM .
[0018] Furthermore, the color correction module first passes the input RGB image through a Gaussian blur, then passes it through a 3×3 convolution layer, two residual attention fusion modules, a 3×3 convolution layer, and a Sigmoid activation function, and multiplies it with the input feature T2 in the bidirectional image conversion and processing flow to obtain feature T color , and then the feature T color Add to input feature T2 to get feature T CCM .
[0019] Further, in step S2, the low-light sensor model is based on a CMOS sensor, converting the photons of incident light into electrons, then from electrons to voltages, and finally from voltages to digital signals to simulate a noisy electronic imaging pipeline; wherein due to the quantum properties of light, the number of uncertain electrons collected is Poisson distributed:
[0020] (I+N p )~p(I)
[0021] Where N p represents photon shot noise, P represents Poisson distribution, and I is the number of photoelectrons proportional to the scene illumination; the process of converting electrons into voltage uses a zero-mean Gaussian distribution to model the row noise:
[0022] N read ~TL(λ,μc , σ TL )
[0023] N r ~N(0,σ r )
[0024] where λ and σ TL They represent the shape parameter and scale parameter of the read noise, μ c is the DC noise component. r is the scale parameter of the row noise; the process of converting from voltage to digital signal uses uniform distribution to model the quantization noise:
[0025]
[0026] Where U(·) is a uniform distribution, q is a quantization step size, and the low-light sensor model is:
[0027] N=N p +N read +N r +N q
[0028] Where N p is the photon shot noise, N read is the read noise, N r is the line noise, N q is the quantization noise.
[0029] Furthermore, in step S3, the U-net network performs nonlinear mapping on the input image through 5 downsampling convolution blocks to obtain a feature image after nonlinear mapping, wherein each convolution block is sequentially composed of a convolution layer containing [16, 32, 64, 128, 128] convolution kernels of size 3*3, a batch normalization layer and a ReLU activation function, and the up and down sampling operations use bilinear downsampling; the original feature image after nonlinear mapping obtained by downsampling is gradually upsampled through 5 upsampling convolution blocks, and the feature images obtained in the downsampling process are spliced to finally obtain a reconstructed low-light enhanced image, wherein each convolution block is sequentially composed of a convolution layer containing [16, 32, 64, 128, 128] convolution kernels of size 3*3, a batch normalization layer and a PRelu activation function.
[0030] Furthermore, the loss of the U-net network is calculated as:
[0031] Loss = Loss_L2 + Loss_ssim
[0032] Where Loss_L2 is the mean square error of the pixel values between the reconstructed low-light image and the ground truth image, and Loss_ssims represents the structural similarity exponential loss function between the reconstructed low-light image and the ground truth image.
[0033] Furthermore, the step S4 is specifically as follows: inputting the real low-light image into the U-net network trained in step S3 to obtain a reconstructed low-light enhanced image, and evaluating the reconstructed low-light enhanced image using peak signal-to-noise ratio, structural similarity and image similarity.
[0034] The second aspect of the present invention provides a low-light image reconstruction system based on deep learning, which is used for the low-light image reconstruction method based on deep learning, and includes: an image conversion module, a low-light noise module, a model training module and an image reconstruction module: the image conversion module constructs a bidirectional image conversion and processing flow reconstruction network and an adaptive exposure synthesis network, collects image samples to train the two networks, the low-light noise module connects the bidirectional image conversion and processing flow reconstruction network and the adaptive exposure synthesis network, uses a low-light sensor model to model the noise, and obtains a synthesized low-light image, the model training module uses the synthesized low-light image to train the preset U-net network, and the image reconstruction module inputs the real low-light image into the U-net network trained by the model training module for reconstruction to obtain a low-light reconstructed image.
[0035] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0036] The present invention provides a low-light image reconstruction method and system based on deep learning, proposes a bidirectional image conversion and processing flow reconstruction network, which can realize the mutual conversion of RGB images and RAW images, and proposes an adaptive exposure synthesis network (AESN) to simulate the exposure level of low-light images. In this process, low-light sensor noise modeling based on a physical model is integrated to synthesize realistic paired low-light image training samples, which solves the problem of adaptively adjusting the exposure level when synthesizing low-light images, and does not need to spend a lot of time and manpower and material resources when obtaining real paired low-light image training samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a flow chart of a low-light image reconstruction method based on deep learning in the present invention.
[0038] Figure 2 Schematic diagram of a bidirectional image conversion and processing flow reconstruction network in an embodiment of the present invention.
[0039] Figure 3Schematic diagram of a residual attention fusion module in an embodiment of the present invention.
[0040] Figure 4 Schematic diagram of a channel space attention module in an embodiment of the present invention.
[0041] Figure 5 FIG. 4 is a schematic diagram of a color correction module in an embodiment of the present invention.
[0042] Figure 6 Schematic diagram of an adaptive exposure synthesis network in an embodiment of the present invention.
[0043] Figure 7 Schematic diagram of a U-net network in one embodiment of the present invention.
[0044] Figure 8 is a real low-light image used as input in an embodiment of the present invention.
[0045] Fig. 9 This is a low-light reconstructed image reconstructed in an embodiment of the present invention.
[0046] Fig.10 This is a schematic diagram of a low-light image reconstruction system based on deep learning according to the present invention. DETAILED DESCRIPTION
[0047] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0049] Example 1
[0050] like Figure 1 As shown, the present invention provides a low-light image reconstruction method based on deep learning, and the method comprises the following steps:
[0051] S1. Build a bidirectional image conversion and processing flow reconstruction network and an adaptive exposure synthesis network, and collect image samples to train the two networks.
[0052] S2, connects the bidirectional image conversion and processing flow reconstruction network and the adaptive exposure synthesis network, uses the low-light sensor model to model the noise, and obtains a synthesized low-light image.
[0053] S3. Use synthetic low-light images to train the preset U-net network.
[0054] S4. Input the real low-light image into the trained U-net network for reconstruction to obtain the low-light reconstructed image.
[0055] According to the above technical features, the present invention proposes an algorithm based on deep learning to realize the mutual conversion between RGB images and RAW images, and proposes an adaptive exposure synthesis network (AESN) to simulate the exposure level of low-light images. In this process, low-light sensor noise modeling based on a physical model is integrated to synthesize realistic paired low-light image training samples, which solves the problem that images are difficult to convert from the RGB domain and the RAW domain. At the same time, it also solves the problem of adaptively adjusting the exposure level when synthesizing low-light images, and there is no need to spend a lot of time and manpower and material resources to obtain real paired low-light image training samples.
[0056] like Figure 2 As shown, in step S1, the bidirectional image conversion and processing flow reconstruction network (BICPF) has the following image conversion and processing process: the preset RGB image is passed through a convolution layer with a size of 3×3 to obtain the feature T0, and then through three residual attention fusion modules (RAFM) to obtain the deep feature T1, and then through a convolution layer with a size of 3×3 to obtain the demosaiced image I dem , and then use the Bayer sampling function to omit two color channels for each pixel to obtain a RAW image; the RAW image first uses the Bayer unification technology to unify the image channels into the RGGB mode, and the unified RAW image passes through a 3×3 convolution layer and two residual attention fusion modules to obtain feature T2. Feature T2 and the preset RGB image are input into the color correction module (CCM) to obtain feature T3, and then pass through a residual attention fusion module, a 3×3 convolution layer, and pixel shuffle to obtain an RGB image.
[0057] When training the bidirectional image conversion and processing flow reconstruction network, it is necessary to separately train the conversion of RGB images into RAW images and the conversion of RAW images into RGB images. After the training is completed, they are combined to realize the conversion of RGB images into RAW images and then into RGB images. The loss functions of the two networks both use absolute error loss:
[0058]
[0059] in, is the RGB and RAW images reconstructed by the network, I rgb , I raw It is a true RGB, RAW image.
[0060] like Figure 3 As shown, the residual attention fusion module takes the input feature T input After five channel spatial attention modules, it passes through a convolutional layer of size 3×3 and is added to the input feature T to obtain the output feature T RAFM ;like Figure 4 As shown, the channel space attention module takes the input feature T input First, pass through a convolutional layer of size 3×3, a ReLU activation function, and a convolutional layer of size 3×3 to obtain the feature T a , and then pass through average pooling, a convolutional layer with a size of 1×1, a ReLU activation function, a convolutional layer with a size of 1×1, a Sigmoid activation function, and the feature T a Multiply to get feature T c , the feature T c After average pooling and maximum pooling respectively, they are connected and then passed through a 3×3 convolutional layer, a Sigmoid activation function, and a 1×1 convolutional layer to obtain the feature T. s , and the characteristic T c And the input features T input Add to get the output feature T CSAM .
[0061] like Figure 5 As shown in FIG. 1 , the color correction module first passes the input RGB image through a Gaussian blur, then passes through a 3×3 convolutional layer, two residual attention fusion modules, a 3×3 convolutional layer, and a Sigmoid activation function, and multiplies the input feature T2 in the bidirectional image conversion and processing flow to obtain the feature T color , and then the feature T color Add to input feature T2 to get feature T CCM .
[0062] like Figure 6 As shown, in step S2, the low-light sensor model is based on a CMOS sensor, which converts the photons of incident light into electrons, then converts them from electrons into voltages, and finally converts them from voltages into digital signals to simulate a noisy electronic imaging pipeline; wherein due to the quantum properties of light, the number of uncertain electrons collected is Poisson distributed:
[0063] (I+N p )~P(I)
[0064] Where N prepresents photon shot noise, P represents Poisson distribution, and I is the number of photoelectrons proportional to the scene illumination; it should be noted that after collecting electrons at each position, they are usually integrated, amplified, and read out as a measurable charge or voltage at the end of the exposure time, so this noise is called pixel circuit noise. It includes thermal noise, reset noise, source following noise, and row noise, and in low light conditions, the DC noise component can cause the image to appear color cast. In the present invention, in order to simplify the modeling, thermal noise, reset noise, and source following noise are combined into read noise modeling.
[0065] The electron-to-voltage conversion process uses a zero-mean Gaussian distribution to model the line noise:
[0066] N read ~TL(λ,μ c , σ TL )
[0067] N r ~N(0,σ r )
[0068] where λ and σ TL They represent the shape parameter and scale parameter of the read noise, μ c is the DC noise component. r is the scale parameter of the row noise; the process of converting from voltage to digital signal uses uniform distribution to model the quantization noise:
[0069]
[0070] Where U(·) is a uniform distribution, q is a quantization step size, and the low-light sensor model is:
[0071] N=N p +N read +N r +N q
[0072] Where N p is the photon shot noise, N read is the read noise, N r is the line noise, N q is the quantization noise.
[0073] like Figure 7As shown, in the step S3, the U-net network performs nonlinear mapping on the input image through 5 downsampling convolution blocks to obtain a feature image after nonlinear mapping, wherein each convolution block is sequentially composed of a convolution layer containing [16, 32, 64, 128, 128] convolution kernels of size 3*3, a batch normalization layer and a ReLU activation function, and the up and down sampling operations use bilinear downsampling; the original feature image after nonlinear mapping obtained by downsampling is gradually upsampled through 5 upsampling convolution blocks, and the feature images obtained in the downsampling process are spliced to finally obtain a reconstructed low-light enhanced image, wherein each convolution block is sequentially composed of a convolution layer containing [16, 32, 64, 128, 128] convolution kernels of size 3*3, a batch normalization layer and a PRelu activation function.
[0074] The loss of the U-net network is calculated as:
[0075] Loss = Loss_L2 + Loss_ssim
[0076] Where Loss_L2 is the mean square error of the pixel values between the reconstructed low-light image and the ground truth image, and Loss_ssims represents the structural similarity exponential loss function between the reconstructed low-light image and the ground truth image.
[0077] The step S4 specifically includes: inputting the real low-light image into the U-net network trained in step S3 to obtain a reconstructed low-light enhanced image, and evaluating the reconstructed low-light enhanced image using peak signal-to-noise ratio, structural similarity and image similarity.
[0078] Example 2
[0079] Based on the above embodiment 1, combined with Figure 8-Figure 9 This embodiment describes in detail the specific process of inputting the real low-light image into the model trained in step S3 to obtain the reconstructed low-light enhanced image.
[0080] In this embodiment, three evaluation indicators are used to evaluate the reconstructed low-light enhanced image: peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and image similarity (LPIPS). Figure 8 The real low-light image shown in FIG. 1 is input into the model of the present invention, and the output is as follows: Fig. 9 The reconstructed image shown in the figure, during the training process, the evaluation performance of different training sets on the U-net network is shown in Table 1. From the results, it can be seen that the performance of the synthetic low-light dataset of the present invention in the three evaluation indicators is only slightly inferior to that of the real low-light dataset. It can be seen that the network proposed in the present invention can not only effectively suppress noise but also be sharper in texture reconstruction.
[0081]
[0082] Table 1 Performance of different datasets on U-net reconstruction network
[0083] Example 3
[0084] like Fig.10 As shown, the present invention also provides a low-light image reconstruction system based on deep learning, which is used for the low-light image reconstruction method based on deep learning, and includes: an image conversion module, a low-light noise module, a model training module and an image reconstruction module: the image conversion module constructs a bidirectional image conversion and processing flow reconstruction network and an adaptive exposure synthesis network, collects image samples to train the two networks, the low-light noise module connects the bidirectional image conversion and processing flow reconstruction network and the adaptive exposure synthesis network, uses a low-light sensor model to model the noise, and obtains a synthesized low-light image, the model training module uses the synthesized low-light image to train the preset U-net network, and the image reconstruction module inputs the real low-light image into the U-net network trained by the model training module for reconstruction to obtain a low-light reconstructed image.
[0085] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media capable of storing program codes.
[0086] Alternatively, if the above-mentioned embodiments of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media capable of storing program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0087] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. The icons describing the structural positional relationships in the accompanying drawings are only used for exemplary purposes and are not to be construed as limiting the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and is not possible to list all the embodiments here. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the claims of the present invention.
Claims
1. A low-light image reconstruction method based on deep learning, characterized in that: The following steps are involved: S1. Build a bidirectional image conversion and processing flow reconstruction network and an adaptive exposure synthesis network, collect image samples to train the two networks; S2, connecting the bidirectional image conversion and processing flow reconstruction network and the adaptive exposure synthesis network, using the low-light sensor model to model the noise, and obtaining a synthesized low-light image; S3, using synthetic low-light images to train the preset U-net network; S4. Input the real low-light image into the trained U-net network for reconstruction to obtain the low-light reconstructed image.
2. The method for low-light image reconstruction based on deep learning according to claim 1, characterized in that: In step S1, the bidirectional image conversion and processing flow reconstructs the network, and its image conversion and processing process is specifically as follows: the preset RGB image is passed through a convolution layer of size 3×3 to obtain the feature T0, and then through three residual attention fusion modules to obtain the deep feature T1, and then through a convolution layer of size 3×3 to obtain the demosaiced image I dem , and then use the Bayer sampling function to omit two color channels for each pixel to obtain a RAW image; the RAW image first uses the Bayer unification technology to unify the image channels into the RGGB mode, and the unified RAW image passes through a 3×3 convolution layer and two residual attention fusion modules to obtain feature T2. Feature T2 and the preset RGB image are input into the color correction module to obtain feature T3, and then pass through a residual attention fusion module, a 3×3 convolution layer, and pixel shuffling to obtain an RGB image.
3. The method for low-light image reconstruction based on deep learning according to claim 2, characterized in that: When training the bidirectional image conversion and processing flow reconstruction network, it is necessary to separately train the conversion of RGB images into RAW images and the conversion of RAW images into RGB images. After the training is completed, they are combined to realize the conversion of RGB images into RAW images and then into RGB images. The loss functions of the two networks both use absolute error loss: in, is the RGB and RAW images reconstructed by the network, I rgb , I raw It is a true RGB, RAW image.
4. The method for low-light image reconstruction based on deep learning according to claim 3, characterized in that: The residual attention fusion module takes the input feature T input After five channel spatial attention modules, it passes through a convolutional layer of size 3×3 and is added to the input feature T to obtain the output feature T RAFM ; The channel space attention module takes the input feature T input First, pass through a convolutional layer of size 3×3, a ReLU activation function, and a convolutional layer of size 3×3 to obtain the feature T a , and then pass through average pooling, a convolutional layer with a size of 1×1, a ReLU activation function, a convolutional layer with a size of 1×1, a Sigmoid activation function, and the feature T a Multiply to get feature T c , the feature T c After average pooling and maximum pooling respectively, they are connected and then passed through a 3×3 convolutional layer, a Sigmoid activation function, and a 1×1 convolutional layer to obtain the feature T. s , and the characteristic T c And the input features T input Add the output feature T CSAM .
5. The method for low-light image reconstruction based on deep learning according to claim 4, characterized in that: The color correction module first passes the input RGB image through a Gaussian blur, then passes it through a 3×3 convolution layer, two residual attention fusion modules, a 3×3 convolution layer, and a Sigmoid activation function, and multiplies it with the input feature T2 in the bidirectional image conversion and processing flow to obtain feature T color , and then the feature T color Add to input feature T2 to get feature T CCM .
6. The method for low-light image reconstruction based on deep learning according to claim 5, characterized in that: In step S2, the low-light sensor model is based on a CMOS sensor, which converts the photons of incident light into electrons, then from electrons to voltages, and finally from voltages to digital signals to simulate a noisy electronic imaging pipeline; due to the quantum properties of light, the number of uncertain electrons collected is Poisson distributed: (I+N p )~P(I) Where N p represents photon shot noise, P represents Poisson distribution, and I is the number of photoelectrons proportional to the scene illumination; the process of converting electrons into voltage uses a zero-mean Gaussian distribution to model the line noise: N read ~TL(λ,μ c ,s TL ) N r ~N(0,σ r ) where λ and σ TL They represent the shape parameter and scale parameter of the read noise, μ c is the DC noise component. r is the scale parameter of the row noise; the process of converting from voltage to digital signal uses uniform distribution to model the quantization noise: Where U(·) is a uniform distribution, q is a quantization step size, and the low-light sensor model is: N=N p +N read +N r +N q Where N p is the photon shot noise, N read is the read noise, N r is the line noise, N q is the quantization noise.
7. The method for low-light image reconstruction based on deep learning according to claim 6, characterized in that: In the step S3, the U-net network performs nonlinear mapping on the input image through 5 downsampling convolution blocks to obtain a feature image after nonlinear mapping, wherein each convolution block is sequentially composed of a convolution layer containing [16, 32, 64, 128, 128] convolution kernels of size 3*3, a batch normalization layer and a ReLU activation function, and the up and down sampling operations use bilinear downsampling; the original feature image after nonlinear mapping obtained by downsampling is gradually upsampled through 5 upsampling convolution blocks, and the feature images obtained in the downsampling process are spliced to finally obtain a reconstructed low-light enhanced image, wherein each convolution block is sequentially composed of a convolution layer containing [16, 32, 64, 128, 128] convolution kernels of size 3*3, a batch normalization layer and a PRelu activation function.
8. The method for low-light image reconstruction based on deep learning according to claim 7, characterized in that: The loss of the U-net network is calculated as: Loss = Loss_L2 + Loss_ssim Where Loss_L2 is the mean square error of the pixel values between the reconstructed low-light image and the ground truth image, and Loss_ssims represents the structural similarity exponential loss function between the reconstructed low-light image and the ground truth image.
9. The method for low-light image reconstruction based on deep learning according to claim 1, characterized in that: The step S4 specifically includes: inputting the real low-light image into the U-net network trained in step S3 to obtain a reconstructed low-light enhanced image, and evaluating the reconstructed low-light enhanced image using peak signal-to-noise ratio, structural similarity and image similarity.
10. A low-light image reconstruction system based on deep learning, the system being used for the low-light image reconstruction method based on deep learning according to any one of claims 1 to 9, characterized in that: The invention comprises an image conversion module, a low-light noise module, a model training module and an image reconstruction module: the image conversion module constructs a bidirectional image conversion and processing flow reconstruction network and an adaptive exposure synthesis network, collects image samples to train the two networks, the low-light noise module connects the bidirectional image conversion and processing flow reconstruction network and the adaptive exposure synthesis network, uses a low-light sensor model to model the noise, and obtains a synthesized low-light image, the model training module uses the synthesized low-light image to train a preset U-net network, and the image reconstruction module inputs the real low-light image into the U-net network trained by the model training module for reconstruction, and obtains a low-light reconstructed image.
Citation Information
Patent Citations
Low-illumination image denoising enhancement method based on attention mechanism
CN116433516A