RetinexNet improved low-illumination image enhancement method based on CMOS image sensor and electronic equipment

Abstract

The invention relates to the technical field of deep learning image processing, in particular to a RetinexNet improved low-illumination image enhancement method based on a CMOS image sensor and electronic equipment, and the method comprises the following steps: S1, collecting and preprocessing an image; s2, decomposing the network model; s3, a blind de-noising network model is established; s4, a brightness adaptive adjustment model; s5, performing image fusion module and output; compared with the prior art, the method has the advantages that the brightness, detail recovery and noise suppression effects of the low-illumination image can be remarkably improved, and the method has remarkable advantages in image quality, structure retention and visual effect, and is particularly suitable for image enhancement in a low-illumination environment; the method provided by the invention can be widely applied to the image enhancement aspect of mobile equipment in a low-light environment, such as a smart phone, a tablet personal computer, a notebook computer, a digital camera, a security camera and the like, so as to capture and transmit image and video signals.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning image processing, and specifically to a Retin exNet improved low-light image enhancement method and electronic equipment based on a CMOS image sensor. Background Art

[0002] With the development of digital image processing technology, low-light image enhancement has become an important research topic in computer vision. Low-light images often face problems such as insufficient brightness, excessive noise and blurred details. Especially in the visual systems of mobile devices such as mobile phones, robots and autonomous driving, how to enhance the image quality of low-light images is crucial for subsequent analysis. Although traditional enhancement methods (such as histogram equalization, gamma correction and Retinex algorithm) improve brightness and contrast, they are prone to over-enhancement, noise amplification, and it is difficult to balance the detail recovery of highlight and low-light areas. Solving the balance between image brightness, detail recovery and noise suppression, accurately enhancing brightness while retaining image details, and avoiding overexposure of highlights or blurring of low lights are research problems at this stage.

[0003] With the rapid development of deep learning technology, image enhancement methods based on convolutional neural networks (CNNs) have gradually become an important research direction in the field of low-light image processing. Compared with traditional image enhancement methods, deep learning methods can automatically learn features from a large amount of image data through an end-to-end training model, and optimize image brightness, contrast, details, and noise according to different scenarios and needs. However, although deep learning-based image enhancement methods have improved the quality of low-light images to a certain extent, existing technologies still face multiple technical challenges. First, high computational complexity is a major problem currently faced by deep learning methods. Deep neural networks usually require a large amount of training data and computing resources, especially in image processing applications, which places high demands on hardware, especially for devices with limited resources. Secondly, noise and detail loss are common in low-light images. Traditional denoising methods often blur image details, while over-enhancing image details may amplify noise. How to find a balance between denoising and detail recovery has become a major technical challenge. In addition, the effective separation of illumination and reflection components in low-light images remains a technical bottleneck for deep learning methods. Most existing technologies fail to fully consider the separation of illumination and reflection parts of images, resulting in detail loss when brightness is enhanced, or poor illumination effect when details are restored. Although some methods (such as RetinexNet) try to separate illumination and reflection components, this separation process is often not accurate enough in low-light environments, affecting the enhancement effect.

[0004] For the low-light RetinexNet image enhancement method, existing patent literature has also proposed some solutions, such as the CNN-based low-light image enhancement method with color restoration and edge sharpening effects with application number 202110428992.8, which aims to restore image color, improve details and enhance edges; through the improved RetinexNet algorithm, the low-light image is first decomposed into reflection component and illumination component, and denoising, sharpening and color adjustment are performed respectively, and then the final image is synthesized by element-by-element multiplication; however, this method may have limited restoration effect for images with extremely low light or high noise, and the detail enhancement in high-contrast areas still needs to be optimized. The HSV and AM-RetinexNet-based image enhancement method with application number 202210917717.7 converts the RGB image into HSV components, adaptively adjusts and enhances the saturation and brightness components respectively, and finally restores it to an RGB image and performs adaptive color restoration; using the AM-RetinexNet network, this method can effectively process low-illumination images with non-uniform brightness such as underground spaces, and improve the problem of uneven image brightness distribution; however, this method requires image decomposition and reconstruction, which may result in the enhancement effect being less than expected in extremely complex backgrounds or dynamic scenes, and there is a certain instability. A low-light image enhancement method based on the improved RetinexNet with application number 202410601363.4 includes three main parts: a decomposition network, a brightness adjustment network and a denoising network; by using residual modules and jump connections in the decomposition network, image details are improved and noise is suppressed; the brightness adjustment network is combined with the DAN attention mechanism to improve the contrast of the illumination component and restore details; the denoising network further reduces noise and optimizes image quality; this method effectively improves the contrast, details and noise control of the image; however, this method mainly focuses on brightness and detail enhancement, and the color restoration processing is relatively weak, which may cause color distortion or unnatural phenomena in some images. Patent No. 202210644966.3 is a low-light image enhancement method with adaptive RetinexNet under a fusion strategy. DecomNet is used to decompose low-light images and extract illumination and reflectivity. RestorationNet and EnhanceNet are used to reduce noise and enhance illumination for reflectivity respectively. Finally, the image is reconstructed by fusing low-light images, coarsely enhanced images, and virtual overexposed images. This method effectively improves the color distortion problem, retains edge and detail information, and suppresses noise. However, the fusion process of virtual overexposed images with low-light images and coarsely enhanced images may lead to unstable image quality in some cases, especially in complex backgrounds or high dynamic range images, which leads to limited fusion. Summary of the invention

[0005] The object of the present invention is to provide a RetinexNet improved low-light image enhancement method and electronic device based on a CMOS image sensor, which can solve the shortcomings of traditional low-light image enhancement methods in detail restoration, brightness enhancement and noise suppression.

[0006] Based on the above purpose, the present invention adopts the following technical solution:

[0007] A low-light image enhancement method based on RetinexNet improved by CMOS image sensor, comprising the following steps:

[0008] S1. Image acquisition and preprocessing: Use CMOS image sensor to acquire low-light images; use microlens-based illumination optimization method to process low-light images through the focusing and light collecting functions of microlens array to improve the brightness of low-light images; and preprocess low-light images, including normalization, color correction and size adjustment, to output low-light images S low h ;

[0009] S2, decomposition network model: use the hybrid attention mechanism to decompose the low-light image S output in step S1 low h Decomposed into reflection component R low and the illumination component I low h :

[0010] S low h =R low I low h

[0011] Reflection component R low Represents the surface details and texture information of the object in the image, while the illumination component I low h represents the brightness change in the image, which is affected by the lighting conditions. Through the deep learning decomposition network, this process can effectively separate the brightness and details of the low-light image, providing a more accurate basis for subsequent processing steps such as illumination enhancement and noise suppression. The decomposed target minimizes the reconstruction error to learn the reflection and illumination components.

[0012] S3, blind denoising network model: input the reflection component R obtained in step S2 low Perform image denoising and dynamically adjust the denoising intensity according to the noise level of the image to obtain the optimized reflection component map R low ′; This model can accurately protect the edge details of the image while removing image noise, avoiding edge blur or detail loss during the denoising process, thereby ensuring that the clarity and structural information of the image are effectively preserved;

[0013] S4, brightness adaptive adjustment model: input the illumination component I obtained in step S2 low h , a multi-scale attention mechanism is used to focus on the illumination component I low h Enhance in different areas to get the adjusted illumination component map I low ′; The model improves the brightness of low-light areas through adaptive adjustment, while effectively suppressing over-enhancement of highlight areas to avoid detail loss or overexposure; the multi-scale attention mechanism can dynamically adjust the brightness enhancement intensity according to the local features of the image, thereby achieving a balance of brightness in the global and local ranges and improving the overall visual effect of the image;

[0014] S5, image fusion module and output: The optimized reflection component image R obtained in step S3 is converted into low ' and the adjusted illumination component map I obtained in step S4 low ′ is fused to obtain the final image and output it; an image fusion algorithm is used to ensure the balance of image brightness, details and contrast, so that the final image maintains the clarity of the highlight area and enhances the brightness of the low-light area.

[0015] Preferably, the specific process of the illumination optimization method based on microlenses in step S1 image acquisition and preprocessing includes:

[0016] S11, modular grouping of microlenses: modularly grouping the microlens array according to the symmetry characteristics of the microlens array, grouping the microlenses in the same direction into one group, and the translation amount of each group of microlenses is the same, so that the translation operation can be uniformly performed;

[0017] S12, microlens array translation operation: according to a specific direction of the microlenses in the pixel structure (such as a horizontal direction or a vertical direction), calculate and perform a translation operation of the microlens array so that the translation amount of each group of microlenses is consistent;

[0018] S13, light intensity enhancement: By translating the microlens array, the light intensity incident on the photodiode is increased, thereby improving the light receiving efficiency of the image sensor, increasing the light intensity of each pixel, and obtaining an image with enhanced brightness and clearer details.

[0019] Preferably, the specific process of the S3 blind denoising network model includes:

[0020] S31, evaluating noise intensity: estimating the noise intensity of the input image, and evaluating the noise level of each pixel or local area;

[0021] S32, adjusting and optimizing the denoising process: according to the noise estimation result obtained in step S31, using the denoising processing module to adaptively adjust the denoising intensity and dynamically optimize the denoising process;

[0022] S33, Adaptive edge protection: In the denoising process, the adaptive edge sharpening module is used to enhance the edge clarity of the edge area to avoid blurring of the image edge.

[0023] Preferably, the specific process of evaluating the noise intensity in step S31 includes:

[0024] Assume that the noise estimate of the local area W(x,y) is The noise intensity estimate calculated by the noise estimation module is It can be estimated by the change of gray value in the local area:

[0025]

[0026] Where W(x, y) is the local area, (i, j) represents the coordinates of any point in the area; |W| is the number of pixels in the area; R(i, j) is the original reflection component of the local area; R low (i, j) is the reflection component after denoising or smoothing in the local area.

[0027] Preferably, the specific process of adjusting and optimizing the denoising process in step S32 includes:

[0028] Assume that the denoising intensity is λ(x,y), which is related to the local noise intensity Generally speaking, the greater the noise intensity, the greater the denoising intensity;

[0029] The denoising module is implemented using a fully convolutional neural network model, which consists of a four-layer encoder-decoder structure with skip connections. During the encoding process, a downsampling module consisting of 5×5 convolutions is used for downsampling operations:

[0030] F down =Conv 5×5 (R low (i,j))*ReLU()

[0031] In the formula, F down is the downsampled feature map, Conv 5×5 It is a 5×5 convolution operation, and ReLU() is the activation function;

[0032] The residual structure allows the network to learn residual mappings instead of directly learning unmapped features, which helps alleviate the gradient vanishing problem; the output F of the residual block res It is expressed as:

[0033] F res=Conv 5×5 (F down )+F down

[0034] In the decoding process, 2×2 convolution is used for upsampling; the upsampled feature map F up express:

[0035] F up =Conv 2×2 (F res )*ReLU()

[0036] The encoder and decoder are connected through a jump connection. The jump connection directly transfers the feature map in the encoder to the corresponding layer of the decoder to preserve the details and edge information of the image. The output F after the jump connection is skip :

[0037] F skip =F up +F down

[0038] A 3×3 convolution kernel is added to each jump connection for channel matching to preserve image details and reduce noise, which helps to preserve image details and reduce noise; the output F after channel matching match :

[0039] F match =Conv 2×2 (F skip )

[0040] The adaptive denoising process is:

[0041] R deno (x,y)=f deno (F match ,λ(x,y))

[0042] In the formula, R deno (x,y) represents the reflected component after denoising, f deno is the denoising function based on the fully convolutional neural network, λ(x,y) is the denoising strength calculated according to the noise estimation module;

[0043] The denoising strength λ(x,y) is expressed as:

[0044]

[0045] Where α and β are hyperparameters used to adjust the effect of noise intensity on denoising; this can increase the denoising intensity in areas with strong noise, while the denoising intensity is low in areas with low noise, thereby effectively preserving details.

[0046] Preferably, the specific process of step S33 adaptive edge protection includes:

[0047] S331, edge detection: The gradient of the image is calculated through convolution operation. The edge detection Sobel operator calculates the gradient of the image in the horizontal and vertical directions respectively to obtain the edge strength E(x,y):

[0048]

[0049] In the formula, G x (x,y) and G y (x, y) are the gradients of the image in the horizontal and vertical directions respectively; Sobel x and Sobel y It is a Sobel filter in the horizontal and vertical directions; P(x,y) is the pixel value of the pixel at the (x,y) position; the greater the intensity E(x,y) of the edge area, the stronger the edge of the area.

[0050] S332, adaptive sharpening algorithm: dynamically adjust the sharpening strength according to the edge strength E(x, y) obtained in step S331; specifically, in the edge area, the sharpening strength should be larger; in the smooth area, the sharpening strength is smaller; after adaptive sharpening, R sharp (x,y) is expressed as:

[0051]

[0052] In the formula, λ sharp is the adjustment coefficient of the sharpening strength, E(x,y) is the edge detection result, which indicates the edge strength of the point. It is the reflection component of the image after smoothing (such as the image after Gaussian blur);

[0053] In the edge area, the value of E(x,y) is large and the sharpening strength λ sharp The details of the area will be enhanced, while in the non-edge area, E(x,y) is smaller and the sharpening intensity is lower to avoid over-sharpening:

[0054]

[0055] Among them, λ max Indicates the maximum sharpening strength, E max is the maximum value of the edge strength.

[0056] S333, edge sharpening enhancement: Combine adaptive sharpening and contrast enhancement technology to further improve the sharpness and layering of the image, and enhance the details and edges of the image; especially in low-light environments, the edges of the image are more likely to be blurred; the contrast enhancement formula is:

[0057] Renhanced (x,y)=R sharp (x,y)·(1+μE(x,y))

[0058] Where μ is the contrast enhancement coefficient, which is used to further enhance the edge contrast and make the image sharper. enhanced (x, y) is also the image reflection component R after the final sharpening of the image sha (x,y)=R low ′.

[0059] Preferably, the specific process of step S4 of adaptively adjusting the brightness model includes:

[0060] A lighting enhancement network based on SccUnet is proposed. The network uses dilated convolution in the first two layers to expand the receptive field and extract more image features, thereby enhancing the network's perception of low-illuminance areas. In order to better focus on the low-illuminance areas in the image and avoid image blur caused by traditional upsampling and downsampling operations, a common convolution layer with self-correcting convolution blocks is introduced in the design, and the coordinate attention mechanism is integrated into the jump connection. This design enables the network to capture the features of low-illuminance areas more accurately, thereby effectively improving the image brightness and retaining detail information. In addition, by connecting the output of each deconvolution layer in the dilation path and adopting a multi-scale fusion strategy, the loss of feature information is further reduced, thereby improving the overall quality of the image and the ability to restore details.

[0061] Preferably, the specific process of the image fusion module and output in step S5 includes:

[0062] S51, multi-scale convolution layer: Based on the traditional convolution layer, multiple scale convolution kernels are added to extract the reflection component R from different scales through multiple scale convolution kernels. low ′ and the illumination component I low ′ features; in order to better preserve the details of the image; multi-scale operations can capture different frequency information of the image, helping the network to focus on the global brightness and local details of the image at the same time:

[0063]

[0064] In the formula, Conv1, Conv2 and Conv3 represent convolution operations of different scales, and ⊕ represents the connection operation; each convolution layer extracts features of different scales to maintain the details of the image;

[0065] S52, residual module: Adding residual modules to the network ensures the stability of information flow through jump connections and solves the gradient vanishing problem in deep networks. The residual module achieves efficient feature transmission by directly adding the input to the output, while improving the ability to retain details during the decomposition process.

[0066] The processed reflection component R low a and the illumination component I low a The calculation of can be done through the residual module:

[0067]

[0068] Where Residual represents the output of the residual module, and Represents the previous state of the reflection component and the illumination component respectively.

[0069] S53, skip connection and adaptive weight adjustment: Introduce skip connection to ensure that low-level features can be directly transmitted to high-level layers to retain detailed information; use adaptive weight mechanism in skip connection to dynamically adjust the information flow between different layers, and automatically adjust the reflection component R according to the content and context information of the input image. low a and the illumination component I low a The weight of the reflection component and lighting component The operation of skip connection can be expressed as:

[0070]

[0071] In the formula, λ R and λ I is the weight coefficient dynamically adjusted according to the input image.

[0072] S54, image fusion and output: the reflection component and lighting component Fuse and output the final image.

[0073] Preferably, the specific process of image fusion and output in step S54 includes:

[0074] Final image S final (x,y) is represented as:

[0075]

[0076] Where w1(x, y) and w2(x, y) are weighted functions used for weighting, indicating the relative contribution of the illumination component and the reflection component. They are usually calculated dynamically based on the local features of the image (such as image brightness, edge intensity, and noise estimation). Generally speaking, the highlight area (area with higher brightness) can maintain a higher reflection component. This preserves edges and details; low-light areas (areas with lower brightness) can enhance the lighting component This increases brightness and prevents the image from being too dark.

[0077] The weight functions w1(x,y) and w2(x,y) are expressed as:

[0078]

[0079] w2(x,y)=1-w1(x,y)

[0080] I light (x, y) is the illumination intensity of the image, through the illumination component Get; I thresh is a preset threshold used to distinguish between highlight areas and low light areas; γ is an adjustment factor used to control the speed of change of the weight.

[0081] An electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, any step in any of the above-mentioned RetinexNet improved low-light image enhancement methods based on a CMOS image sensor is implemented.

[0082] The beneficial effects of the present invention include:

[0083] The present invention introduces a microlens-based illumination optimization algorithm, which can significantly enhance the brightness of the image during the image acquisition stage, improve the light reception efficiency, reduce the problem of light attenuation, and effectively avoid overexposure; the translation operation of the microlens array increases the light reception intensity, further improves the imaging quality of the image sensor, and provides a high-quality image foundation for subsequent image enhancement.

[0084] The present invention adopts a decomposition network model mixed with an attention mechanism, which can accurately decompose low-light images into a reflection component (containing detail information) and an illumination component (containing brightness information); through a deep learning algorithm, the brightness and details of the image are effectively separated, providing a clear basis for subsequent brightness enhancement, denoising, and sharpening steps, avoiding the processing difficulties caused by the mixing of brightness and detail information in traditional methods.

[0085] The present invention uses an adaptive edge-protected blind denoising network to dynamically adjust the denoising strength according to the noise level of the input image; while achieving effective noise removal, it can also avoid the problems of image edge blur and detail loss that may be caused by traditional denoising methods; the network model ensures that the clarity and structural information of the image can be effectively preserved, especially the details and edge information in the image, further improving the visual effect under low light conditions.

[0086] The present invention adopts adaptive brightness adjustment to enhance visual effects. By applying a multi-scale attention mechanism, the brightness of low-light areas can be enhanced, and excessive enhancement of highlight areas can be suppressed, thereby achieving a balance between the overall brightness and details of the image. The mechanism dynamically adjusts the brightness enhancement intensity according to the local features of the image, ensuring a balance in brightness within the global and local ranges, and avoiding overexposure and detail loss that may occur in traditional methods.

[0087] Compared with the prior art, the present invention can significantly improve the brightness, detail restoration and noise suppression effects of low-light images, and has significant advantages in image quality, structure retention and visual effects, and is particularly suitable for image enhancement in low-light environments; the method of the present invention can be widely used in image enhancement of mobile devices in low-light environments, such as smart phones, tablet computers, laptops, digital cameras, security cameras, etc., to capture and transmit image and video signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a schematic diagram of the process of the present invention;

[0089] Figure 2 The microlens-based illumination optimization method of the present invention;

[0090] Figure 3 It is a schematic diagram of the decomposition network model of the present invention;

[0091] Figure 4 Schematic diagram of the blind denoising network model of the present invention;

[0092] Figure 5 It is a schematic diagram of the adaptive edge protection algorithm module of the present invention;

[0093] Figure 6 It is a schematic diagram of the brightness adaptive adjustment model of the present invention. DETAILED DESCRIPTION

[0094] The following is a further explanation of the present invention in conjunction with specific embodiments. Figure 1 As shown, this embodiment is a RetinexNet improved low-light image enhancement method based on a CMOS image sensor, comprising the following steps:

[0095] S1. Image acquisition and preprocessing: Use CMOS image sensor to acquire low-light images, and use microlens-based illumination optimization method to process low-light images through the focusing and light collecting functions of microlens array; use CMOS image sensor to improve the light receiving efficiency of the image, reduce light attenuation, while maintaining the details and contrast of the image, and avoid overexposure, such as Figure 2 As shown, the main process includes:

[0096] First, the brightness adjustment model is performed. The original image S(x, y) is a two-dimensional image, where x and y are the spatial coordinates of the image. The brightness of the image can usually be expressed as:

[0097] S low (x,y)=R low I low (x,y)→S low h (x,y)=R low I low h (x,y)

[0098] S11, modular grouping of microlenses: modularly grouping the microlens array according to the symmetry characteristics of the microlens array, grouping the microlenses in the same direction into one group, and the translation amount of each group of microlenses is the same, so that the translation operation can be uniformly performed;

[0099] S12, microlens array translation operation: according to the specific direction of the microlens in the pixel structure (such as the horizontal direction or the vertical direction), the microlens array translation operation is calculated and performed so that the translation amount of each group of microlenses is consistent; the calculation formulas of the translation amounts Δx and Δy are:

[0100] Δx=f x ·Δd x Δy=f y ·Δd y

[0101] In the formula, f x and f y are the translation factors in the horizontal and vertical directions, Δd x and Δd y is the translation distance of the microlens, which is set based on the illumination requirements of the image sensor and the size of the microlens array; the illumination intensity after translation I shift (x,y) is:

[0102] I shift (x,y)=I low (x+Δx,y+Δy)

[0103] S13, light intensity enhancement: By shifting the microlens array, the light intensity incident on the photodiode is increased, thereby improving the light receiving efficiency of the image sensor and increasing the light intensity I of each pixel. lighy (x,y)=I low h (x,y):

[0104]

[0105] Where L(i,j) is the light intensity I at position (i,j) low , W(xi,yj) is the weight of the influence of the microlens translation on the receiving area of ​​the photodiode; where α(x,y) is the light intensity magnification factor, which depends on the low light and high light conditions of the image area:

[0106]

[0107] In the formula, f s (S low (x, y)) is the image feature function extracted at scale s, which is usually implemented through a convolutional neural network (CNN) or other feature extraction algorithms, and α0 is the global brightness enhancement coefficient.

[0108] After completing the image acquisition, the collected low-light image S low Perform preprocessing, including normalization, color correction, and size adjustment, and output the low-light image S low h .

[0109] S2, decomposition network model: Figure 3 As shown, a hybrid attention mechanism is used to convert the low-light image S output in step S1 into low h Decomposed into reflection component R low and the illumination component I low h :

[0110] S low h =R low I low h

[0111] Reflection component R low Represents the surface details and texture information of the object in the image, while the illumination component I low h represents the brightness change in the image, which is affected by the lighting conditions. Through the deep learning decomposition network, this process can effectively separate the brightness and details of the low-light image, providing a more accurate basis for subsequent processing steps such as illumination enhancement and noise suppression. The decomposed target minimizes the reconstruction error to learn the reflection and illumination components.

[0112] Optimize the total loss function L of the decomposition network model dec By reconstruction loss L rec , reflection component consistency loss L ref , illumination component smoothing loss L ill and the structural similarity loss L ssim Composition: L recis the mean square error, which is used to measure the reconstruction error at the pixel level to ensure that the reconstruction of the image is consistent with the original image to reduce the decomposition error; L ref Ensure that the reflection components of low-light images and high-light images are consistent; L ill Constrain the smoothness of the illumination component to reduce noise and detail interference; L ssim Measures the structural similarity of the decomposed reflection components.

[0113] L dec =L rec +α ir L ref +α is L ill +α ss L ssim

[0114] L rec =‖S low -R low I low ‖2

[0115] L ref =‖R low -R high ‖1

[0116]

[0117] L ssim =1-SSIM(R low ,R high )

[0118] In the formula, α ir ,α is ,α ss is the weight coefficient of each item in the loss function. high is the reflectance component of the specular image.

[0119] S3, blind denoising network model: input the reflection component R obtained in step S2 low Perform image denoising and dynamically adjust the denoising intensity according to the noise level of the image to obtain the optimized reflection component map R low ′; This model can accurately protect the edge details of the image while removing image noise, avoiding edge blur or detail loss during the denoising process, thereby ensuring that the clarity and structural information of the image are effectively preserved; the specific process includes:

[0120] S31, evaluating noise intensity: evaluating the noise intensity of the input image through a noise estimation module, and estimating the noise level of each pixel or local area;

[0121] Assume that the noise estimate of the local area W(x,y) is The noise intensity estimate calculated by the noise estimation module is It can be estimated by the change of gray value in the local area:

[0122]

[0123] Where W(x, y) is the local area, (i, j) represents the coordinates of any point in the area; |W| is the number of pixels in the area; R(i, j) is the original reflection component of the local area; R low (i, j) is the reflection component after denoising or smoothing in the local area.

[0124] S32, adjusting and optimizing the denoising process: According to the noise estimation result obtained in step S31, the denoising processing module is used to adaptively adjust the denoising intensity and dynamically optimize the denoising process, such as Figure 4 shown.

[0125] Assume that the denoising intensity is λ(x,y), which is related to the local noise intensity Generally speaking, the greater the noise intensity, the greater the denoising intensity.

[0126] The denoising module is implemented using a fully convolutional neural network model, which consists of a four-layer encoder-decoder structure with skip connections. During the encoding process, a downsampling module consisting of 5×5 convolutions is used for downsampling operations:

[0127] F down =Conv 5×5 (R low (i,j))*ReLU()

[0128] In the formula, F down is the downsampled feature map, Conv 5×5 It is a 5×5 convolution operation and ReLU() is the activation function.

[0129] The residual structure allows the network to learn residual mappings instead of directly learning unmapped features, which helps alleviate the gradient vanishing problem; the output F of the residual block res It is expressed as:

[0130] F res =Conv 5×5 (F down )+F down

[0131] In the decoding process, 2×2 convolution is used for upsampling; the upsampled feature map F up express:

[0132] F up =Conv 2×2 (F res)*ReLU()

[0133] The encoder and decoder are connected through a jump connection. The jump connection directly transfers the feature map in the encoder to the corresponding layer of the decoder to preserve the details and edge information of the image. The output F after the jump connection is skip :

[0134] F skip =F up +F down

[0135] A 3×3 convolution kernel is added to each jump connection for channel matching to preserve image details and reduce noise, which helps to preserve image details and reduce noise; the output F after channel matching match :

[0136] F match =Conv 2×2 (F skip )

[0137] The adaptive denoising process is:

[0138] R deno (x,y)=f deno (F match ,λ(x,y))

[0139] In the formula, R deno (x,y) represents the reflected component after denoising, f deno is the denoising function based on the fully convolutional neural network, and λ(x,y) is the denoising strength calculated according to the noise estimation module.

[0140] The denoising strength λ(x,y) is expressed as:

[0141]

[0142] Where α and β are hyperparameters used to adjust the effect of noise intensity on denoising; this can increase the denoising intensity in areas with strong noise, while the denoising intensity is low in areas with low noise, thereby effectively preserving details.

[0143] The total loss function L generated in the adaptive denoising process den-total By the edge loss L edge , denoising loss L noise and adversarial loss L adver Composition: L edge Ensure the clarity of the image edges; L noise Used to control the suppression of image noise to ensure that the denoised image is consistent with the target image; L adverThe realism of the image is enhanced by generative adversarial networks (GANs), ensuring that the denoised image is more visually natural.

[0144] The weighted sum of each part loss is used to balance denoising, edge preservation and image realism, and the adaptive denoising total loss function L is obtained. den-total :

[0145] L den-total =λ edge L deno +λ noise L noise +λ adver L adver

[0146]

[0147] λ edge , noise and λ adver A hyperparameter to balance various losses; R tar Represents the target reflection component before denoising; N is the total number of pixels in the image.

[0148] S33, Adaptive edge protection: Figure 5 As shown in the figure, in the denoising process, an adaptive edge sharpening module is used to enhance edge clarity in the edge area to avoid blurring of the image edge. The specific process includes:

[0149] S331, edge detection: The gradient of the image is calculated through convolution operation. The edge detection Sobel operator calculates the gradient of the image in the horizontal and vertical directions respectively to obtain the edge strength E(x,y):

[0150]

[0151] In the formula, G x (x,y) and G y (x, y) are the gradients of the image in the horizontal and vertical directions respectively; Sobel x and Sobel y It is a Sobel filter in the horizontal and vertical directions; P(x,y) is the pixel value of the pixel at the (x,y) position; the greater the intensity E(x,y) of the edge area, the stronger the edge of the area.

[0152] S332, adaptive sharpening algorithm: dynamically adjust the sharpening strength according to the edge strength E(x, y) obtained in step S331; specifically, in the edge area, the sharpening strength should be larger; in the smooth area, the sharpening strength is smaller; after adaptive sharpening, R sharp (x,y) is expressed as:

[0153]

[0154] In the formula, λ sharp is the adjustment coefficient of the sharpening strength, E(x,y) is the edge detection result, which indicates the edge strength of the point. It is the reflection component of the image after smoothing (such as the image after Gaussian blur).

[0155] In the edge area, the value of E(x,y) is large and the sharpening strength λ sharp The details of the area will be enhanced, while in the non-edge area, E(x,y) is smaller and the sharpening intensity is lower to avoid over-sharpening:

[0156]

[0157] Among them, λ max Indicates the maximum sharpening strength, E max is the maximum value of the edge strength.

[0158] S333, edge sharpening enhancement: Combine adaptive sharpening and contrast enhancement technology to further improve the sharpness and layering of the image, and enhance the details and edges of the image; especially in low-light environments, the edges of the image are more likely to be blurred; the contrast enhancement formula is:

[0159] R enhanced (x,y)=R sharp (x,y)·(1+μE(x,y))

[0160] Where μ is the contrast enhancement coefficient, which is used to further enhance the edge contrast and make the image sharper. enhanced (x, y) is also the image reflection component R after the final sharpening of the image sha (x,y)=R low ′.

[0161] The loss of the edge protection module process includes the gradient loss L gra and contrast loss L con2 ; L gra Measures the sharpness of image edges, L con2 In order to enhance the layering of image details; the total loss function of image sharpening L sha It is expressed as:

[0162] L sha =L gra +λ gr-co L con2

[0163]

[0164] In the formula, Respectively represent the gradients of the sharpened image reflection component and the target image reflection component. The closer the edge strength is to the target image, the smaller the loss is. sha (x,y) is the reflected component of the image after sharpening, is the neighborhood mean of the reflected component of the sharpened image; N is the total number of pixels in the image. The lower the loss value, the more the image layering meets the expected effect; λ gr-co is the weight coefficient of sharpening loss.

[0165] S4, brightness adaptive adjustment model: input the illumination component I obtained in step S2 low h , a multi-scale attention mechanism is used to focus on the illumination component I low h Enhance in different areas to get the adjusted illumination component map I low ′; The model improves the brightness of low-light areas through adaptive adjustment, while effectively suppressing excessive enhancement of highlight areas to avoid detail loss or overexposure; the multi-scale attention mechanism can dynamically adjust the brightness enhancement intensity according to the local features of the image, thereby achieving a balance of brightness in the global and local ranges and improving the overall visual effect of the image.

[0166] like Figure 6 As shown, in step S4, a light enhancement network based on SccUnet is proposed. The network uses dilated convolution in the first two layers to expand the receptive field and extract more image features, thereby enhancing the network's perception of low-illuminance areas. In order to better focus on the low-illuminance areas in the image and avoid image blur caused by traditional upsampling and downsampling operations, a common convolution layer with a self-correcting convolution block is introduced in the design, and the coordinate attention mechanism is integrated into the jump connection. This design enables the network to capture the features of low-illuminance areas more accurately, thereby effectively improving the image brightness and retaining detail information. In addition, by connecting the output of each deconvolution layer in the dilation path and adopting a multi-scale fusion strategy, the loss of feature information is further reduced, thereby improving the overall quality of the image and the ability to restore details.

[0167] In the first two layers of SccUnet, dilated convolution is used to expand the receptive field; this helps the network extract more feature information in a larger range, especially in low-light areas; dilated convolution can effectively capture the dependencies between distant pixels without increasing the amount of computation by inserting holes between convolution kernels, thereby improving the network's ability to perceive spatial information; the dilated convolution process outputs the value y(x) of position x in the image:

[0168]

[0169] Where x(k) is the pixel value in the input image, w(xk) is the dilated convolution kernel, is the index set of the convolution kernel, and x is the position of the output image; the dilated convolution increases the receptive field by adjusting the dilation rate of w(xk).

[0170] The network designs self-correction convolution blocks (SccBlocks), which can better capture low-light areas in images and avoid image blur caused by traditional upsampling and downsampling operations; these convolution blocks adaptively adjust the convolution weights so that the network can focus on the key features of low-light areas, thereby improving the ability to restore image details; the process can be expressed as:

[0171]

[0172] Where W(x) is the weight adaptively adjusted according to the local information of the image, x(k) is the local area in the input feature map, and finally the enhanced feature map is generated by the adaptive convolution block.

[0173] The coordinate attention mechanism is introduced into the network, which can further enhance the network's attention to low-light areas. SccUnet integrates the coordinate attention mechanism into the jump connection. The coordinate attention mechanism introduces position information in the spatial dimension, which enables the network to pay more attention to the low-light areas of the image more accurately, thereby effectively enhancing the brightness of these areas and maintaining the details of other areas. The coordinate attention weight can be expressed as:

[0174]

[0175] In the formula, score coord (x, y) represents the feature score at the position coordinate. After normalization, A coord (x,y) is the attention weight at each position.

[0176] S5, image fusion module and output: The optimized reflection component image R obtained in step S3 is converted into low ' and the adjusted illumination component map I obtained in step S4 low ′ to obtain the final image and output it; use the image fusion algorithm to ensure the balance of image brightness, details and contrast, so that the final image can maintain the clarity of the highlight area and enhance the brightness of the low-light area. The specific process includes:

[0177] S51, multi-scale convolution layer: Based on the traditional convolution layer, multiple scale convolution kernels are added to extract the reflection component R from different scales through multiple scale convolution kernels. low ′ and the illumination component Ilow ′ features; in order to better preserve the details of the image; multi-scale operations can capture different frequency information of the image, helping the network to focus on the global brightness and local details of the image at the same time:

[0178]

[0179] In the formula, Conv1, Conv2 and Conv3 represent convolution operations of different scales, and ⊕ represents a connection operation; each convolution layer extracts features of different scales to preserve the details of the image.

[0180] S52, residual module: Adding a residual module to the network ensures the stability of information flow through jump connections and solves the gradient vanishing problem in deep networks. The residual module achieves efficient feature transmission by directly adding the input to the output, while improving the ability to retain details during the decomposition process.

[0181] The processed reflection component R low a and the illumination component I low a The calculation of can be done through the residual module:

[0182]

[0183] Where Residual represents the output of the residual module, and Represents the previous state of the reflection component and the illumination component respectively.

[0184] S53, skip connection and adaptive weight adjustment: Introduce skip connection to ensure that low-level features can be directly transmitted to high-level layers to retain detailed information; use adaptive weight mechanism in skip connection to dynamically adjust the information flow between different layers, and automatically adjust the reflection component R according to the content and context information of the input image. low a and the illumination component I low a The weight of the reflection component and lighting component The operation of skip connection can be expressed as:

[0185]

[0186] In the formula, λ R and λ I is the weight coefficient dynamically adjusted according to the input image.

[0187] S54, image fusion and output: the reflection component and lighting component Fuse and output the final image.

[0188] Final image S final (x,y) is represented as:

[0189]

[0190] Where w1(x, y) and w2(x, y) are weighted functions used for weighting, indicating the relative contribution of the illumination component and the reflection component. They are usually calculated dynamically based on the local features of the image (such as image brightness, edge intensity, and noise estimation). Generally speaking, the highlight area (area with higher brightness) can maintain a higher reflection component. This preserves edges and details; low-light areas (areas with lower brightness) can enhance the lighting component This increases brightness and prevents the image from being too dark.

[0191] The weight functions w1(x,y) and w2(x,y) are expressed as:

[0192]

[0193] w2(x,y)=1-w1(x,y)

[0194] In the formula, I light (x, y) is the illumination intensity of the image, through the illumination component Get; I thresh is a preset threshold used to distinguish between highlight areas and low light areas; γ is an adjustment factor used to control the speed of change of the weight.

[0195] The above description is only a further explanation of the present invention in combination with specific embodiments. All descriptions do not limit the scope of protection of the present invention. Any changes or replacement schemes that can be easily thought of by any technician in this field within the technical scope disclosed by the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A RetinexNet improved low-light image enhancement method based on CMOS image sensor, characterized in that: The following steps are involved: S1. Image acquisition and preprocessing: Use CMOS image sensor to acquire low-light images; use microlens-based illumination optimization method to process low-light images through the focusing and light collecting functions of microlens array to improve the brightness of low-light images; and preprocess low-light images, including normalization, color correction and size adjustment, to output low-light images S low h ; S2, decomposition network model: use the hybrid attention mechanism to decompose the low-light image S output in step S1 low h Decomposed into reflection component R low and the illumination component I low h ; S3, blind denoising network model: input the reflection component R obtained in step S2 low Perform image denoising and dynamically adjust the denoising intensity according to the noise level of the image to obtain the optimized reflection component map R low ′; S4, brightness adaptive adjustment model: input the illumination component I obtained in step S2 low h , a multi-scale attention mechanism is used to focus on the illumination component I low h Enhance in different areas to get the adjusted illumination component map I low ′; S5, image fusion module and output: The optimized reflection component image R obtained in step S3 is converted into low ' and the adjusted illumination component map I obtained in step S4 low ′ is fused to obtain the final image and output it.

2. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 1, characterized in that: The specific process of the illumination optimization method based on microlenses in the image acquisition and preprocessing of step S1 includes: S11, modular grouping of microlenses: modularly grouping the microlens array according to the symmetry characteristics of the microlens array, grouping the microlenses in the same direction into one group, and the translation amount of each group of microlenses is the same, so that the translation operation can be uniformly performed; S12, microlens array translation operation: according to the specific direction of the microlenses in the pixel structure, calculate and perform the translation operation of the microlens array so that the translation amount of each group of microlenses is consistent; S13, light intensity enhancement: By translating the microlens array, the light intensity incident on the photodiode is increased, thereby improving the light receiving efficiency of the image sensor, increasing the light intensity of each pixel, and obtaining an image with enhanced brightness and clearer details.

3. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 1, characterized in that: The specific process of the blind denoising network model in step S3 includes: S31, evaluating noise intensity: estimating the noise intensity of the input image, and evaluating the noise level of each pixel or local area; S32, adjusting and optimizing the denoising process: according to the noise estimation result obtained in step S31, using the denoising processing module to adaptively adjust the denoising intensity and dynamically optimize the denoising process; S33, Adaptive edge protection: In the denoising process, the adaptive edge sharpening module is used to enhance the edge clarity of the edge area to avoid blurring of the image edge.

4. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 3, characterized in that: The specific process of evaluating the noise intensity in step S31 includes: Assume that the noise estimate of the local area W(x,y) is The noise intensity estimate calculated by the noise estimation module is It can be estimated by the change of gray value in the local area: Where W(x, y) is the local area, (i, j) represents the coordinates of any point in the area; |W| is the number of pixels in the area; R(i, j) is the original reflection component of the local area; R low (i, j) is the reflection component after denoising or smoothing in the local area.

5. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 3, characterized in that: The specific process of adjusting and optimizing the denoising process in step S32 includes: The denoising processing module is implemented using a fully convolutional neural network model, including a four-layer encoder-decoder structure with skip connections; a downsampling module composed of 5×5 convolutions is used during the encoding process to perform downsampling operations: F down =Conv 5×5 (R low (i,j))*ReLU() In the formula, F down is the downsampled feature map, Conv 5×5 It is a 5×5 convolution operation, and ReLU() is the activation function; The output F of the residual block res It is expressed as: F res =Conv 5×5 (F down )+F down In the decoding process, 2×2 convolution is used for upsampling; the upsampled feature map F up It is expressed as: F up =Conv 2×2 (F res )*ReLU() The encoder and decoder are connected via a jump connection, which directly transfers the feature map in the encoder to the corresponding layer of the decoder; the output F after the jump connection skip : F skip =F up +F down A 3×3 convolution kernel is added to each jump connection for channel matching to preserve image details and reduce noise, which helps to preserve image details and reduce noise; the output F after channel matching match : F match =Conv 2×2 (F skip ) The adaptive denoising process is: R deno (x,y)=f deno (F match ,λ(x,y)) In the formula, R deno (x,y) represents the reflected component after denoising, f deno is the denoising function based on the fully convolutional neural network, λ(x,y) is the denoising strength calculated according to the noise estimation module; The denoising strength λ(x,y) is expressed as: Where α and β are hyperparameters used to adjust the effect of noise intensity on denoising.

6. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 3, characterized in that: The specific process of step S33 adaptive edge protection includes: S331, edge detection: calculate the gradient of the image through convolution operation to obtain edge strength; S332, adaptive sharpening algorithm: dynamically adjust the sharpening intensity according to the edge intensity obtained in step S331; S333, Edge Sharpening Enhancement: Combines adaptive sharpening and contrast enhancement technology to further improve the sharpness and layering of the image, and enhance the details and edges of the image.

7. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 1, characterized in that: The specific process of the brightness adaptive adjustment model in step S4 includes: A lighting enhancement network based on SccUnet is used, and dilated convolutions are adopted in the first two layers to expand the receptive field and extract more image features, thereby enhancing the network's perception of low-light areas. Ordinary convolution layers with self-correcting convolution blocks are introduced, and the coordinate attention mechanism is integrated into the jump connection.

8. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 1, characterized in that: The specific process of the image fusion module and output in step S5 includes: S51, multi-scale convolution layer: extract reflection components R from different scales through convolution kernels of multiple scales low ′ and the illumination component I low ′ features; S52, residual module: The residual module directly adds the input to the output and outputs the processed reflection component R low a and the illumination component I low a ; S53, skip connection and adaptive weight adjustment: Introduce skip connection to ensure that low-level features can be directly transmitted to high-level layers to retain detailed information; use adaptive weight mechanism in skip connection to dynamically adjust the information flow between different layers, and automatically adjust the reflection component R according to the content and context information of the input image. low a and the illumination component I low a The weight of the reflection component and lighting component S54: Image fusion and output: The reflection component and lighting component Fuse and output the final image.

9. The improved low-light image enhancement method based on RetinexNet of CMOS image sensor according to claim 8, characterized in that: The specific process of image fusion and output in step S54 includes: Final image S final (x,y) is represented as: Where w1(x, y) and w2(x, y) are weight functions used for weighting, indicating the relative contributions of the illumination component and the reflection component; The weight functions w1(x, y) and w2(x, y) are expressed as: w2(x,y)=1-w1(x,y) I light (x, y) is the illumination intensity of the image, through the illumination component Get; I thresh is a preset threshold used to distinguish between highlight areas and low light areas; γ is an adjustment factor used to control the speed of change of the weight.

10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, any step in the RetinexNet improved low-light image enhancement method based on a CMOS image sensor as described in any one of claims 1 to 9 is implemented.

Images