A low-light image enhancement method and system

Abstract

The application provides a low-light image enhancement method and system, and a training method of an enhancement model comprises the following steps: inputting a training sample set into a low-light image enhancement model, wherein the low-light image enhancement model comprises an initial tone adjustment subnetwork and an initial noise removal subnetwork; inputting the training sample into the initial tone adjustment subnetwork, outputting a first target image by a first tone adjustment module, inputting the first target image and a low-light image into a second tone adjustment module to output a second target image, and minimizing a tone loss function to obtain a trained tone adjustment subnetwork; inputting a third target image output by the trained tone adjustment subnetwork from the training sample set into the initial noise removal subnetwork, generating first predicted filter weights and second predicted filter weights by using a residual network and a U-shaped network on the third target image, generating third filter weights by the first predicted filter weights and the second predicted filter weights through a third convolutional network, inputting the third target image into a fourth convolutional network to generate a result image based on the third filter weights, and minimizing noise loss to obtain a trained low-light image enhancement model. The application is more efficient in calculation, and has a better image enhancement effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for enhancing low-light images. Background Technology

[0002] Low-light conditions during image capture, such as nighttime, dimly lit indoor environments, backlighting, or overcast skies, often obscure many details and crucial information in low-light images. This not only affects the visual experience for the human eye but also limits the application scenarios of computer vision algorithms. Therefore, illumination enhancement for low-light images has significant application value. The goal of low-light enhancement techniques is to improve image brightness and contrast, perform color correction, and remove noise to recover hidden details. Thus, low-light image enhancement has broad significance in numerous applications, including improving human perception of image information and expanding the application scope of computer vision models.

[0003] In existing low-light image enhancement techniques, unsupervised learning methods often produce enhancement results that do not accurately reflect human visual perception due to the lack of labeled data for supervision. Supervised learning methods, on the other hand, rely on standard images for supervision of tone and noise quality, making it impossible to precisely adjust tone and noise separately, thus resulting in lower accuracy of the enhancement results. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a low-light image enhancement method and system to eliminate or improve the problem of poor accuracy in tone adjustment and noise reduction in existing low-light image enhancement technologies.

[0005] One aspect of the present invention provides a method for training a low-light image enhancement model, the method comprising the following steps: Obtain a training sample set, which includes multiple training samples, each training sample including a low-light image and a labeled image corresponding to normal lighting conditions; The training samples in the training sample set are input into a pre-established low-light image enhancement model; wherein, the low-light image enhancement model includes an initial tone adjustment sub-network and an initial noise removal sub-network, the initial tone adjustment sub-network includes a first tone adjustment module and a second tone adjustment module, and the initial noise removal sub-network includes a residual network, a U-shaped network, a third convolutional network and a fourth convolutional network; The first tone adjustment module uses a first convolutional neural network to generate first tone mapping weights. The tone mapping algorithm adjusts the pixel values ​​of the low-light image based on the first tone mapping weights and outputs a first target image. The first target image and the low-light image in the input training samples are then fused into a fused image and input to the second tone adjustment module. The second tone adjustment module uses a second convolutional neural network to generate second tone mapping weights. The tone mapping algorithm adjusts the pixel values ​​of the fused image based on the second tone mapping weights and outputs a second target image. The trained tone adjustment subnetwork is obtained by minimizing a first tone loss function constructed based on the difference between the first target image and the labeled image in the input training samples, and minimizing a second tone loss function constructed based on the difference between the second target image and the labeled image in the input training samples. The training sample set is input into the third target image obtained by the trained tone adjustment sub-network, and then input into the initial noise removal sub-network. The residual network of the initial tone adjustment sub-network extracts the query feature matrix and key feature matrix of the third target image, and calculates the first prediction filter weight of the neighboring pixels of each pixel in the third target image with itself as the center based on the query feature matrix and the key feature matrix. The U-shaped network of the initial tone adjustment sub-network uses the third target image to predict multiple filter kernels and corresponding mixing coefficients of the third target image, and calculates the second prediction filter weights based on the multiple filter kernels and the mixing coefficients; The third convolutional network of the initial tone adjustment sub-network calculates the confidence score using the query feature matrix and the mixing coefficients; based on the confidence score, the first prediction filter weights and the second prediction filter weights are weighted and fused to obtain the third filter weights; The fourth convolutional network of the initial tone adjustment sub-network performs filtering processing on the third target image based on the third filtering weights to generate the result image; Minimize the noise loss function constructed based on the result image and the labeled images in the input training samples to obtain the trained noise removal subnetwork, thereby obtaining the trained low-light image enhancement model.

[0006] In some embodiments of the present invention, the tone mapping algorithm is a 3D LUT algorithm.

[0007] In some embodiments of the present invention, the tone mapping algorithm adjusts the pixel values ​​of the low-light image based on the first tone mapping weight and outputs a first target image, including: after adjusting the pixel values ​​of the low-light image, generating the first target image using a trilinear interpolation algorithm; The tone mapping algorithm adjusts the pixel values ​​of the fused image based on the second tone mapping weights and outputs a second target image, including: after adjusting the pixel values ​​of the fused image, generating the second target image using a trilinear interpolation algorithm.

[0008] In some embodiments of the present invention, the first prediction filtering weight of the neighboring pixels in the region centered on each pixel in the third target image is calculated based on the query feature matrix and the key feature matrix, using the following formula: ; in, This represents the first predictive filter weight. This represents the query feature matrix. This represents the transpose of the key feature matrix. Indicates The pixels adjacent to the central region Representing feature dimension, This indicates the position code.

[0009] In some embodiments of the present invention, a second predictive filter weight is calculated based on multiple filter kernels and the mixing coefficients, using the following formula: ; in, This represents the second predictive filter weights. Indicates multiple filter kernels. This represents the mixing coefficient. Indicates the index of the filter kernel.

[0010] In some embodiments of the present invention, the first prediction filter weight and the second prediction filter weight are weighted and fused based on the confidence level to obtain the third filter weight, as shown in the formula: ; in, This represents the third filter weight. This represents the first predictive filter weight. This represents the second predictive filter weights. This indicates the confidence level.

[0011] In some embodiments of the present invention, the fourth convolutional network of the initial tone adjustment subnetwork performs filtering processing on the third target image based on the third filtering weights to generate a result image, as shown in the formula: ; in, The resulting image is represented. This refers to the second target image. This represents the third filter weight.

[0012] Another aspect of the present invention provides a low-light image enhancement method, comprising: The low-light image to be processed is input into the low-light image enhancement model trained by any of the training methods described above, and the enhanced result image is output.

[0013] Another aspect of the present invention provides a low-light image enhancement system, including a processor, a memory, and a computer program stored in the memory, the processor being configured to execute the computer program, and when the computer program is executed, the system performing the steps of the method as described in any of the preceding claims.

[0014] Another aspect of the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0015] The beneficial effects of the present invention are at least as follows: The low-light image enhancement model proposed in this invention combines a tone adjustment subnetwork and a noise removal subnetwork to address the tone and noise issues in low-light images, respectively. This enables more accurate content restoration and noise suppression in low-light images captured under low-light conditions. The tone adjustment subnetwork performs two tone adjustments—a combination of global and local adjustments—to obtain a tone-restored image. The image then passes through a noise removal subnetwork, which employs a kernel-based denoising method to remove noise without affecting the tone. The method and system described in this invention achieve better low-light image enhancement results and have higher computational efficiency.

[0016] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0017] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1This is a flowchart of a low-light image enhancement method according to an embodiment of the present invention.

[0019] Figure 2 This is a diagram showing the overall structure of the TN-Net model according to another embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the 3D LUT algorithm flow according to another embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0022] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0023] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0024] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0025] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0026] Low-light image enhancement is an image processing technique that typically improves image quality by increasing brightness, contrast, and reducing noise, making images taken in low-light environments easier to observe and understand. This invention significantly improves the quality of low-light images by processing image tone and noise separately. Tone adjustment refers to enhancing the visual effect of an image by changing its color and tone distribution. This includes adjusting parameters such as color saturation, contrast, and hue to make the image appear clearer, more vibrant, or more natural. Noise reduction refers to reducing noise or unwanted colors in the image caused by sensor noise, ambient light, and other factors, thereby improving image quality and clarity. Since noise is a high-frequency signal, a low-pass filter can be used to remove it.

[0027] One embodiment of the present invention provides a training method for a low-light image enhancement model, such as... Figure 1As shown, the method includes the following steps S101 to S110: Step S101: Obtain the training sample set, which includes multiple training samples. Each training sample includes a low-light image and a corresponding labeled image under normal lighting conditions.

[0028] Step S102: Input the training samples from the training sample set into the pre-established low-light image enhancement model. The low-light image enhancement model includes an initial tone adjustment sub-network and an initial noise removal sub-network. The initial tone adjustment sub-network contains consecutive first tone adjustment modules and second tone adjustment modules. The initial noise removal sub-network includes a residual network, a U-shaped network, a third convolutional network, and a fourth convolutional network.

[0029] Step S103: The first tone adjustment module uses a first convolutional neural network to generate first tone mapping weights. The tone mapping algorithm adjusts the pixel values ​​of the low-light image based on these weights, outputting a first target image. The first target image and the low-light images from the input training samples are then fused into a fused image, which is input to the second tone adjustment module. The second tone adjustment module uses a second convolutional neural network to generate second tone mapping weights. The tone mapping algorithm adjusts the pixel values ​​of the fused image based on these weights, outputting a second target image.

[0030] Step S104: Minimize the first tone loss function constructed based on the difference between the first target image and the labeled image in the input training sample, and minimize the second tone loss function constructed based on the difference between the second target image and the labeled image in the input training sample to obtain the trained tone adjustment sub-network.

[0031] Step S105: Input the training sample set into the third target image obtained by the trained tone adjustment sub-network and input it into the initial noise removal sub-network.

[0032] Step S106: The residual network of the initial tone adjustment sub-network extracts the query feature matrix and key feature matrix of the third target image from the third target image. Based on the query feature matrix and key feature matrix, the first prediction filter weight of the neighboring pixels of each pixel in the third target image with itself as the center is calculated.

[0033] Step S107: The U-shaped network of the initial tone adjustment sub-network uses the third target image to predict multiple filter kernels and corresponding mixing coefficients of the third target image, and calculates the second prediction filter weights based on the multiple filter kernels and mixing coefficients.

[0034] Step S108: The third convolutional network of the initial tone adjustment sub-network calculates the confidence score using the query feature matrix and mixing coefficients. Based on the confidence score, the first and second prediction filter weights are weighted and fused to obtain the third filter weights.

[0035] Step S109: The fourth convolutional network of the initial tone adjustment sub-network filters the third target image based on the third filter weights to generate the result image.

[0036] Step S110: Minimize the noise loss function constructed based on the result image and the labeled images in the input training samples to obtain the trained noise removal sub-network, thereby obtaining the trained low-light image enhancement model.

[0037] In step S101, the first target image and the low-light image are fused into a fused image. This fusion can be achieved using a convolutional neural network or an image fusion algorithm. Image fusion algorithms can combine multiple images to generate a single image containing more information. Common image fusion algorithms include alpha fusion, pyramid fusion, and Poisson fusion. The fused image is then used for tone adjustment to output a second target image, which better preserves the details of the original image compared to the first target image.

[0038] In step S106, extracting the query feature matrix and key feature matrix of the third target image utilizes a self-attention mechanism based on the transformer model. The query feature matrix (Q, Query) is used to query the relevance of other positions in the sequence, and the key feature matrix (K, Key) represents information about other positions. Based on the query feature matrix and key feature matrix, the first prediction filter weight of the neighboring pixels in the region centered on each pixel in the second target image is calculated. That is, by calculating the similarity between the query feature matrix and the key feature matrix, it is determined which positions should be assigned higher weights.

[0039] In step S107, the U-Net typically utilizes a convolutional network for image segmentation, consisting of a symmetrical encoder and decoder. The encoder extracts features from the input image, while the decoder maps these features back to the original image size, generating pixel-level segmentation results. Therefore, multiple filter kernels and corresponding mixing coefficients can be generated by combining the features from the filter kernel generation module with the feature extraction function of the U-Net.

[0040] The third target image is processed by a residual network, enabling each pixel to communicate directly with its neighboring pixels at the pixel level. Therefore, this process can estimate more accurate filtering weights in textured regions of the image. The second predicted filtering weights generated by the U-shaped network for the third target image are better at denoising textureless regions of the image. The third filtering weights obtained by combining the two weights achieve a more balanced filtering effect.

[0041] The main difference between low-light images and labeled images lies in the control of lighting conditions during image capture. Taking the method for constructing the LOL dataset as an example, low-light images are captured using the same camera under low-light conditions, and a set of images with short and long exposure times are captured under normal lighting conditions. These images are then averaged to obtain noise-free labeled images. The acquired images are labeled and classified to distinguish between low-light images and labeled images under normal lighting conditions.

[0042] In some embodiments, the tone mapping algorithm is a 3D LUT algorithm.

[0043] 3D LUT (Three Dimensional Look-Up Table) is a technique for image processing and color correction. This algorithm maps pixel values ​​of an input image to output values ​​using a three-dimensional look-up table, thereby achieving color correction and adjustment. This algorithm is commonly used to adjust the hue, contrast, and saturation of an image, as well as to achieve specific color effects. By interpolating and looking up predefined color maps in three-dimensional space, the 3D LUT algorithm can perform efficient and accurate color correction on images.

[0044] In some embodiments, the tone mapping algorithm adjusts the pixel values ​​of the low-light image based on a first tone mapping weight and outputs a first target image, including: after adjusting the pixel values ​​of the low-light image, generating the first target image using a trilinear interpolation algorithm.

[0045] The tone mapping algorithm adjusts the pixel values ​​of the fused image based on the second tone mapping weights and outputs the second target image, including: after adjusting the pixel values ​​of the fused image, generating the second target image using a trilinear interpolation algorithm.

[0046] Trilinear interpolation is a technique for interpolating data in three-dimensional space. It combines the ideas of linear and bilinear interpolation to perform smooth interpolation on irregular or non-uniform grids. Trilinear interpolation uses a weighted average of the values ​​of the eight nearest data points to the target point, weighted by their distances, to obtain the interpolated result for the target point. This embodiment utilizes trilinear interpolation to more accurately estimate the values ​​between data points, achieving smooth interpolation, better preserving image details and texture, and making the image appear clearer and more natural.

[0047] In some embodiments, the first prediction filter weight of each pixel in the third target image, centered on itself, is calculated based on the query feature matrix and the key feature matrix, using the formula: ; in, Indicates the first predictive filter weight. Represents the query feature matrix. This represents the transpose of the key feature matrix. Indicates The central region of neighboring pixels, Representing feature dimension, This indicates the position code.

[0048] In some embodiments, the second predictive filter weight is calculated based on multiple filter kernels and mixing coefficients, using the following formula: ; in, Indicates the weights of the second predictive filter. Indicates multiple filter kernels. Represents the mixing coefficient. Indicates the index of the filter kernel.

[0049] In some embodiments, the first and second prediction filter weights are weighted and fused based on confidence levels to obtain the third filter weight, as shown in the formula: ; in, Indicates the third filter weight. Indicates the first predictive filter weight. Indicates the weights of the second predictive filter. Indicates the confidence level.

[0050] In some embodiments, the fourth convolutional network of the initial tone adjustment subnetwork performs filtering on the third target image based on the third filtering weights, as shown in the formula: ; in, Represents the resulting image. This represents the second target image. This represents the third filter weight.

[0051] This embodiment also provides a low-light image enhancement method, including: The low-light image to be processed is input into the low-light image enhancement model trained by any of the training methods mentioned above, and the enhanced result image is output.

[0052] Another embodiment of the present invention provides a low-light image enhancement method that offers a novel solution to the low-light enhancement problem by using supervision based on tonal and noise quality measurements respectively. This embodiment combines global and local tonal adjustment methods, proposing a TGA network model for tonal adjustment. Simultaneously, an RCA-BPN model is proposed to improve existing methods using a kernel-based denoising approach. The TGA network model and the RCA-BPN model are combined to form TN-Net, serving as the overall model architecture for implementing the low-light image enhancement method. The overall structure of TN-Net is as follows: Figure 2 As shown, the network includes a tone adjustment subnetwork (TGA network model) and a noise removal subnetwork (RCA-BPN model).

[0053] The tone adjustment sub-network adjusts the input image I to a tone-adjusted image T through a two-stage global adjustment (TGA) model. During the training of this model, the intermediate result image T′ of the first global adjustment and the result image T of the second global adjustment are processed through a loss function. and loss Each image is supervised by the labeled image GT. The noise removal subnetwork denoises the tone-adjusted image T using an RCA-BPN model consisting of a Region Center Attention (RCA) branch and a Context Prediction Network (BPN) branch, outputting the final result E. During the training of this model, E is calculated using a loss function. Supervised by the labeled image GT.

[0054] Among them, constructing loss and It is a global mapping function based on the full mapping algorithm. The function takes the pixel values ​​(r, g, b) of an image as input and outputs the pixel values ​​(r1, g1, b1) of the desired tone level adjustment. Calculation The mapping method requires two images: an input image and a target image. It constructs a function mapping relationship by statistically analyzing the values ​​of each pixel (r, g, b) in the input image and their corresponding pixels in the target image, and by calculating the average value of the corresponding target pixels for the same input pixel. This maps the tone of the input image to the level of the target image. Because this is a global statistical method and uses an arithmetic mean to avoid noise errors (in electronic cameras, the noise mean of an image is generally 0), it does not change the image noise distribution, only the image tone.

[0055] like Figure 2 As shown, the goal of the tone adjustment subnetwork is to change the tone of the input image I while maintaining its signal-to-noise ratio, and uses a loss function. and loss The network is trained to focus on tone adjustment. Global adjustment is used because it is stable in changing tone while maintaining the signal-to-noise ratio and is computationally efficient. However, the input low-light image I to be processed is typically very dark, with most pixels having very similar intensities close to 0. The small differences in intensity values ​​in the input image make accurate estimation of the global curve difficult, and small errors in estimating the global curve can lead to a loss of contrast in the adjusted image. Therefore, after obtaining the result image of the first adjustment... Then, to compensate for the lost contrast, a blending effect was added to the image after the second global adjustment. Local processing of I is used to obtain the intermediate input image F. Since F may not retain high tonal quality, ... Figure 2 As shown, it is adjusted a second time to generate the final tone adjustment result T.

[0056] like Figure 2 As shown, during the training of the noise removal subnetwork, the loss function is utilized. To drive the subnetwork to repair noise while maintaining tonal quality, the network architecture uses a kernel-based denoising framework, which denoises T by estimating the volume of the filter weights W and convolving T with the filter weights in W. The noise removal subnetwork constructs Region Center Attention (RCA) and Base Prediction Network (BPN) branches to estimate the filter weights W.

[0057] The motivation behind RCA is to use the self-attention mechanism from Transformer techniques to achieve weight estimation of the filter kernel. Compared to self-attention processing widely used in low-level vision problems, RCA's advantage lies in its ability to estimate the weights of the filter kernel for each pixel. It can calculate its relationship with local neighboring pixels centered on itself. The pixel-to-pixel relationship.

[0058] Another branch, BPN, extracts hierarchical local and global information through its multi-scale architecture and performs well in noise removal, especially in textureless regions.

[0059] Furthermore, such as Figure 3 As shown, a 3D LUT algorithm is used for global adjustment in the tone adjustment sub-network. The 3D-LUT model is named... Obtain the first global adjustment result from the input image I. ,Right now: ； This method uses a 4-layer convolutional neural network with a fully connected (FC) layer to estimate the mixed weights of the underlying 3D lookup tables (LUTs) from γ1 to γN. arrive Where M is the number of bins in each color channel, and N is the base number. The number of layers. In some embodiments, M=33 and N=3. Each CNN layer performs 3×3 convolutions, ReLU (Modified Linear Unit, a type of activation function), and pooling with a stride of 2. Basic Learning occurs during the training process. Then, through... Calculate the weighted average Using trilinear interpolation, denoted as TI, for such images... Perform 3D color conversion to obtain ,Right now Finally, using a fusion module named f, we can... and original input The images are fused to obtain a fused image F, i.e.: ; Here, f is a three-layer CNN, each layer consisting of a 3x3 convolution and ReLU. Next, a second 3D-LUT model (with the same structure as g1), named g2, is used to adjust F to T, i.e.: ; Region Center Attention (RCA) branch is used to estimate the filter kernel weights. This model is inspired by NAT models in the field of recognition. First, a ResNet (residual network) is used, named... Extract Q (query feature matrix) and K (query feature matrix) from T, i.e. Secondly, each pixel in Q... Feature classification slices. For each slices, and Attention between calculations, i.e.: ; in, Represents the query feature matrix. This represents the transpose of the key feature matrix. Indicates Centered Neighboring pixels in the region Representing feature dimension, Indicates position code, It is a pixel The prediction filter weights of adjacent pixels.

[0060] The calculation of this formula is not supported in popular deep learning libraries such as PyTorch and TensorFlow. Therefore, K and The multiplication of the three inputs is implemented in C++.

[0061] Another branch is constructed using the Basic Prediction Network (BPN) model. It uses a U-Net (U-shaped network) to predict a set of global basic filter kernels. and the corresponding mixing coefficient Filter kernel weights It is obtained from the following formula: ; in, Indicates the weights of the second predictive filter. Indicates multiple filter kernels. Represents the mixing coefficient. Indicates the index of the filter kernel.

[0062] Finally, the feature matrix Q and mixing coefficients are processed using a three-layer convolutional neural network (each layer includes 3x3 convolutions and ReLU, with the last layer having 2 feature channels) and softmax. Learning a confidence level Use C to fuse and ,get ,Right now: ; in, Indicates the third filter weight. Indicates the first predictive filter weight. Indicates the weights of the second predictive filter. Indicates the confidence level.

[0063] After obtaining W, perform a convolution operation on T to obtain the result E, that is: ; in, Represents the resulting image. This represents the second target image. This represents the final filter weights.

[0064] To train the tone adjustment subnetwork in the TGA model, using loss and These are used for the first global adjustment result T and the second global adjustment result T, respectively, that is: ; ; Among them, global 3D mapping and Based on the global mapping function of the full mapping algorithm Calculation. To train the noise removal subnetwork in the RCA-BPN model, the loss function is used. , used for the denoising result E, that is: ; Among them, in order to make the result Similar to the labeled image GT, using Instead Because when using During training, according to the MSE metric, the darker the learned E is, the lower the loss value will be. Therefore, this can mislead the model to learn to make E darker.

[0065] TN-Net has low computational complexity. In the tone adjustment subnetwork, the TGA model connects two 3D-LUT models, g1 and g2, and uses a three-layer convolutional neural network f. g1 and g2 are computationally efficient because the 3D-LUT models first downsample the input image to 256x256 before the CNN layers. f is also efficient because it is just a simple three-layer CNN. In the denoising subnetwork, the BPN branch is similar to a U-shaped network, which is computationally efficient because most layers are performed on downsampled features according to the U-shaped network style network structure. In summary, because each component of TN-Net is computationally efficient, the overall computational complexity of TN-Net is not high.

[0066] In summary, this invention provides a low-light image enhancement method and system, which utilizes a trained low-light image enhancement model to enhance low-light images. The training method for this low-light image enhancement model includes: acquiring a training sample set, which comprises multiple training samples, each including a low-light image and a corresponding labeled image under normal lighting conditions; establishing an initial low-light image enhancement model, which includes an initial tone adjustment sub-network and an initial noise removal sub-network; inputting the training samples into the initial tone adjustment sub-network, where a first tone adjustment module outputs a first target image; fusing the first target image with the low-light image and inputting the fused image into a second tone adjustment module, which outputs a second target image; constructing a tone loss function; and minimizing the loss to obtain the trained tone adjustment sub-network; inputting the second target image output by the trained tone adjustment sub-network into the noise removal sub-network; and using a residual network, a U-shaped network, a third convolutional network, and a fourth convolutional network to generate the result image, minimizing the noise loss, and obtaining the trained low-light image enhancement model. The low-light image enhancement model proposed in this invention combines a tone adjustment subnetwork and a noise removal subnetwork to address the tone and noise issues in low-light images, respectively. This enables more accurate content restoration and noise suppression in low-light images captured under low-light conditions. The method and system described in this invention achieve better low-light image enhancement results and have higher computational efficiency.

[0067] Corresponding to the above method, the present invention also provides a system comprising a computer device including a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory, wherein when the computer instructions are executed by the processor, the system implements the steps of the method as described above.

[0068] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0069] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0070] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0071] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a low-light image enhancement model, characterized in that, The method includes the following steps: Obtain a training sample set, which includes multiple training samples, each training sample including a low-light image and a labeled image corresponding to normal lighting conditions; The training samples in the training sample set are input into a pre-established low-light image enhancement model; wherein, the low-light image enhancement model includes an initial tone adjustment sub-network and an initial noise removal sub-network, the initial tone adjustment sub-network includes a first tone adjustment module and a second tone adjustment module, and the initial noise removal sub-network includes a residual network, a U-shaped network, a third convolutional network and a fourth convolutional network; The first tone adjustment module uses a first convolutional neural network to generate first tone mapping weights. The tone mapping algorithm adjusts the pixel values ​​of the low-light image based on the first tone mapping weights and outputs a first target image. The first target image and the low-light image in the input training samples are then fused into a fused image, which is input to the second tone adjustment module. The second tone adjustment module uses a second convolutional neural network to generate second tone mapping weights. The tone mapping algorithm adjusts the pixel values ​​of the fused image based on the second tone mapping weights and outputs a second target image. The trained tone adjustment subnetwork is obtained by minimizing a first tone loss function constructed based on the difference between the first target image and the labeled image in the input training samples, and minimizing a second tone loss function constructed based on the difference between the second target image and the labeled image in the input training samples. The training sample set is input into the third target image obtained by the trained tone adjustment sub-network, and then input into the initial noise removal sub-network. The residual network of the initial tone adjustment sub-network extracts the query feature matrix and key feature matrix of the third target image, and calculates the first prediction filter weight of the neighboring pixels of each pixel in the third target image with itself as the center based on the query feature matrix and the key feature matrix. The U-shaped network of the initial tone adjustment sub-network uses the third target image to predict multiple filter kernels and corresponding mixing coefficients of the third target image, and calculates the second prediction filter weights based on the multiple filter kernels and the mixing coefficients; The third convolutional network of the initial tone adjustment sub-network calculates the confidence score using the query feature matrix and the mixing coefficients; based on the confidence score, the first prediction filter weights and the second prediction filter weights are weighted and fused to obtain the third filter weights; The fourth convolutional network of the initial tone adjustment sub-network performs filtering processing on the third target image based on the third filtering weights to generate the result image; Minimize the noise loss function constructed based on the result image and the labeled images in the input training samples to obtain the trained noise removal subnetwork, thereby obtaining the trained low-light image enhancement model.

2. The training method for the low-light image enhancement model according to claim 1, characterized in that, The tone mapping algorithm is a 3D LUT algorithm.

3. The training method for the low-light image enhancement model according to claim 1, characterized in that, The tone mapping algorithm adjusts the pixel values ​​of the low-light image based on the first tone mapping weight and outputs a first target image, including: after adjusting the pixel values ​​of the low-light image, generating the first target image using a trilinear interpolation algorithm; The tone mapping algorithm adjusts the pixel values ​​of the fused image based on the second tone mapping weights and outputs a second target image, including: after adjusting the pixel values ​​of the fused image, generating the second target image using a trilinear interpolation algorithm.

4. The training method for the low-light image enhancement model according to claim 1, characterized in that, Based on the query feature matrix and the key feature matrix, the first prediction filter weight of the neighboring pixels in the region centered on each pixel in the third target image is calculated, using the following formula: ； in, This represents the first predictive filter weight. This represents the query feature matrix. This represents the transpose of the key feature matrix. Indicated by The pixels adjacent to the central region Representing feature dimension, This indicates the position code.

5. The training method for the low-light image enhancement model according to claim 1, characterized in that, The second predictive filter weight is calculated based on multiple filter kernels and the mixing coefficients, using the following formula: ； in, This represents the second predictive filter weights. Indicates multiple filter kernels. Indicates the mixing coefficient, Indicates the index of the filter kernel.

6. The training method for the low-light image enhancement model according to claim 4, characterized in that, Based on the confidence level, the first and second prediction filter weights are weighted and fused to obtain the third filter weight, as shown in the formula: ； in, This represents the third filter weight. This represents the first predictive filter weight. This represents the second predictive filter weights. This indicates the confidence level.

7. The training method for the low-light image enhancement model according to claim 4, characterized in that, The fourth convolutional network of the initial tone adjustment sub-network filters the third target image based on the third filtering weights to generate the result image, as shown in the formula: ； in, The resulting image is represented. This refers to the second target image. This represents the third filter weight.

8. A low-light image enhancement method, characterized in that, include: The low-light image to be processed is input into the low-light image enhancement model obtained by the training method of the low-light image enhancement model as described in any one of claims 1 to 7, and the enhanced result image is output.

9. A low-light image enhancement system, comprising a processor, a memory, and a computer program stored in the memory, characterized in that, The processor is configured to execute the computer program, and when the computer program is executed, the system implements the steps of the method as described in any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images