Image enhancement device, method, electronic device, and storage medium

Abstract

This invention provides an image enhancement apparatus, method, electronic device, and storage medium, relating to the field of image processing technology. The apparatus includes: an image acquisition module for acquiring a low-light image to be processed; an image enhancement module for inputting the low-light image into a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model; and a noise reduction module for inputting the first brightness-enhanced image into a noise reduction model to obtain noise information output by the noise reduction model, removing the noise information from the first brightness-enhanced image to obtain a second brightness-enhanced image of the low-light image. The brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction. The technical solution provided by this invention can suppress noise generation and reduce the loss of color information while ensuring image brightness enhancement effects.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to an image enhancement device, method, electronic device, and storage medium. Background Technology

[0002] With the development of multimedia technology, the application of images and videos is becoming increasingly widespread. However, due to limitations of image acquisition equipment or the influence of insufficient light environments such as nighttime, the acquired images may suffer from problems such as low visibility, lack of detail, high noise interference, and uneven lighting due to underexposure. This not only brings a poor visual experience but also affects the normal operation of image-based computer vision systems. Therefore, brightness enhancement for low-light images is very important.

[0003] In related technologies, low-light image enhancement can be achieved through histogram equalization-based enhancement methods, Retinex-based enhancement methods, and deep learning-based end-to-end enhancement methods. Histogram equalization-based enhancement methods can achieve good results when processing images with dark foregrounds and backgrounds, but over-enhancement in certain areas can easily lead to loss of detail and color deviation. Retinex-based enhancement methods have limitations in decomposing reflectance and illumination within the Retinex model; coarse estimation of reflectance values ​​can easily result in overexposure or underexposure in the enhanced image. While deep learning-based end-to-end methods can circumvent the limitations of physical principles, they tend to overlook image noise and the loss of color information during the enhancement process.

[0004] Therefore, how to suppress noise generation and reduce the loss of color information while ensuring the enhancement of image brightness is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The present invention provides an image enhancement apparatus, method, electronic device, and storage medium to suppress noise generation and reduce the loss of color information while ensuring image brightness enhancement.

[0006] This invention provides an image enhancement device, comprising:

[0007] The image acquisition module is used to acquire low-light images to be processed;

[0008] An image enhancement module is used to input the low-light image into a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model; the brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction, the multi-stream features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map, the image feature scales of the at least one branch feature map are different and different from the original image feature scale;

[0009] The noise reduction module is used to input the first brightness enhancement image into the noise reduction model, obtain the noise information output by the noise reduction model, remove the noise information from the first brightness enhancement image, and obtain the second brightness enhancement image of the low-light image; the noise reduction model is used to extract the noise information from the first brightness enhancement image.

[0010] According to an image enhancement apparatus provided by the present invention, the image enhancement module includes:

[0011] The first feature extraction unit is used to input the low-light image into the first feature extraction layer of the brightness enhancement model to obtain a first initial feature map output by the first feature extraction layer; the first feature extraction layer is used to extract image features from the low-light image.

[0012] The first splitting unit is used to input the first initial feature map into the first splitting layer of the brightness enhancement model to obtain the mainstream feature map and the at least one splitting feature map output by the first splitting layer; the first splitting layer is used to perform convolution on the first initial feature map at different image feature scales;

[0013] The first information interaction unit is used to input the mainstream feature map and the at least one sub-channel feature map into the first information interaction layer of the brightness enhancement model to obtain the first feature maps corresponding to the mainstream and the at least one sub-channel respectively output by the first information interaction layer; the first information interaction layer is used to interactively fuse the mainstream feature map and the at least one sub-channel feature map.

[0014] An image restoration unit is used to input the first feature map into the image restoration layer of the brightness enhancement model to obtain the first brightness enhancement image output by the image restoration layer; the image restoration layer is used to perform deconvolution fusion on the first feature maps corresponding to the main stream and the at least one branch stream.

[0015] According to an image enhancement apparatus provided by the present invention, the at least one split feature map includes a first split feature map and a second split feature map; the first split unit includes:

[0016] The first splitting subunit is used to input the first initial feature map into the mainstream layer of the first splitting layer to obtain the mainstream feature map output by the mainstream layer; the mainstream layer is used to perform convolution on the first initial feature map at the image feature scale of the first initial feature map;

[0017] The second splitting subunit is used to input the first initial feature map into the first sub-splitting layer of the first splitting layer to obtain the first splitting feature map output by the first sub-splitting layer; the first sub-splitting layer is used to perform a first downsampling on the first initial feature map and to perform convolution on the first feature map obtained by the first downsampling at the image feature scale of the first feature map.

[0018] The third splitting subunit is used to input the first splitting feature map and the mainstream feature map into the second sub-splitting layer of the first splitting layer to obtain the second splitting feature map output by the second sub-splitting layer; the second sub-splitting layer is used to perform a second downsampling on the mainstream feature map, and to perform convolution between the second downsampled second feature map and the first splitting feature map on the image feature scale of the second feature map; wherein, the sampling scale of the first downsampling and the second downsampling are different.

[0019] According to an image enhancement device provided by the present invention, the first information interaction unit includes:

[0020] A sampling subunit is used to input the mainstream feature map and the at least one sub-stream feature map into the sampling layer of the first information interaction layer to obtain the sampling feature map corresponding to each information stream output by the sampling layer; the information stream includes the mainstream and the at least one sub-stream; the sampling layer is used to sample the mainstream feature map and the at least one sub-stream feature map to the feature map size of each information stream respectively;

[0021] The fusion subunit is used to interactively fuse the sampled feature maps corresponding to each information stream to obtain the feature map corresponding to each information stream.

[0022] According to an image enhancement device provided by the present invention, the image restoration layer is specifically used to: for the first feature maps corresponding to the main stream and the at least one sub-stream respectively, starting from the first feature map corresponding to the information stream with the smallest feature map size, deconvolve the fused feature map of the current information stream to the feature map size of the next information stream in a progressively larger manner, and fuse the deconvoluted feature map with the first feature map corresponding to the next information stream, until the first feature map corresponding to the main stream is fused to obtain the first brightness enhanced image;

[0023] The fused feature map of the current information stream is obtained by deconvolving the first feature map of the previous information stream to the size of the feature map of the current information stream, and then fusing the deconvolved feature map with the first feature map of the current information stream.

[0024] According to an image enhancement device provided by the present invention, the brightness enhancement model is obtained by training an initial brightness enhancement model based on a low-light sample image, a first reference illuminance sample image corresponding to the low-light sample image, and an enhancement loss function.

[0025] The enhancement loss function is determined based on at least one of the pixel difference, structural similarity, and color difference between the low-light sample image and the first reference light sample image.

[0026] According to an image enhancement apparatus provided by the present invention, the noise reduction module includes:

[0027] The second feature extraction unit is used to input the first brightness enhancement image into the second feature extraction layer of the noise reduction model to obtain a second initial feature map output by the second feature extraction layer; the second feature extraction layer is used to extract image features from the first brightness enhancement image;

[0028] The second splitting unit is used to input the second initial feature map into the second splitting layer of the denoising model to obtain the denoising main feature map and at least one denoising splitting feature map output by the second splitting layer; the second splitting layer is used to perform convolution on the second initial feature map at different image feature scales;

[0029] The second information interaction unit is used to input the noise reduction mainstream feature map and the at least one noise reduction off-stream feature map into the second information interaction layer of the noise reduction model to obtain the third feature maps corresponding to the noise reduction mainstream and the at least one noise reduction off-stream output by the second information interaction layer; the second information interaction layer is used to interactively fuse the third feature maps of the noise reduction mainstream and the at least one noise reduction off-stream.

[0030] The noise extraction unit is used to input the third feature map into the noise extraction layer of the noise reduction model to obtain the noise information output by the noise extraction layer; the noise extraction layer is used to extract noise from the third feature map corresponding to the main noise reduction stream and the at least one noise reduction sub-stream.

[0031] An image enhancement apparatus according to the present invention further includes:

[0032] The model training module is used to acquire a second reference illuminance sample image, add noise to the second reference illuminance sample image to obtain a noisy sample image, and train an initial denoising model based on the noisy sample image, the second reference illuminance sample image and the denoising loss function to obtain the denoising model.

[0033] The noise reduction loss function is determined based on a first structural similarity and a second structural similarity. The first structural similarity is the structural similarity between the image of the noisy sample image after removing the noise output by the initial noise reduction model and the second reference illumination sample image. The second structural similarity is the similarity between the image of the second reference illumination sample image after adding the noise output by the initial noise reduction model and the noisy sample image.

[0034] The present invention also provides an image enhancement method, comprising:

[0035] Acquire the low-light image to be processed;

[0036] The low-light image is input into a brightness enhancement model to obtain a first brightness enhancement image output by the brightness enhancement model. The brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction. The multi-stream features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of the at least one branch feature map are different from those of the original image feature scale.

[0037] The first brightness-enhanced image is input into the noise reduction model to obtain the noise information output by the noise reduction model; the noise reduction model is used to extract the noise information of the first brightness-enhanced image.

[0038] The noise information is removed from the first brightness-enhanced image to obtain a second brightness-enhanced image of the low-light image.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image enhancement method as described above.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image enhancement method as described above.

[0041] The image enhancement apparatus, method, electronic device, and storage medium provided by this invention first perform brightness enhancement on the low-light image to be processed based on multi-flow feature interaction using a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model. Next, noise information is extracted from the first brightness-enhanced image using a noise reduction model. Then, the noise information is removed from the first brightness-enhanced image to obtain a second brightness-enhanced image of the low-light image, thereby suppressing noise generation while ensuring the image brightness enhancement effect. The multi-flow features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of the at least one branch feature map are different from those of the original image feature scale. Therefore, in the process of brightness enhancement of the low-light image based on multi-flow feature interaction, more semantic information can be obtained while preserving the original image information of the low-light image, retaining more details and color information of the low-light image, thereby suppressing noise generation and reducing the loss of color information while ensuring the image brightness enhancement effect. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is one of the structural schematic diagrams of the image enhancement device provided in the embodiments of the present invention;

[0044] Figure 2 This is a second schematic diagram of the structure of the image enhancement device provided in the embodiments of the present invention;

[0045] Figure 3 This is a schematic diagram illustrating the working principle of the first diversion unit in this embodiment of the invention;

[0046] Figure 4 This is a schematic diagram illustrating the working principle of the first information interaction unit in this embodiment of the invention;

[0047] Figure 5 This is a schematic diagram illustrating the working principle of the image enhancement device provided in this embodiment of the invention;

[0048] Figure 6 This is a flowchart illustrating the image enhancement method provided in an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] It should be noted that the serial numbers assigned to the objects described in this invention, such as "first" and "second", are only used to distinguish the objects being described and do not have any sequential or technical meaning.

[0052] The purpose of low-light image enhancement is to highlight useful features of an image while reducing or eliminating interfering information, making blurry or low-brightness images clearer or brighter. In dimly lit environments such as at night, including under moonlight, firelight, or lamplight, the surrounding light becomes complex, leading to a significant decrease in visible light and contrast, accompanied by substantial noise. Alternatively, uneven lighting can cause large differences in brightness between image areas. Improving the brightness of low-light images is particularly important for fields such as military, surveillance, and security.

[0053] Among the related image enhancement techniques, there are enhancement methods based on histogram equalization, enhancement methods based on Retinex theory, and end-to-end enhancement methods based on deep learning.

[0054] Among them, enhancement methods based on histogram equalization mainly expand the pixel dynamic range of the image histogram statistics, improving visual effects by increasing image contrast. These enhancement algorithms mainly include local histogram equalization and global histogram equalization. These methods achieve good results when processing images with dark foregrounds and backgrounds, and also have a significant advantage in processing time due to low computational cost. However, over-enhancement in certain areas can lead to loss of detail and color deviation. Although histogram equalization-based enhancement algorithms have a significant time advantage, because the brightness enhancement process considers the overall statistical characteristics of the image, they cannot achieve good results when processing complex scene images.

[0055] The main idea of ​​Retinex theory is that the color and brightness of an object perceived by the human eye are determined by the combined effects of the object's surface reflectance and incident light characteristics (or illumination characteristics), which are similar to the characteristics of brightness. Since the introduction of Retinex theory, numerous methods based on it have been proposed. Although these methods can achieve good results in some cases, they still have limitations in decomposing reflectance and illumination within the Retinex model. Designing image decomposition constraints applicable to various scenarios is a major challenge in image enhancement. Furthermore, since solving for the reflectance map is a coarse estimation process, coarse estimates can easily lead to overexposure or underexposure in the image enhancement results.

[0056] With the development of artificial intelligence and the excellent performance of neural networks in image processing, deep learning-based methods have rapidly developed in the field of image enhancement. These methods can be further divided into two categories based on whether they are based on Retinex theory. Both types of methods can achieve good image enhancement results. However, methods that directly generate enhancement results end-to-end tend to ignore the generation of noise, resulting in noisy enhancement results. On the other hand, deep learning methods based on Retinex theory tend to ignore the color cast phenomenon that occurs in low-light images during the enhancement process.

[0057] Based on this, embodiments of the present invention provide an image enhancement device, which may include: an image acquisition module for acquiring a low-light image to be processed; an image enhancement module for inputting the low-light image into a brightness enhancement model to obtain a first brightness enhancement image output by the brightness enhancement model; the brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction, wherein the multi-stream features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map, and the image feature scale of the at least one branch feature map is different from that of the original image feature scale; and a noise reduction module for inputting the first brightness enhancement image into a noise reduction model to obtain noise information output by the noise reduction model, removing the noise information from the first brightness enhancement image to obtain a second brightness enhancement image of the low-light image, wherein the noise reduction model is used to extract the noise information of the first brightness enhancement image. This device can suppress noise generation and reduce the loss of color information while ensuring the image brightness enhancement effect.

[0058] The following is combined with Figures 1-5 The image enhancement apparatus of the present invention will be described. This image enhancement apparatus can be implemented by software, hardware, or a combination of both.

[0059] Figure 1 An exemplary schematic diagram of one of the image enhancement devices provided in an embodiment of the present invention is shown, with reference to... Figure 1As shown, the image enhancement device may include an image acquisition module 110, an image enhancement module 120, and a noise reduction module 130.

[0060] The image acquisition module 110 is used to acquire the low-light image to be processed.

[0061] Image enhancement module 120 is used to input low-light image into brightness enhancement model to obtain first brightness enhancement image output by brightness enhancement model;

[0062] The noise reduction module 130 is used to input the first brightness enhancement image into the noise reduction model, obtain the noise information output by the noise reduction model, remove the noise information from the first brightness enhancement image, and obtain the second brightness enhancement image of the low-light image. The brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-flow feature interaction. The multi-flow features include the mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scale of the at least one branch feature map is different from that of the original image feature scale. The noise reduction model is used to extract the noise information from the first brightness enhancement image.

[0063] Specifically, low-light images are images captured by image acquisition devices in low-light environments.

[0064] Specifically, the brightness enhancement model can be trained on an initial brightness enhancement model based on a low-light sample image, a first reference light-light sample image corresponding to the low-light sample image, and an enhancement loss function. The enhancement loss function is determined based on at least one of the pixel difference, structural similarity, and color difference between the low-light sample image and the first reference light-light image.

[0065] For example, at least one split feature map determined based on the mainstream feature map can be obtained by downsampling the mainstream feature map at different image feature scales, such as downsampling the mainstream feature map by 2 times and 4 times respectively to obtain two split feature maps.

[0066] Alternatively, at least one split feature map determined based on the mainstream feature map may include: convolving the initial feature map of the low-light image while maintaining the image size to obtain the mainstream feature map; downsampling the initial feature map at different image feature scales to obtain feature maps corresponding to different image feature scales; convolving the feature map while maintaining its size to obtain the split feature map of the information stream containing that feature map; starting from the split feature map with the largest scale, progressively fusing each split feature map with the feature maps of the previous information streams through convolution to obtain the split feature map of the next information stream. The image feature scales of each split feature map are different.

[0067] For example, the initial feature map of a low-light image can be convolved while keeping the image size constant to obtain the main feature map; the initial feature map can be downsampled by 2 times, and the first feature map obtained by downsampling can be convolved while keeping the image size constant to obtain the first branch feature map; the main feature map can be downsampled by 4 times, and the second feature map obtained by downsampling can be convolved and fused with the first branch feature map at the image feature scale of the second feature map to obtain the second branch feature map.

[0068] The initial brightness enhancement model may include, but is not limited to, at least one of the following neural networks: Convolutional Neural Networks (CNN), Recurrent Neural Network Transducer (RNN), and Long Short-Term Memory (LSTM).

[0069] Since it is difficult to obtain image pairs with consistent scene content but inconsistent brightness in natural scenes, in this embodiment of the invention, a training sample dataset can be obtained by adjusting the image acquisition device or by post-processing the images acquired by the image acquisition device. For example, in this embodiment of the invention, at least one of the following databases can be used as the training sample dataset for the brightness enhancement model and the noise reduction model:

[0070] (1) Low-Light Image Pair Database (LOL) contains 500 low-light sample images and their corresponding normal exposure sample images, each image being, for example, 400*600 pixels in size. LOL obtains low-light sample images by adjusting the exposure time in natural scenes (including residential areas, suburbs, streets, etc.).

[0071] (2) The composite database is created by selecting M original images (e.g., 1000) from the RAISE database and using the interface provided by Adobe Lightroom to adjust the values ​​of the image brightness channels to composite these images into low-light sample images, thereby obtaining M corresponding image pairs. Finally, these image pairs are resized to a set size, such as 400*600.

[0072] It is understandable that the training sample dataset can directly use samples from these two databases, or select a predetermined number of image pairs from these two databases as the training sample dataset. The image pairs refer to low-light sample images and their corresponding baseline illumination sample images (i.e., normal exposure sample images).

[0073] Based on this training dataset, during the training phase of the brightness enhancement model, low-light sample images from the training dataset are used as the initial input to the brightness enhancement model. The output of the initial brightness enhancement model is obtained. Based on this output and the first reference illuminance sample image corresponding to the low-light sample image, the loss value of the enhancement loss function is calculated. The model parameters of the initial brightness enhancement model are adjusted according to this loss value. This process is iterated repeatedly until the loss value meets the set requirements, resulting in a trained brightness enhancement model. For example, during the training process, stochastic gradient descent can be used to update the weights of the initial brightness enhancement model, and the initial brightness enhancement model can be retrained using the updated weights.

[0074] For example, taking an enhancement loss function determined based on pixel differences, structural similarity, and color difference between a low-light sample image and a first reference illumination image as an example, this enhancement loss function may include: an L1 loss function (L1), a structural similarity loss function (L... s ) and color difference loss function (L c The details are as follows:

[0075] The L1 loss function provides a quantitative method to measure the pixel-level difference between the enhanced image and the real image output by the initial brightness enhancement model during training. Specifically, it measures the pixel-level difference between the enhanced image obtained after inputting a low-light sample image into the model and the corresponding first baseline illumination sample image. The L1 loss function can be expressed by the following formula (1):

[0076]

[0077] Where N represents the number of pixels in the image, y i Let f(S) represent the i-th pixel in the first reference illumination sample image corresponding to the low-illuminance sample image. i This represents the output of the initial brightness enhancement model, and ||·||1 represents the L1 paradigm.

[0078] The structural similarity loss function is an indicator for evaluating the structural similarity between two images. In this embodiment of the invention, the structure of the brightness enhancement model is multi-scale, and a multi-scale structural loss function can be used. The structural similarity MS(a1,a2) can be expressed by the following formula (2):

[0079]

[0080] Where a1 and a2 represent the low-light sample image and the corresponding first reference light-light sample image, l(a1,a2) represents luminance, and c(a1,a2) represents contrast. Let represent the structural similarity of the i-th pixel between a1 and a2; i represents the pixel, and m represents the total number of pixels; α, β, and γ represent weight parameters used to adjust their importance. The larger the structural similarity value, the higher the similarity between the two. In order to maintain the consistency of the loss function during training, the multi-scale structural loss function can be converted into the following formula (3):

[0081] L MS (a1,a2)=1-MS(a1,a2) (3)

[0082] Among them, L MS This represents the structural similarity loss.

[0083] Furthermore, the structural similarity loss function L can be further... S It is expressed as the following formula (4):

[0084] L S =L MS (y i ,f(S) i (4)

[0085] Among them, y i Let f(S) represent the i-th pixel in the first reference illumination sample image corresponding to the low-illuminance sample image. i This represents the output of the initial brightness enhancement model.

[0086] Because low-light images lose a lot of color information, the enhancement result is prone to color cast. Therefore, in this embodiment of the invention, CIEDE2000 is introduced as a color loss function to calculate the color difference between two images. Specifically, the real image and the enhanced image output by the model can be converted from the RGB color space to the Lab color space. Then, the color difference between each pixel in the first reference illumination sample image (real image) and the enhanced image output by the model is calculated according to the CIEDE2000 method. The color difference loss function L... c It can be expressed as the following formula (5):

[0087]

[0088] Where N represents the number of pixels in the image, ΔE represents the color difference of each pixel in the Lab color space between the two images, and y i Let f(S) represent the i-th pixel in the first reference illumination sample image corresponding to the low-illuminance sample image. i This represents the output of the initial brightness enhancement model.

[0089] Furthermore, the enhancement loss function can be expressed as the following formula (6):

[0090] L=ρL1+σLs +δL c (6)

[0091] Where L1 is the L1 loss function, L S L represents the structural similarity loss function. c Let denot be the color difference loss function, where ρ, σ, and δ are the weights of the loss function, respectively.

[0092] Thus, on the one hand, by introducing color loss into the enhancement loss function to measure the color similarity between the enhanced result and the real image, color difference can be considered during the image enhancement process, further avoiding color distortion in the enhancement result. On the other hand, the enhancement loss function is a joint loss function that considers multiple perspectives, enabling the image enhancement result to be closer to the real image.

[0093] For training the denoising model, since the above database lacks noisy images, in this embodiment of the invention, Poisson noise can be added to the normal exposure sample images by combining the distribution characteristics of noise in low exposure images, and the noise sample images with added noise and the corresponding normal exposure sample images can be used as the training dataset for training the denoising model.

[0094] Specifically, the denoising model can be obtained by training an initial denoising model based on a noisy sample image, a second reference illumination sample image, and a denoising loss function. The initial denoising model can include, but is not limited to, at least one neural network such as CNN, RNN, and LSTM. The denoising loss function can be determined based on a first structural similarity and a second structural similarity. The first structural similarity is the structural similarity between the noisy sample image after removing the noise output from the initial denoising model and the second reference illumination sample image. The second structural similarity is the similarity between the second reference illumination sample image after adding the noise output from the initial denoising model and the noisy sample image.

[0095] During the training process of the denoising model, a noisy sample image can be input into the initial denoising model to obtain the noise output by the initial denoising model. The noisy sample image is then subtracted from the noise to obtain the image enhancement result. Based on the image enhancement result and the second reference illumination sample image corresponding to the noisy sample image, the loss value of the denoising loss function is calculated. The model parameters of the initial denoising model are adjusted based on the loss value. This process is repeated iteratively until the loss value meets the set requirements, and a trained denoising model is obtained.

[0096] The image enhancement device provided by this invention first performs brightness enhancement on the low-light image to be processed based on multi-flow feature interaction using a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model. Then, a noise reduction model extracts noise information from the first brightness-enhanced image. Finally, the noise information is removed from the first brightness-enhanced image to obtain a second brightness-enhanced image of the low-light image, thereby suppressing noise generation while ensuring the image brightness enhancement effect. The multi-flow features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of these at least one branch feature map are different from those of the original image feature scale. Therefore, in the process of brightness enhancement of the low-light image based on multi-flow feature interaction, more semantic information can be obtained while preserving the original image information of the low-light image, retaining more details and color information of the low-light image, thereby suppressing noise generation and reducing the loss of color information while ensuring the image brightness enhancement effect.

[0097] based on Figure 1 The image enhancement apparatus corresponding to the embodiment, Figure 2 A second schematic diagram of the image enhancement device provided in an embodiment of the present invention is shown as an example. (Refer to...) Figure 2 As shown, the image enhancement device may further include a model training module 210. The model training module 210 can be used to acquire a second reference illuminance sample image, add noise to the second reference illuminance sample image to obtain a noisy sample image, and train an initial denoising model based on the noisy sample image, the second reference illuminance sample image, and a denoising loss function to obtain a denoising model.

[0098] The noise reduction loss function is determined based on the first structural similarity and the second structural similarity. The first structural similarity is the structural similarity between the image of the noisy sample image after removing the noise output of the initial noise reduction model and the second reference illumination sample image. The second structural similarity is the similarity between the image of the second reference illumination sample image after adding the noise output of the initial noise reduction model and the noisy sample image.

[0099] Specifically, the noise reduction loss function can be expressed as the following formula (7):

[0100] L N =L p +L n (7)

[0101] Among them, L p =L MS (y,y n -f(y n )) represents the first structural similarity, L n =LMs (y n ,y+f(y n )) represents the second structural similarity; y n f(y) represents the noisy sample image. n ) represents the noise output by the initial noise reduction model, and y represents the second reference illumination sample image.

[0102] based on Figure 1 or Figure 2 In one example embodiment of the image enhancement apparatus corresponding to the embodiments, the image enhancement module 120 may include:

[0103] The first feature extraction unit is used to input the low-light image into the first feature extraction layer of the brightness enhancement model to obtain the first initial feature map output by the first feature extraction layer; the first feature extraction layer is used to extract image features from the low-light image.

[0104] The first splitting unit is used to input the first initial feature map into the first splitting layer of the brightness enhancement model to obtain the main feature map and at least one splitting feature map output by the first splitting layer; the first splitting layer is used to perform convolution on the first initial feature map at different image feature scales;

[0105] The first information interaction unit is used to input the mainstream feature map and at least one sub-stream feature map into the first information interaction layer of the brightness enhancement model to obtain the first feature maps corresponding to the mainstream and at least one sub-stream respectively output by the first information interaction layer; the first information interaction layer is used to perform interactive fusion of the mainstream feature map and at least one sub-stream feature map.

[0106] The image restoration unit is used to input the first feature map into the image restoration layer of the brightness enhancement model to obtain the first brightness enhancement image output by the image restoration layer; the image restoration layer is used to perform deconvolution fusion on the first feature maps corresponding to the main stream and at least one branch stream.

[0107] For example, the first feature extraction layer may include a preset number of basic convolutional blocks, such as 3 basic convolutional blocks, each of which may consist of a convolutional layer with a stride of 1 and a kernel size of 3*3.

[0108] For example, at least one shunt feature map output by the first shunt layer may include a first shunt feature map and a second shunt feature map; the first shunt unit may include:

[0109] The first splitting subunit is used to input the first initial feature map into the mainstream layer of the first splitting layer to obtain the mainstream feature map output by the mainstream layer; wherein the mainstream layer is used to perform convolution on the first initial feature map at the image feature scale of the first initial feature map;

[0110] The second splitting subunit is used to input the first initial feature map into the first sub-splitting layer of the first splitting layer to obtain the first splitting feature map output by the first sub-splitting layer; wherein the first sub-splitting layer is used to perform a first downsampling on the first initial feature map and to perform convolution on the first feature map obtained by the first downsampling at the image feature scale of the first feature map;

[0111] The third splitting subunit is used to input the first splitting feature map and the mainstream feature map into the second sub-splitting layer of the first splitting layer to obtain the second splitting feature map output by the second sub-splitting layer; wherein the second sub-splitting layer is used to perform a second downsampling on the mainstream feature map, and to convolve the second feature map obtained by the second downsampling with the first splitting feature map on the image feature scale of the second feature map; wherein the sampling scales of the first downsampling and the second downsampling are different.

[0112] In this process, the feature maps in each information flow structure maintain the same size throughout the convolution process. Downsampling the main feature maps can expand the receptive field of the convolution kernel.

[0113] For example, Figure 3 An exemplary schematic diagram of the working principle of the first shunt unit is shown, with reference to... Figure 3 As shown, taking a convolution kernel size of 3, a first downsampling of 2 times, and a second downsampling of 4 times as an example, the first initial feature map is input into the mainstream layer. The mainstream layer convolves the first initial feature map, for example, with a convolution kernel size of 3 and a stride of 1, to obtain the mainstream feature map output by the mainstream layer. The first sub-splitter layer downsamples the first initial feature map by 2 times to obtain the first feature map, and then convolves the first feature map, for example, with a convolution kernel size of 3 and a stride of 2, to obtain the first sub-splitter feature map output by the first sub-splitter layer. The second sub-splitter layer downsamples the mainstream feature map by 4 times to obtain the second feature map. The second feature map is then convolved with the first sub-splitter feature map with a stride of 4, convolving to the same image feature scale as the second feature map, to obtain the second sub-splitter feature map output by the second sub-splitter layer. At this point, feature maps of the first initial feature map at different image feature scales can be obtained.

[0114] For example, the first information interaction unit may include:

[0115] The sampling subunit is used to input the mainstream feature map and at least one sub-stream feature map into the sampling layer of the first information interaction layer to obtain the sampling feature map corresponding to each information stream output by the sampling layer; the information stream includes the mainstream and at least one sub-stream; the sampling layer is used to sample the mainstream feature map and at least one sub-stream feature map to the feature map size of each information stream respectively;

[0116] The fusion subunit is used to interactively fuse the sampled feature maps corresponding to each information stream to obtain the first feature map corresponding to each information stream.

[0117] For example, combining Figure 3 , Figure 4 An exemplary schematic diagram of the working principle of the first information interaction unit is shown below. Figure 4 As shown, the first splitting unit passes through Figure 3 Taking the obtained mainstream feature map, first sub-stream feature map, and second sub-stream feature map as examples, the mainstream feature map can be sampled by 1x to obtain sampled feature map 41. The first sub-stream feature map can be upsampled by 2x to obtain sampled feature map 42, and the second sub-stream feature map can be upsampled by 4x to obtain sampled feature map 43. Then, the fusion subunit of the first information interaction unit interacts and fuses sampled feature maps 41, 42, and 43 to obtain the first feature map A corresponding to the mainstream. This first feature map A maintains the same scale as the mainstream feature map. Similarly, by sampling the first sub-stream feature map by 1x, downsampling the mainstream feature map by 2x, and upsampling the second sub-stream feature map by 2x, the obtained sampled feature maps are interactably fused through the fusion subunit of the first information interaction unit to obtain the first feature map B corresponding to the first sub-stream. This first feature map B maintains the same scale as the first sub-stream feature map. Similarly, the second split feature map is sampled by 1x, the mainstream feature map is downsampled by 4x, and the first split feature map is downsampled by 2x through the sampling layer. The sampled feature maps obtained are then fused through the fusion subunit of the first information interaction unit to obtain the first feature map C corresponding to the second split. The first feature map C maintains the same scale as the second split feature map.

[0118] In this way, by performing convolution (downsampling) or deconvolution (upsampling) operations on the feature maps of the three structures, we can obtain the downsampled or upsampled feature map corresponding to each structure. Then, the feature map of this structure is passed to the other two structures for interactive fusion, which can realize information interaction between the three structures and reduce information loss during the downsampling process. The first feature map A, the first feature map B, and the first feature map C obtained all contain image information from the three information streams, which can realize information supplementation.

[0119] For example, the image restoration layer can be specifically used to: for the first feature maps corresponding to the main stream and at least one sub-stream, starting from the first feature map corresponding to the information stream with the smallest feature map size, deconvolve the fused feature map of the current information stream to the feature map size of the next information stream in a progressively larger manner, and fuse the deconvoluted feature map with the first feature map corresponding to the next information stream, until the first feature map corresponding to the main stream is fused to obtain the first brightness-enhanced image; wherein, the fused feature map of the current information stream is obtained by deconvolve the first feature map of the previous information stream to the feature map size of the current information stream, and then fuse the deconvoluted feature map with the first feature map of the current information stream.

[0120] For example, with Figure 4 Taking the three obtained first feature maps as an example, in the image restoration layer, starting from the first feature map C, the first feature map C is deconvolved to the same image feature scale as the first feature map B, and the deconvolved feature map is fused with the first feature map B to obtain the fused feature map. Figure 1 Next, the fusion features will be... Figure 1 The image is deconvolved to the same feature scale as the first feature map A, and then fused with the deconvolved feature map to obtain the fused feature map. Figure 2 The fusion characteristics at this time Figure 2 This can then be output as the first brightness-enhanced image. In this way, by performing deconvolution operations on the first feature maps of the two streams, in a progressive manner, starting with the smallest stream, each deconvolution merges the current-sized stream or main stream feature map. After information interaction, the three information streams can be fused into the main information stream of the same size as the input. The resulting first brightness-enhanced image retains the main information stream features of the low-light image to be processed at the original image feature scale, preserving more details of the original image. Moreover, it integrates feature information from different scales, resulting in richer semantics.

[0121] based on Figure 1 or Figure 2 In one example embodiment of the image enhancement apparatus corresponding to the embodiments, the noise reduction module may include:

[0122] The second feature extraction unit is used to input the first brightness enhancement image into the second feature extraction layer of the noise reduction model to obtain the second initial feature map output by the second feature extraction layer; the second feature extraction layer is used to extract image features from the first brightness enhancement image;

[0123] The second splitting unit is used to input the second initial feature map into the second splitting layer of the denoising model to obtain the denoising main feature map and at least one denoising splitting feature map output by the second splitting layer; the second splitting layer is used to perform convolution on the second initial feature map at different image feature scales;

[0124] The second information interaction unit is used to input the denoising mainstream feature map and at least one denoising offshoot feature map into the second information interaction layer of the denoising model to obtain the third feature maps corresponding to the denoising mainstream and at least one denoising offshoot output by the second information interaction layer; the second information interaction layer is used to interactively fuse the third feature maps of the denoising mainstream and at least one denoising offshoot.

[0125] The noise extraction unit is used to input the third feature map into the noise extraction layer of the noise reduction model to obtain the noise information output by the noise extraction layer; the noise extraction layer is used to extract noise from the third feature map corresponding to the main noise reduction stream and at least one noise reduction sub-stream.

[0126] Based on this, the noise reduction module can adopt a network structure and working principle similar to that of the image enhancement module 120. It uses a multi-stream information interaction method to perform image feature extraction, stream splitting, multi-stream information interaction fusion, and noise extraction on the first brightness enhancement image output by the image enhancement module 120, obtaining the noise information of the first brightness enhancement image. Then, this noise information is removed from the first brightness enhancement image to obtain a second brightness enhancement image, which is the image after brightness enhancement and noise reduction processing of the low-noise image. This achieves noise reduction during the progressive image enhancement process, making the obtained second brightness enhancement image closer to a normally exposed image. The difference between the noise reduction model used by the noise reduction module and the brightness enhancement model is that the last layer of the noise reduction model does not obtain the denoised result, but rather the noise information.

[0127] The image enhancement apparatus based on the above embodiments, Figure 5 An exemplary schematic diagram illustrating the working principle of the image enhancement device provided in an embodiment of the present invention is shown below. Figure 5As shown, the low-light image to be processed is input into a brightness enhancement model. Image features are extracted through the first feature extraction layer of the brightness enhancement model to obtain a first initial feature map. This first initial feature map is then split into a main feature map and at least one branch feature map through a first splitting layer. These feature maps are then fused through information interaction in a first information interaction layer to obtain feature maps corresponding to each information stream. Finally, these feature maps are progressively deconvolutionally fused through an image restoration layer to restore an image to the same scale as the first initial feature map, resulting in the first brightness-enhanced image. Next, the first brightness-enhanced image is input into a denoising model. Image features are extracted through the second feature extraction layer of the denoising model to obtain a second initial feature map. This second initial feature map is then split into a main feature map and at least one branch feature map through a second splitting layer. These feature maps are then fused through information interaction in a second information interaction layer to obtain feature maps corresponding to each information stream. Finally, these feature maps are progressively deconvolutionally fused through a noise extraction layer to obtain the noise information corresponding to the first brightness-enhanced image. Then, by subtracting the noise information from the first brightness-enhanced image, the second brightness-enhanced image after brightness enhancement and noise reduction of the low-light image can be obtained.

[0128] The image enhancement device provided in this invention performs preliminary feature extraction on the low-light image to be processed using a brightness enhancement model. Then, it progressively constructs sub-information flow feature maps from the extracted feature maps. The size of all information flow feature maps is consistent with the input of the corresponding information flow, enabling the acquisition of more semantic information while preserving the original image information. Subsequently, after information interaction and fusion of multiple information flows, the image restoration unit restores the image to obtain a preliminarily enhanced first brightness-enhanced image. The first brightness-enhanced image is input into a denoising model, which extracts noise information. This noise information is then subtracted from the first brightness-enhanced image to obtain a second brightness-enhanced image after brightness enhancement and denoising. In this way, by progressively enhancing and denoising the low-light image to be processed, the low-light image can be restored to an image closer to normal exposure, while preserving most of the image's details and color information. This provides better data support for advanced vision tasks and improves the accuracy of subsequent advanced vision tasks. In addition, by training the brightness enhancement model using a combination of three loss functions—L1 loss function, structural similarity loss function, and color difference loss function—the color deviation generated during the enhancement process is taken into account, which can further avoid color distortion during the enhancement process and make the enhanced image closer to the normally exposed image.

[0129] The image enhancement method provided by the present invention is described below. The image enhancement method described below can be referred to in correspondence with the image enhancement device described above.

[0130] Figure 6An exemplary flowchart of the image enhancement method provided in an embodiment of the present invention is shown below. Figure 6 As shown, the image enhancement method may include the following steps 610 to 640.

[0131] Step 610: Obtain the low-light image to be processed.

[0132] Step 620: Input the low-light image into the brightness enhancement model to obtain the first brightness enhancement image output by the brightness enhancement model.

[0133] The brightness enhancement model is used to enhance the brightness of low-light images based on multi-stream feature interaction. The multi-stream features include the mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scale of the at least one branch feature map is different from that of the original image feature scale.

[0134] For example, the brightness enhancement model can be trained on an initial brightness enhancement model based on a low-light sample image, a first reference light-light sample image corresponding to the low-light sample image, and an enhancement loss function. The enhancement loss function is determined based on at least one of the pixel difference, structural similarity, and color difference between the low-light sample image and the first reference light-light sample image.

[0135] Step 630: Input the first brightness-enhanced image into the denoising model to obtain the noise information output by the denoising model.

[0136] The noise reduction model is used to extract noise information from the first brightness-enhanced image.

[0137] For example, the denoising model can be trained based on the following steps: acquiring a second reference illuminance sample image and adding noise to the second reference illuminance sample image to obtain a noisy sample image; training an initial denoising model based on the noisy sample image, the second reference illuminance sample image, and the denoising loss function to obtain the denoising model; wherein, the denoising loss function is determined based on a first structural similarity and a second structural similarity, the first structural similarity being the structural similarity between the image after removing the noise output by the initial denoising model from the noisy sample image and the second reference illuminance sample image, and the second structural similarity being the similarity between the image after adding the noise output by the initial denoising model to the second reference illuminance sample image and the noisy sample image.

[0138] Step 640: Remove noise information from the first brightness enhancement image to obtain the second brightness enhancement image of the low-light image.

[0139] The image enhancement method provided by this invention first performs brightness enhancement on the low-light image to be processed based on multi-flow feature interaction using a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model. Next, a noise reduction model extracts noise information from the first brightness-enhanced image. Then, the noise information is removed from the first brightness-enhanced image to obtain a second brightness-enhanced image of the low-light image, thereby suppressing noise generation while ensuring the image brightness enhancement effect. The multi-flow features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of these at least one branch feature map are different from those of the original image feature scale. Therefore, in the process of brightness enhancement of the low-light image based on multi-flow feature interaction, more semantic information can be obtained while preserving the original image information of the low-light image, retaining more details and color information of the low-light image, thereby suppressing noise generation and reducing the loss of color information while ensuring the image brightness enhancement effect.

[0140] based on Figure 6 In one example embodiment, the image enhancement method corresponding to the embodiment inputs a low-light image into a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model. This may include: inputting the low-light image into a first feature extraction layer of the brightness enhancement model to obtain a first initial feature map output by the first feature extraction layer, which is used to extract image features from the low-light image; inputting the first initial feature map into a first branching layer of the brightness enhancement model to obtain a mainstream feature map and at least one branching feature map output by the first branching layer, which is used to perform convolution on the first initial feature map at different image feature scales; inputting the mainstream feature map and each branching feature map into a first information interaction layer of the brightness enhancement model to obtain a first feature map corresponding to the mainstream and each branching feature map output by the first information interaction layer, which is used to perform interactive fusion on the mainstream feature map and each branching feature map; and inputting the first feature map into an image restoration layer of the brightness enhancement model to obtain a first brightness-enhanced image output by the image restoration layer, which is used to perform deconvolution fusion on the first feature maps corresponding to the mainstream and each branching feature map.

[0141] For example, the split feature map may include a first split feature map and a second split feature map; inputting the first initial feature map into the first split layer of the brightness enhancement model to obtain the mainstream feature map and at least one split feature map output by the first split layer may include: inputting the first initial feature map into a first sub-split layer of the first split layer to obtain a first split feature map output by the first sub-split layer, wherein the first sub-split layer is used to perform a first downsampling on the first initial feature map and to perform convolution on the first feature map obtained by the first downsampling at the image feature scale of the first initial feature map; inputting the first split feature map and the mainstream feature map into a second sub-split layer of the first split layer. The first sub-sub ...

[0142] For example, inputting the mainstream feature map and at least one sub-stream feature map into the first information interaction layer of the brightness enhancement model to obtain the first feature maps corresponding to the mainstream and at least one sub-stream output by the first information interaction layer may include: inputting the mainstream feature map and at least one sub-stream feature map into the sampling layer of the first information interaction layer to obtain the sampled feature maps corresponding to each information stream output by the sampling layer; the information stream includes the mainstream and at least one sub-stream; the sampling layer is used to sample the mainstream feature map and at least one sub-stream feature map to the feature map size of each information stream respectively; for each information stream, the sampled feature maps corresponding to each information stream are interactively fused to obtain the feature map corresponding to each information stream.

[0143] For example, the image restoration layer can be specifically used to: for the first feature maps corresponding to the main information stream and each branch information stream, starting from the first feature map corresponding to the information stream with the smallest feature map size, deconvolve the fused feature map of the current information stream to the feature map size of the next information stream in a progressively larger manner, and fuse the deconvoluted feature map with the first feature map corresponding to the next information stream, until the first feature map corresponding to the main information stream is fused to obtain the first brightness enhanced image.

[0144] The fusion feature map of the current information stream is obtained by deconvolving the first feature map of the previous information stream to the size of the current information stream, and then fusing the deconvolved feature map with the first feature map of the current information stream.

[0145] based on Figure 6 In an example embodiment, the image enhancement method corresponding to the embodiment inputs a first brightness-enhanced image into a denoising model to obtain noise information output by the denoising model. This may include: inputting the first brightness-enhanced image into a second feature extraction layer of the denoising model to obtain a second initial feature map output by the second feature extraction layer; the second feature extraction layer is used to extract image features from the first brightness-enhanced image; inputting the second initial feature map into a second splitting layer of the denoising model to obtain a denoising mainstream feature map and at least one denoising splitting feature map output by the second splitting layer; the second splitting layer is used to perform convolution on the second initial feature map at different image feature scales; inputting the denoising mainstream feature map and at least one denoising splitting feature map into a second information interaction layer of the denoising model to obtain a third feature map corresponding to the denoising mainstream and at least one denoising splitting, respectively, output by the second information interaction layer; the second information interaction layer is used to perform interactive fusion of the third feature maps of the denoising mainstream and at least one denoising splitting; inputting the third feature map into a noise extraction layer of the denoising model to obtain noise information output by the noise extraction layer; the noise extraction layer is used to extract noise from the third feature maps corresponding to the denoising mainstream and at least one denoising splitting, respectively.

[0146] Figure 7 An example is a schematic diagram of the structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 can communicate with each other through the communication bus 740. The processor 710 can call logic instructions in the memory 730 to execute the image enhancement methods provided in the above-described method embodiments. These methods may include, for example,: acquiring a low-light image to be processed; inputting the low-light image into a brightness enhancement model to obtain a first brightness enhancement image output by the brightness enhancement model; inputting the first brightness enhancement image into a denoising model to obtain noise information output by the denoising model; and removing noise information from the first brightness enhancement image to obtain a second brightness enhancement image of the low-light image. The brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction. The multi-stream features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of the at least one branch feature map are different from those of the original image feature scale. The denoising model is used to extract noise information from the first brightness enhancement image.

[0147] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image enhancement method provided in the above-described method embodiments. This method may include, for example,: acquiring a low-light image to be processed; inputting the low-light image into a brightness enhancement model to obtain a first brightness enhancement image output by the brightness enhancement model; inputting the first brightness enhancement image into a noise reduction model to obtain noise information output by the noise reduction model; removing noise information from the first brightness enhancement image to obtain a second brightness enhancement image of the low-light image; wherein, the brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction, the multi-stream features include a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map, the image feature scales of the at least one branch feature map being different and different from the original image feature scale; the noise reduction model is used to extract noise information from the first brightness enhancement image.

[0149] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image enhancement method provided in the above-described method embodiments. This method may include, for example,: acquiring a low-light image to be processed; inputting the low-light image into a brightness enhancement model to obtain a first brightness enhancement image output by the brightness enhancement model; inputting the first brightness enhancement image into a denoising model to obtain noise information output by the denoising model; removing the noise information from the first brightness enhancement image to obtain a second brightness enhancement image of the low-light image; wherein the brightness enhancement model is used to perform brightness enhancement on the low-light image based on multi-stream feature interaction, the multi-stream features including a mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map, the image feature scales of the at least one branch feature map being different and different from the original image feature scale; and the denoising model is used to extract noise information from the first brightness enhancement image.

[0150] For example, a computer-readable storage medium may include a non-transitory computer-readable storage medium.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image enhancement device, characterized in that, include: The image acquisition module is used to acquire low-light images to be processed; The image enhancement module is used to input the low-light image into the brightness enhancement model to obtain the first brightness enhancement image output by the brightness enhancement model; The brightness enhancement model is used to enhance the brightness of the low-light image based on multi-stream feature interaction. The multi-stream features include the mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of the at least one branch feature map are different from those of the original image feature scale. The noise reduction module is used to input the first brightness enhancement image into the noise reduction model, obtain the noise information output by the noise reduction model, remove the noise information from the first brightness enhancement image, and obtain the second brightness enhancement image of the low-light image. The noise reduction model is used to extract noise information from the first brightness-enhanced image; The image enhancement module includes: The first feature extraction unit is used to input the low-light image into the first feature extraction layer of the brightness enhancement model to obtain a first initial feature map output by the first feature extraction layer; the first feature extraction layer is used to extract image features from the low-light image. The first splitting unit is used to input the first initial feature map into the first splitting layer of the brightness enhancement model to obtain the mainstream feature map and the at least one splitting feature map output by the first splitting layer; the first splitting layer is used to perform convolution on the first initial feature map at different image feature scales; The first information interaction unit is used to input the mainstream feature map and the at least one sub-stream feature map into the first information interaction layer of the brightness enhancement model to obtain the first feature maps corresponding to the mainstream and at least one sub-stream respectively output by the first information interaction layer; the first information interaction layer is used to interactively fuse the mainstream feature map and the at least one sub-stream feature map. An image restoration unit is used to input the first feature map into the image restoration layer of the brightness enhancement model to obtain the first brightness enhancement image output by the image restoration layer; the image restoration layer is used to perform deconvolution fusion on the first feature maps corresponding to the main stream and the at least one branch stream.

2. The image enhancement device according to claim 1, characterized in that, The at least one shunt feature map includes a first shunt feature map and a second shunt feature map; The first splitting unit includes: The first splitting subunit is used to input the first initial feature map into the mainstream layer of the first splitting layer to obtain the mainstream feature map output by the mainstream layer; the mainstream layer is used to perform convolution on the first initial feature map at the image feature scale of the first initial feature map; The second splitting subunit is used to input the first initial feature map into the first sub-splitting layer of the first splitting layer to obtain the first splitting feature map output by the first sub-splitting layer; the first sub-splitting layer is used to perform a first downsampling on the first initial feature map and to perform convolution on the first feature map obtained by the first downsampling at the image feature scale of the first feature map. The third splitting subunit is used to input the first splitting feature map and the mainstream feature map into the second sub-splitting layer of the first splitting layer to obtain the second splitting feature map output by the second sub-splitting layer; the second sub-splitting layer is used to perform a second downsampling on the mainstream feature map, and to perform convolution between the second downsampled second feature map and the first splitting feature map on the image feature scale of the second feature map; wherein, the sampling scale of the first downsampling and the second downsampling are different.

3. The image enhancement device according to claim 1, characterized in that, The first information interaction unit includes: A sampling subunit is used to input the mainstream feature map and the at least one sub-stream feature map into the sampling layer of the first information interaction layer to obtain the sampling feature map corresponding to each information stream output by the sampling layer; the information stream includes the mainstream and the at least one sub-stream; the sampling layer is used to sample the mainstream feature map and the at least one sub-stream feature map to the feature map size of each information stream respectively; The fusion subunit is used to interactively fuse the sampled feature maps corresponding to each information stream to obtain a feature map corresponding to each information stream.

4. The image enhancement device according to claim 1, characterized in that, The image restoration layer is specifically used for: taking the first feature maps corresponding to the main stream and the at least one branch stream respectively, starting from the first feature map corresponding to the information stream with the smallest feature map size, deconvolve the fused feature map of the current information stream to the feature map size of the next information stream in a progressively larger manner, and fuse the deconvoluted feature map with the first feature map corresponding to the next information stream, until the first feature map corresponding to the main stream is fused to obtain the first brightness enhanced image; The fused feature map of the current information stream is obtained by deconvolving the first feature map of the previous information stream to the size of the feature map of the current information stream, and then fusing the deconvolved feature map with the first feature map of the current information stream.

5. The image enhancement apparatus according to any one of claims 1 to 4, characterized in that, The brightness enhancement model is obtained by training an initial brightness enhancement model based on a low-light sample image, a first reference light-light sample image corresponding to the low-light sample image, and an enhancement loss function. The enhancement loss function is determined based on at least one of the pixel difference, structural similarity, and color difference between the low-light sample image and the first reference light sample image.

6. The image enhancement apparatus according to any one of claims 1 to 4, characterized in that, The noise reduction module includes: The second feature extraction unit is used to input the first brightness enhancement image into the second feature extraction layer of the noise reduction model to obtain a second initial feature map output by the second feature extraction layer; the second feature extraction layer is used to extract image features from the first brightness enhancement image; The second splitting unit is used to input the second initial feature map into the second splitting layer of the denoising model to obtain the denoising main feature map and at least one denoising splitting feature map output by the second splitting layer; the second splitting layer is used to perform convolution on the second initial feature map at different image feature scales; The second information interaction unit is used to input the noise reduction mainstream feature map and the at least one noise reduction off-stream feature map into the second information interaction layer of the noise reduction model to obtain the third feature maps corresponding to the noise reduction mainstream and the at least one noise reduction off-stream output by the second information interaction layer; the second information interaction layer is used to interactively fuse the third feature maps of the noise reduction mainstream and the at least one noise reduction off-stream. The noise extraction unit is used to input the third feature map into the noise extraction layer of the noise reduction model to obtain the noise information output by the noise extraction layer; the noise extraction layer is used to extract noise from the third feature map corresponding to the main noise reduction stream and the at least one noise reduction sub-stream.

7. The image enhancement apparatus according to any one of claims 1 to 4, characterized in that, Also includes: The model training module is used to acquire a second reference illuminance sample image, add noise to the second reference illuminance sample image to obtain a noisy sample image, and train an initial denoising model based on the noisy sample image, the second reference illuminance sample image and the denoising loss function to obtain the denoising model. The noise reduction loss function is determined based on a first structural similarity and a second structural similarity. The first structural similarity is the structural similarity between the image of the noisy sample image after removing the noise output by the initial noise reduction model and the second reference illumination sample image. The second structural similarity is the similarity between the image of the second reference illumination sample image after adding the noise output by the initial noise reduction model and the noisy sample image.

8. An image enhancement method, characterized in that, include: Acquire the low-light image to be processed; The low-light image is input into the brightness enhancement model to obtain the first brightness enhancement image output by the brightness enhancement model. The brightness enhancement model is used to enhance the brightness of the low-light image based on multi-stream feature interaction. The multi-stream features include the mainstream feature map of the low-light image at the original image feature scale and at least one branch feature map determined based on the mainstream feature map. The image feature scales of the at least one branch feature map are different from those of the original image feature scale. The first brightness-enhanced image is input into the noise reduction model to obtain the noise information output by the noise reduction model; The noise reduction model is used to extract noise information from the first brightness-enhanced image; The noise information is removed from the first brightness-enhanced image to obtain a second brightness-enhanced image of the low-light image; The step of inputting the low-light image into a brightness enhancement model to obtain a first brightness-enhanced image output by the brightness enhancement model includes: The low-light image is input into the first feature extraction layer of the brightness enhancement model to obtain a first initial feature map output by the first feature extraction layer; the first feature extraction layer is used to extract image features from the low-light image. The first initial feature map is input into the first split layer of the brightness enhancement model to obtain the mainstream feature map and the at least one split feature map output by the first split layer; the first split layer is used to perform convolution on the first initial feature map at different image feature scales; The mainstream feature map and the at least one sub-stream feature map are input into the first information interaction layer of the brightness enhancement model to obtain the first feature maps corresponding to the mainstream and at least one sub-stream respectively output by the first information interaction layer; the first information interaction layer is used to interactively fuse the mainstream feature map and the at least one sub-stream feature map. The first feature map is input into the image restoration layer of the brightness enhancement model to obtain the first brightness enhancement image output by the image restoration layer; the image restoration layer is used to perform deconvolution fusion on the first feature maps corresponding to the main stream and the at least one branch stream.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image enhancement method as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image enhancement method as described in claim 8.

Images