Low-illumination image enhancement method and device and electronic equipment

Abstract

The invention provides a low-illumination image enhancement method and device and electronic equipment, and the method comprises the steps: carrying out the light segmentation of a low-illumination image, and obtaining a first image corresponding to a dark light region and a second image corresponding to a bright light region; performing wavelet transformation on the first image to obtain a third image; and processing the third image according to a pre-trained image enhancement model to obtain a fourth image, the image enhancement model being obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and being used for selecting at least part of the sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, the sub-models for enhancing different areas are different; performing wavelet inverse transformation on the fourth image to obtain a fifth image; and synthesizing the fifth image and the second image, and performing post-processing on the synthesized image to obtain an enhanced image. The image enhancement model provided by the embodiment of the invention integrates the advantages of a plurality of sub-models, the enhancement effect is better, and the generalization ability is higher.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a low-illumination image enhancement method, device and electronic equipment. Background Art

[0002] Images captured at night in low-light conditions are a great challenge for computer vision applications. Limited by lighting conditions and the influence of outdoor weather, the visual quality of low-light images is often not ideal and may contain various degradation factors, such as low contrast, low visibility, and high noise. The superposition of these factors will have a serious negative impact on downstream visual tasks (such as object detection and visual tracking), and may even cause visual systems to malfunction. In order to improve the visual quality of low-light images, image enhancement is required.

[0003] Low-Light Image Enhancement (LLIE) aims to restore normal illumination from images with low light and severe noise, and effectively suppress imaging noise. Currently, there are many algorithm models that can achieve low-light image enhancement. However, these algorithm models have different advantages and disadvantages, which makes it complicated to choose a suitable algorithm model. In addition, these algorithm models usually perform well on specific data sets, but may have insufficient generalization capabilities when faced with various lighting conditions and scene changes. Summary of the invention

[0004] The present application provides a low-illumination image enhancement method, device and electronic device, which are used to solve the problem that it is difficult to select a suitable algorithm model and the generalization ability of the algorithm model is insufficient when enhancing the low-illumination image.

[0005] To solve the above technical problems, this application is implemented as follows: In a first aspect, a low-illumination image enhancement method is provided, comprising: Performing light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; Performing wavelet transform on the first image to obtain a third image; Processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; Performing an inverse wavelet transform on the fourth image to obtain a fifth image; The fifth image and the second image are synthesized, and the synthesized image is post-processed to obtain an enhanced image corresponding to the low-illuminance image.

[0006] In a second aspect, a low-light image enhancement device is provided, comprising: An image segmentation module performs light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; A wavelet transform module, performing wavelet transform on the first image to obtain a third image; an image enhancement module, processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; A wavelet inverse transformation module, performing a wavelet inverse transformation on the fourth image to obtain a fifth image; A post-processing module synthesizes the fifth image and the second image, and performs post-processing on the synthesized image to obtain an enhanced image corresponding to the low-illuminance image.

[0007] In a third aspect, an electronic device is provided, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as described in the first aspect.

[0008] According to a fourth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method described in the first aspect.

[0009] According to a fifth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the method described in the first aspect.

[0010] In the embodiment of the present application, when the low-illuminance image is enhanced, the image enhancement model used is a model obtained by fusing multiple sub-models for image enhancement based on the expert model. Therefore, the advantages of each sub-model can be combined, so that the image enhancement result has the advantages of multiple sub-models at the same time, thereby improving the image enhancement effect; when performing image enhancement processing, a suitable sub-model can be dynamically and flexibly selected from multiple sub-models for enhancement processing according to the characteristics of the image area, and different image areas can select different sub-models for image enhancement. Therefore, it is possible to effectively capture the local features of the image and enhance the quality of each image area in a targeted manner. This can improve the visual effect of the overall image and effectively deal with various influencing factors in low-light images, so that it can be better generalized to other low-light images, thereby improving the robustness, stability and generalization ability of the model; since only part of the multiple sub-models are used to enhance the same image area, instead of using all the sub-models, the calculation speed can be increased and resource usage can be reduced; since a variety of processing techniques can be used to pre-process (i.e., light segmentation and wavelet transform) and post-process (i.e., inverse wavelet transform and post-processing of the synthesized image) the low-light image before and after image enhancement, the image quality can be improved and the image enhancement effect can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 is a flowchart of a low-light image enhancement method according to an embodiment of the present application; Figure 2 is a schematic diagram of image registration according to an embodiment of the present application; Figure 3 is a schematic diagram of image registration according to another embodiment of the present application; Figure 4 is a schematic diagram of an embodiment of the present application performing image enhancement processing according to an image enhancement model; Figure 5 is a schematic diagram of synthesizing an enhanced image of a normal area and an image of a bright area in one embodiment of the present application; Figure 6 is a schematic diagram of post-processing an image according to an embodiment of the present application; Figure 7is a schematic diagram of a process of low-light image enhancement according to an embodiment of the present application; Figure 8 is a schematic diagram of an embodiment of the present application for enhancing a low-light image; Fig. 9 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application; Fig.10 It is a schematic diagram of the structure of a low-illumination image enhancement device according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] In related technologies, when enhancing low-light images, some LLLE methods based on deep learning have achieved good enhancement effects on public datasets such as LOL_v1. However, in practical applications, these methods face at least the following three challenges: (1) Algorithm selection and optimization.

[0014] Although there are many excellent LLIE algorithms available, each of them has different advantages and disadvantages. This makes it complicated to choose the right algorithm in practical applications. In order to obtain the best enhancement effect, it is necessary to comprehensively evaluate the characteristics of different algorithms and learn from each other according to specific application scenarios. This requires that in the process of algorithm selection and optimization, not only the enhancement effect should be considered, but also the demand for computing resources and the feasibility of actual deployment should be evaluated.

[0015] (2) Generalization ability and adaptability.

[0016] Existing methods usually perform well on specific datasets, but in practical applications, they may have insufficient generalization capabilities when faced with various lighting conditions and scene changes. The image quality of public datasets is relatively high, while the actual dark light images to be enhanced (i.e., low-light images) may have problems such as angle distortion, insufficient training sets, light interference, severe noise, and images that are too large. These factors may significantly reduce the effectiveness of existing LLIE methods in practical applications. Therefore, how to improve the adaptability of the algorithm to different environments to ensure stable performance in various practical application scenarios is a challenge that needs to be solved urgently.

[0017] (3) Computational efficiency and real-time performance.

[0018] Deep learning models usually require a lot of computing resources, which makes computing efficiency an important constraint in real-time applications. In order to ensure that images can be quickly processed and enhanced in real time in practical applications, it is necessary to optimize computing efficiency while ensuring the image enhancement effect. This involves improving model structure, reducing computational complexity, and using more efficient hardware and algorithm optimization strategies to meet the challenges of real-time processing. Improving computing efficiency can not only speed up processing, but also improve user experience and system practicality.

[0019] An embodiment of the present application provides a low-illumination image enhancement method, device and electronic device, the method comprising: performing light segmentation on a low-illumination image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-illumination image; performing wavelet transform on the first image to obtain a third image; processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing multiple sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least part of the sub-models from the multiple sub-models to perform enhancement processing on different areas of the third image, and different sub-models are used to perform enhancement processing on different areas; performing inverse wavelet transform on the fourth image to obtain a fifth image; synthesizing the fifth image and the second image, and post-processing the synthesized image to obtain an enhanced image corresponding to the low-illumination image.

[0020] When the technical solution provided in the embodiment of the present application performs enhancement processing on low-illuminance images, since the image enhancement model used is a model obtained by fusing multiple sub-models for image enhancement based on the expert model, the advantages of each sub-model can be combined, so that the image enhancement result has the advantages of multiple sub-models at the same time, thereby improving the image enhancement effect; when performing image enhancement processing, a suitable sub-model can be dynamically and flexibly selected from multiple sub-models for enhancement processing according to the characteristics of the image area, and different image areas can select different sub-models for image enhancement, therefore, it is possible to effectively capture the local features of the image and specifically enhance the quality of each image area. The method can improve the visual effect of the overall image and effectively deal with various influencing factors in low-light images, so that it can be better generalized to other low-light images and improve the robustness, stability and generalization ability of the model. Since only part of the multiple sub-models are used to enhance the same image area, instead of using all the sub-models, the calculation speed can be increased and the resource usage can be reduced. Before and after image enhancement, a variety of processing techniques can be used to pre-process (i.e., light segmentation and wavelet transform) and post-process (i.e., inverse wavelet transform and post-processing of the synthetic image) the low-light image, therefore, the image quality can be improved and the image enhancement effect can be further improved.

[0021] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be described clearly and completely below in conjunction with the drawings in one or more embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0022] The terms "first", "second", etc. in the present application and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate so that the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the present application and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0023] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0024] Figure 1 It is a flowchart of a low-illumination image enhancement method according to an embodiment of the present application. Figure 1 The low illumination image enhancement method shown is described as follows.

[0025] S102: performing light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image.

[0026] Low-light images are usually taken at night with street lights. The ambient light is weak at night, while the street lights are concentrated. The street lights will directly enter the camera lens, causing the local area to be too bright, exceeding the dynamic range of the camera, thus losing details and affecting the image appearance, that is, overexposure. Since overexposure not only affects the visual effect and image quality, but also affects the image enhancement effect, before performing image enhancement on the low-light image to be processed, it is necessary to detect and segment the overexposed area caused by street lights in the low-light image.

[0027] It should be noted that the overexposed area can also be represented as a bright area or a light area, and the area other than the overexposed area in the low-light image can be represented as a non-overexposed area, a dark area, a non-light area, or a normal area. For the sake of uniform description, the bright area and the dark area will be used as examples for explanation below.

[0028] In the embodiment of the present application, when detecting and segmenting the bright light area caused by street lamps in a low-light image, it can be achieved by performing light segmentation on the low-light image. When performing light segmentation on the low-light image, the dark light area and the bright light area caused by the street lamp light source can be identified by pixel brightness analysis, and finally a first image corresponding to the dark light area of ​​the low-light image and a second image corresponding to the bright light area can be obtained.

[0029] In some embodiments, performing light segmentation on a low-light image to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image may include: Determine the brightness of each pixel in the low-light image; Determine bright pixels and dark pixels in the low-light image according to the brightness of each pixel in the low-light image and a preset brightness threshold; Binarize the low-light image according to the bright-light pixels and the dark-light pixels to generate a binary image; A first image corresponding to a dark area and a second image corresponding to a bright area in the low-illuminance image are determined according to the binary image.

[0030] When determining the brightness of each pixel in a low-light image, the RGB value or grayscale value of each pixel in the low-light image can be analyzed, and the RGB value or grayscale value can reflect the brightness of the pixel. After obtaining the brightness of each pixel, the brightness of each pixel can be compared with a preset brightness threshold. For any pixel in a low-light image, if the brightness of the pixel is greater than or equal to the preset brightness threshold, the pixel can be determined to be a bright pixel (that is, an overexposed pixel); if the brightness of the pixel is less than the preset brightness threshold, the pixel can be determined to be a dark pixel (that is, a non-overexposed pixel or a normal pixel). After analyzing each pixel in the low-light image, all bright pixels and dark pixels in the low-light image can be determined. Among them, the brightness threshold can be set according to the characteristics of the night image, and the specific size of the brightness threshold is not limited here.

[0031] After determining the bright pixels and dark pixels in the low-light image, the low-light image can be binarized, that is, the brightness pixels and the dark pixels are separated to generate a binary image, in which the white area represents the bright area and the black area represents the dark area. After the binary image is generated, the first image corresponding to the dark area and the second image corresponding to the bright area in the low-light image can be finally obtained according to the binary image.

[0032] S104: Perform wavelet transform on the first image to obtain a third image.

[0033] After obtaining the first image corresponding to the dark area of ​​the low-light image and the second image corresponding to the bright area, in the subsequent image enhancement processing, the first image can be enhanced and the second image is not enhanced. In this way, the local overexposure effect caused by street lights can be reduced, while retaining the details and beauty of the image, and improving the overall image quality and processing effect.

[0034] Since low-light images usually have more significant noise and lack clear features, and the scale of the real image to be enhanced is generally too large, when enhancing the first image, the first image can be firstly subjected to wavelet transform to reduce the image scale, optimize image quality and suppress noise.

[0035] Wavelet transform is a signal processing technique that reveals the multi-scale characteristics of a signal by decomposing it into sub-signals of different frequencies. In image processing, wavelet transform is widely used in tasks such as image compression, denoising and feature extraction. It can effectively capture the edges and details of the image, thereby providing efficient image representation and processing capabilities. Daubechies (db, Daubechies) wavelet transform is known for its compact support and efficient denoising capabilities. The db wavelet has a short support length and high smoothness, which enables it to better retain important features when processing signals while suppressing noise, thereby better handling the various noises and details that may exist in low-light images.

[0036] When performing wavelet transform on the first image, optionally, the first image can be processed using an improved Daubechies wavelet transform (db4 or db8), which can decompose a large-size image into sub-images of multiple scales, and can retain low-frequency components with clear contours when scaling the image, while reducing the scale, effectively reducing the occupation of computing resources. The Daubechies wavelet transform can effectively capture edge and detail information in an image, especially in low-light conditions, which is crucial for restoring the visibility of the image.

[0037] It should be noted that after performing wavelet transform on the first image, images of multiple scales can be obtained. When performing image enhancement, the low-frequency part thereof can be enhanced, that is, the third image mentioned above can be a low-frequency image obtained after wavelet transform, and images of other scales can be saved so that the original image can be restored when performing inverse wavelet transform later, so as to reconstruct and optimize the image quality and ensure the details of the restored image.

[0038] S106: Process the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing multiple sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the multiple sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions.

[0039] The multiple sub-models can be the models with excellent image enhancement effects at present. Optionally, the multiple sub-models can have different LLIE algorithms, each with its own advantages and disadvantages. In this way, when the image enhancement model obtained by fusing these sub-models is used for image enhancement processing, the advantages of each sub-model can be combined, so that the image enhancement result has the advantages of multiple sub-models at the same time, thereby improving the image enhancement effect.

[0040] When the third image is processed according to the pre-trained image enhancement model, the image enhancement model can select at least part of the sub-models from the multiple sub-models for image enhancement for different areas of the third image, and the sub-models selected for different image areas can be different. In this way, since the appropriate sub-model can be dynamically and flexibly selected from the multiple sub-models for enhancement processing according to the characteristics of the image area, and different image areas can select different sub-models for image enhancement, it is not only possible to effectively capture the local features of the image and enhance the quality of each image area in a targeted manner, thereby improving the visual effect of the overall image, but also to effectively deal with various influencing factors in low-light images, so that it can be better generalized to other low-light images, and improve the robustness, stability and generalization ability of the model. In addition, since only part of the sub-models in the multiple sub-models are used to enhance the same image area, not all the sub-models are used, the calculation speed can be increased and the resource usage can be reduced.

[0041] Before using the image enhancement model to enhance the third image, the image enhancement model needs to be trained first. In some implementations, the image enhancement model can be trained in the following manner: Step 1: Obtain multiple high and low illumination image pairs, a standard training set, and multiple sub-models.

[0042] Multiple high- and low-illumination image pairs can be obtained by manually pairing images captured in real scenes. A high- and low-illumination image pair includes at least one high-illumination image (an image captured in a daytime environment) and one low-illumination image (an image captured in a night scene).

[0043] The standard training set can be a currently available dataset with a large amount of data, such as the LOL_v1 dataset, which is famous for its high image quality and lack of light interference.

[0044] The multiple sub-models are the multiple sub-models included in the image enhancement model. When obtaining multiple sub-models, they can be comprehensively selected from the low-light image enhancement (LLIE) model with excellent current effects. In order to combine the advantages of different models, a variety of sub-models can be selected. These sub-models have their own characteristics in structure and training method, and can complement each other's shortcomings, thereby achieving a more comprehensive enhancement effect.

[0045] In some implementations, the obtained multiple sub-models and their basic information may be shown in the following Table 1: Table 1

[0046] In Table 1, DiffLLE and PyDIff use diffusion models to improve the processing speed and detail recovery of low-light images while maintaining high perceptual fidelity. The snr-skf method improves image quality through semantic embedding, color histogram loss, and adversarial loss to ensure color consistency and natural details. Retinexformer is based on retinal theory and uses self-attention networks to capture long-range dependencies between image parts. Self-DACE is an intuitive, lightweight, fast, and unsupervised two-stage brightness enhancement algorithm that includes denoising. URetinex-Net is based on the deep unfolding network of Retinex. By decomposing low-light images into reflection layers and illumination layers, it combines learning modules and data-driven optimization to achieve noise suppression and detail preservation. These models cover both supervised and unsupervised learning paradigms, and use a variety of methods including retinal models, diffusion models, and self-attention models.

[0047] It should be noted that the embodiment of the present application takes the 7 models shown in Table 1 as an example for the selected sub-models. In other possible implementations, other models with excellent enhancement effects for image enhancement may also be selected, which are not specifically limited here. In addition, in order to reduce the amount of computation and video memory usage of image enhancement processing, smaller models may be given priority when selecting sub-models.

[0048] Step 2: Perform image registration on multiple high and low illumination image pairs to obtain the first training set.

[0049] Considering that the high-illuminance image and low-illuminance image in the image pair are taken during the day and night respectively, not at the same time, the perspective of the image pair may change during this period due to camera operation and other reasons, and the difference in perspective will make the pixel-based learning method invalid. Therefore, after obtaining multiple high- and low-illuminance image pairs, these high- and low-illuminance image pairs need to be registered.

[0050] In some implementations, when performing image registration on a plurality of high- and low-illumination image pairs, for any of the high- and low-illumination image pairs, the following may be included: Mask the text information in the high and low illumination image pairs; A feature matching method based on deep learning is used to process high and low illumination image pairs; The processed image is projected and transformed to obtain the registered high and low illumination image pairs.

[0051] Since the text information in the image (such as date, etc.) will interfere with the image registration and affect the registration effect, the text information in the high and low illumination image pairs can be blocked before image registration. After blocking the text information, the high and low illumination image pairs can be processed based on the feature matching method of deep learning. Specifically, a standard image (usually an image taken during the day) can be given first, and then some key points can be selected on the standard image and the image to be registered respectively, and then the key points and the features around the key points can be used to match the key points. Optionally, the feature matching method based on deep learning can be LightGlue, which is a local feature matching method based on deep learning. This method is more efficient in terms of memory and computing, and has the ability to adapt to the complexity when registering different image pairs. The reasoning speed is faster for image pairs with little change in perspective.

[0052] After processing the high and low illumination image pairs using the deep learning-based feature matching method, the processed images can be projected. Projection transformation can also be called homography transformation. Projection transformation of images based on the correct homography matrix can achieve the transformation of viewing angle, that is, image registration. In the specific implementation, at least four pairs of matching feature points are required to obtain a unique homography transformation matrix. Because the key points may be mismatched, in fact, dozens or hundreds of matching key points will be found in the previous step, and then the homography matrix is ​​obtained according to cv2.findHomography in the OpenCV function library. Finally, the image can be changed to achieve registration of the image pair.

[0053] To Figure 2 and Figure 3 Take the image shown in the figure as an example. Figure 2 When the images to be registered are registered, Figure 2 The standard image shown is the reference image, and we can get Figure 2 The image after registration is shown in Figure 2. Figure 3 When the images to be registered are registered, Figure 3 The standard image shown is the reference image, and we can get Figure 3 The registered images are shown.

[0054] After performing image registration on a plurality of high and low illumination image pairs according to the method described above, a plurality of registered high and low illumination image pairs can be obtained, and these image pairs can constitute a first training set.

[0055] Step 3: Perform independent pre-training on multiple sub-models based on the standard training set and determine the pre-training weights.

[0056] Since the number of images in the first training set is small, when pre-training multiple sub-models, the pre-training can be performed based on the standard training set, that is, each sub-model is pre-trained independently on the standard training set. After pre-training multiple sub-models independently, the basic feature extraction capability of each sub-model and the pre-training weight of each sub-model can be obtained. The pre-training weight of the sub-model can characterize the parameters used by the neural network of the sub-model, and these parameters affect the image enhancement effect of the sub-model itself.

[0057] Step 4: Based on the expert model, multiple sub-models are integrated to obtain a fusion model.

[0058] After completing the pre-training of multiple sub-models, these pre-trained sub-models can be fused, that is, these pre-trained sub-models can be fused (or integrated) into an overall network structure.

[0059] The model fusion method combines the prediction results of multiple different models and integrates the advantages of each model to improve the overall performance. Specifically, it uses a variety of model selection and weighting mechanisms to more effectively process different data types and complex scenes, and improve the stability and reliability of the enhancement effect. The key lies in how to effectively integrate and optimize the predictions of multiple models to achieve higher accuracy and generalization. Expert model (Mixture of Experts, MoE) is a technology for model fusion, which generates a final prediction by assigning input data to multiple expert models and weighting these outputs according to data characteristics. Each expert performs well on a specific subset of data. This specialization takes advantage of the advantages of expert models and improves the accuracy and generalization of the system. The core of MoE is the "gating mechanism", which selects the expert model most relevant to the input, optimizes the use of computing resources and improves the prediction effect. In the embodiment of the present application, when multiple sub-models are fused, multiple sub-models can be fused based on the expert model, that is, the expert model is innovatively applied to the enhancement processing task of small and medium-sized low-light images.

[0060] Step 5: According to the pre-trained weights and fusion weights, fine-tune the fusion model on the first training set to obtain the image enhancement model.

[0061] After obtaining the fusion model, the fusion model can be further trained. When training the fusion model, the fusion model can be trained on the first training set according to the pre-trained weights of multiple sub-models and the fusion weights of multiple sub-models, that is, the fusion model is fine-tuned on the first training set. In the process of training the fusion model, the fusion weights between the sub-models can be dynamically adjusted, and the overall performance of the network can be optimized by adjusting the fusion weights of the sub-models, thereby achieving comprehensive optimization and performance improvement of the final image enhancement model. Optionally, in order to ensure the performance of the fusion model on the first training set, the fine-tuning process of the fusion model will be carried out for 1000 epochs (batches), which helps to adapt to the characteristics of the first training set.

[0062] Table 2 shows the comparison of the image enhancement model of the embodiment of the present application with other models on the LOL_v1 dataset in terms of the three indicators of PSNR, SSIM and LPIPS: Table 2

[0063] After the image enhancement model is trained based on the above method, when performing image enhancement on a low-illuminance image, after segmenting to obtain a first image corresponding to the dark light area and performing wavelet transform on the first image to obtain a third image, the third image can be enhanced according to the trained image enhancement model to obtain a fourth image. The fourth image is the enhanced image corresponding to the third image.

[0064] In some implementations, processing the third image according to the pre-trained image enhancement model to obtain the fourth image may include the following steps 1 to 4: Step 1: Divide the third image into a plurality of image blocks.

[0065] Here, dividing the third image into a plurality of image blocks can be regarded as dividing the third image into a plurality of different regions, each region corresponding to an image block. The specific number of the plurality of image blocks can be determined according to actual needs, for example, it can be divided into 50*50 image blocks, which is not specifically limited here.

[0066] Step 2: For each image block, the embedding result of the image block is input into the image enhancement model to obtain the target sub-model corresponding to the image block.

[0067] After the third image is divided into multiple image blocks, when the multiple image blocks are enhanced, the enhancement processing method of each image block is the same. Therefore, for the convenience of description, the embodiment of the present application takes one of the image blocks as an example to illustrate how to enhance the image block.

[0068] When performing enhancement processing on an image block, the embedding result of the image block can be determined first. Specifically, the image block can be expanded from two dimensions to one dimension, and then multiplied by a matrix to obtain a new vector, which can be used as a representation of the image block, that is, the embedding result of the image block.

[0069] After the embedding result of the image block is obtained, the embedding result can be input into the image enhancement model to obtain a target sub-model corresponding to the image block, and the target sub-model can include at least part of the sub-models in the multiple sub-models, and the target sub-model is used to perform image enhancement on the image block. That is, after the embedding result of the image block is input into the image enhancement model, the image enhancement model can select a suitable sub-model from the multiple sub-models as the target sub-model of the image block according to the image features of the image block. Among them, for different image blocks, since the image features of different image blocks are different, the corresponding target sub-models are also different.

[0070] In some implementations, inputting the embedding result of the image block into the image enhancement model to obtain the target sub-model corresponding to the image block may include: Inputting the embedding result of the image block into the image enhancement model to obtain a first vector, a second vector and a third vector; Multiply the first vector and the second vector to obtain a correlation coefficient matrix between the multiple sub-models; Multiplying the correlation coefficient matrix by the third vector to obtain target weight values ​​of multiple sub-models; According to the target weight values ​​of the multiple sub-models, at least some of the sub-models are selected from the multiple sub-models as the target sub-models corresponding to the image block.

[0071] The first vector can be represented as a query vector Q, the second vector can be represented as a key vector K, and the third vector can be represented as a value vector V. Among them, the first vector and the second vector can be used to calculate the correlation between multiple sub-models, and the third vector is used to characterize the initial weight values ​​of multiple sub-models. After obtaining these three vectors, the initial weight values ​​of multiple sub-models can be adjusted according to these three vectors to obtain the final target weight value, which can be used as the basis for selecting the sub-model. Specifically, the first vector and the second vector can be multiplied to obtain a coefficient matrix, which can characterize the correlation between multiple sub-models and is a correlation coefficient matrix. After that, the correlation coefficient matrix can be multiplied by the third vector to obtain the target weight values ​​of multiple sub-models. In this way, the weight of the sub-model with higher similarity to other sub-models will be relatively increased, while the weight of the sub-model with lower similarity will decrease. Compared with other models and general MoE methods, this method can make the selection of sub-models not only consider their own performance, but also comprehensively consider the correlation with other sub-models, thereby more effectively adjusting the weight of each sub-model.

[0072] After the target weight values ​​of the multiple target sub-models are obtained, the sub-model with the highest weight can be selected from the multiple sub-models as the target sub-model corresponding to the image block according to the target weight values.

[0073] Step 3: Enhance the image block according to the target sub-model to obtain the image enhancement result of the image block.

[0074] After a target sub-model of an image block is selected from multiple sub-models, the image block can be enhanced according to the target sub-model, and finally an image enhancement result of the image block can be obtained.

[0075] In some implementations, performing enhancement processing on the image block according to the target sub-model to obtain an image enhancement result of the image block may include: Performing image enhancement on the image block according to the sub-model in the target sub-model to obtain multiple image enhancement results; Perform channel fusion on multiple image enhancement results to obtain an image enhancement fusion result; The image enhancement fusion result is processed into blocks to obtain multiple image sub-blocks; Upsampling the multiple image sub-blocks to obtain a fourth vector, where the fourth vector is used to characterize the relationship between image enhancement results of different sub-models corresponding to different image sub-blocks; Performing self-attention transformation on the fourth vector to obtain weighted weight values ​​between different sub-models corresponding to different image sub-blocks; The multiple image sub-blocks are weightedly fused according to the weighted weight values ​​to obtain the image enhancement result of the image block.

[0076] Specifically, after obtaining the target sub-model corresponding to the image block, the image block can be enhanced according to each sub-model in the target sub-model to obtain multiple image enhancement results, and one sub-model in the target sub-model can correspond to one image enhancement result. After obtaining multiple image enhancement results, these image enhancement results can be channel fused to obtain image enhancement fusion results. For example, assuming that the target sub-model of the image block includes m sub-models, then when performing image enhancement, the m sub-models can be used to perform enhancement processing on the image block respectively, and m image enhancement results are obtained. Then, these m image enhancement results are combined in the channel dimension. For example, the image block is 50*50*3 before enhancement, and after enhancement, m small blocks of 50*50*3 will be obtained, which are combined in the channel dimension to be 50*50*(3*m). The image block of 50*50*(3*m) is the image enhancement fusion result.

[0077] After obtaining the image enhancement fusion result, the image enhancement fusion result can be divided into blocks (also expressed as secondary block processing) to obtain multiple image sub-blocks. For example, after obtaining the above 50*50*(3*m) image enhancement fusion result, secondary block processing can be performed on this basis, for example, 50*50 is divided into 25*25 sizes, and the multiple image sub-blocks finally obtained are 25*25*(4*3*m).

[0078] After obtaining multiple image sub-blocks, the multiple image sub-blocks can be upsampled to generate a fourth vector. The length of the fourth vector can be equal to the number of image sub-blocks, and the fourth vector contains the relationship information between the image enhancement results of different sub-models in different image sub-blocks, that is, the image block uses multiple sub-models for enhancement processing, and the fourth vector can represent the relationship information between the image enhancement results obtained after these sub-models enhance the image block, that is, the fourth vector can represent the relationship between the image enhancement results of different sub-models corresponding to different image sub-blocks. Afterwards, the fourth vector can be self-attention transformed to obtain the weights between the sub-models and between different blocks, that is, to obtain the weighted weight values ​​between the different sub-models corresponding to different image sub-blocks, and the multiple image sub-blocks are weightedly fused according to the weighted weight values ​​to obtain the image enhancement results of the image block.

[0079] Taking the above-mentioned multiple image sub-blocks 25*25*(4*3*m) as an example, after upsampling these image sub-blocks, they can become 1*1*4*3*m, that is, a one-dimensional vector of length 4*3*m (that is, the fourth vector mentioned above), and then after performing self-attention transformation on them, another one-dimensional vector of length 4*3*m can be obtained, which is the preliminary weight. This vector is the weight of the m sub-models on the four small blocks. Next, normalization is performed. Here, the entire vector is not normalized, but the weights of the m sub-models on each small block are normalized. Then, the normalized weights are multiplied with the input of 25*25*(4*3*m), and then the m channels corresponding to each small block are added, which completes the weighted fusion process. The fused image obtained at this time is the image enhancement result corresponding to the image block, and its size is 25*25*(4*3), that is, 50*50*3, which is the same as the size of the image block before image enhancement, 50*50*3.

[0080] Step 4: Combine and smooth the image enhancement results of multiple image blocks to obtain a fourth image.

[0081] After each of the multiple image blocks is enhanced based on step 2 and step 3, multiple image enhancement results corresponding to the multiple image blocks can be finally obtained, and one image block corresponds to one image enhancement result. Afterwards, the multiple image enhancement results corresponding to the multiple image blocks can be combined to restore the original shape of the third image, and then the combined result can be smoothed to obtain the fourth image.

[0082] Based on the above steps, it can be seen that the model fusion enhancement idea of ​​the embodiment of the present application is: first divide the image to be enhanced into multiple image blocks, input each image block into the image enhancement model, and obtain the m sub-models (m is greater than or equal to 2 and less than or equal to the total number of sub-models) that are most suitable for weighting on the image block. These sub-models are then weighted for attention on the image block, and finally the enhanced image block can be obtained. Finally, the enhancement results of all the divided image blocks are merged, and then smoothed to obtain the enhancement result of the image to be enhanced.

[0083] In order to understand the image enhancement model's image enhancement process, please refer to Figure 4 .

[0084] Figure 4 In the process of enhancing the original image, the original image can be divided into blocks to obtain multiple image blocks ( Figure 4 Only one image block is shown). Then, the embedding result of the image block is input into the image enhancement model, which generates three vectors Q, K and V. Q is the query vector, K is the key vector, Q and K are used to calculate the correlation between experts (sub-models), and V is the value vector, which represents the initial weight of each expert (sub-model). Based on the three vectors Q, K and V, the final weight of each expert (sub-model) can be calculated. Based on the final weight, some sub-models can be selected from the N sub-models to perform image enhancement on the image block. Figure 4 As shown in the figure, assuming that M (M is less than or equal to N) sub-models are selected to enhance the image block, M image enhancement results can be obtained. After that, the M image enhancement results can be channel fused, and the fusion results can be divided into multiple small blocks twice to obtain multiple small blocks. After upsampling and self-attention transformation of the multiple small blocks, the weighted image blocks can be obtained, that is, the enhancement results of the image blocks. After combining and smoothing the enhancement results of the multiple image blocks, a weighted image can be obtained, which is the enhanced image obtained after the original image is enhanced.

[0085] S108: Perform inverse wavelet transform on the fourth image to obtain a fifth image.

[0086] After the third image is enhanced based on S106 to obtain the fourth image, the fourth image may be subjected to an inverse wavelet transform (also referred to as an inverse wavelet transform) to obtain a fifth image to restore the spatial information of the image. The fifth image is an image obtained after the first image is enhanced and has the same size as the first image.

[0087] S110: synthesize the fifth image and the second image, and post-process the synthesized image to obtain an enhanced image corresponding to the low-illuminance image.

[0088] After obtaining the fifth image, since the low-light image is segmented by light in S102, in order to restore the original image, the fifth image and the second image may be synthesized, that is, the enhancement result of the dark light area in the low-light image and the bright light area are synthesized. Figure 5 shown. Figure 5 In the figure, (a) is the original dark light image, i.e., the low-light image to be processed. After performing light segmentation on the image, the normal area image shown in (b) can be obtained. After processing the normal area image based on the above S104 and S108, the enhanced image of the normal area shown in (c) can be obtained. After that, (c) can be compared with the overexposed area image ( Figure 5 The final result image shown in (d) can be obtained by synthesizing the two images.

[0089] It should be noted that in some possible implementations, when synthesizing the enhanced image of the dark light area with the bright light area, the pixel brightness of the dark light area can be detected, and if the pixel brightness is higher than a set threshold, synthesis may not be performed. This strategy is intended to reduce the unnatural black border phenomenon at the edge of light segmentation and reduce the noise caused by light segmentation.

[0090] After the fifth image and the second image are synthesized, the synthesized image may be post-processed to improve image quality and enhance visual effects.

[0091] In some embodiments, post-processing the composite image may include: Perform histogram equalization on the composite image; determining an average brightness value of the composite image according to a histogram of the composite image; Determine the gamma correction parameters based on the average brightness value; The synthetic image is gamma corrected according to the parameters to obtain an enhanced image corresponding to the low-illumination image.

[0092] Histogram equalization and gamma correction are commonly used image enhancement techniques to improve the contrast and brightness of an image. Histogram equalization redistributes the pixel intensity of the image so that the histogram of the image is as evenly distributed as possible, thereby enhancing the contrast of the image. It is particularly suitable for images with unclear brightness contrast, making the details clearer. Gamma correction changes the overall brightness and contrast of the image by adjusting the brightness curve of the image. Specifically, gamma correction uses a power law function to convert pixel values ​​to new values ​​to achieve nonlinear adjustment of image brightness. This process can correct the phenomenon of overbrightness or overdarkness of the image, making the details of the image more visible under different brightness conditions. Combining histogram equalization and gamma correction can effectively improve image quality and enhance visual effects.

[0093] When post-processing the composite image using histogram equalization and adaptive parameter gamma correction method, first, the composite image can be histogram equalized to improve its contrast. Then, the average brightness value of the image can be calculated based on the histogram, and the brightness value can be input into a preset function to determine the parameters of the gamma correction. Finally, the calculated gamma value is applied to perform gamma correction, thereby further optimizing the brightness and contrast of the image, and finally obtaining the final enhanced image of the low-light image. The processing effect is as follows Figure 6 shown. Figure 6 In the figure, the dark image in the upper corner is a low-light image, the MoE image in the upper right corner is a synthetic image without post-processing obtained by enhancing the low-light image using the image enhancement method provided in an embodiment of the present application, the light image in the lower left corner is an image taken in a daytime scene, and the lower right corner is the final enhanced image obtained by post-processing the MoE image.

[0094] Table 3 shows the comparison of the enhancement effect of the image enhancement method provided by the embodiment of the present application and other methods on the three indicators of PSNR, SSIM and LPIPS: Table 3

[0095] It can be seen that the image enhancement method provided in the embodiment of the present application has better effect.

[0096] To facilitate understanding of the low-light image enhancement method provided in the embodiments of the present application, please refer to Figure 7 The embodiment shown. Figure 7 The illustrated embodiment includes the following steps.

[0097] Step 1: Perform light segmentation on the low-light image to determine the dark and bright areas.

[0098] When performing light segmentation on a low-light image, a brightness segmentation threshold can be set, such as 180, to identify and separate dark areas and bright areas in the image.

[0099] Step 2: Perform wavelet transform on the segmented dark and light areas using a Daubechies wavelet basis of length 8.

[0100] Step 3: Perform fusion enhancement processing on the wavelet transformed image to improve image quality and details.

[0101] Here, the image enhancement model provided in the embodiment of the present application can be used to perform fusion enhancement processing on the image after wavelet transformation. The specific implementation method can refer to the corresponding contents recorded in the above S106 to S110, which will not be repeated here.

[0102] Step 4: Perform inverse wavelet transform on the fused and enhanced image to restore the spatial information of the image.

[0103] Step 5: Synthesize the bright area of ​​the low-light image and the image after inverse wavelet transformation to obtain a synthesized image.

[0104] Step 6: Post-process the synthesized image to obtain an enhanced image corresponding to the low-illumination image.

[0105] Step 7: (Optional step) Compare the enhanced image with the high-illuminance image corresponding to the low-illuminance image and calculate the enhancement effect.

[0106] Figure 8 It is a schematic diagram of enhancing a low-light image according to an embodiment of the present application.

[0107] like Figure 8 As shown in the figure, the same object is photographed in the daytime to obtain a high-illuminance image, and in the night to obtain a low-illuminance image. When the low-illuminance image is enhanced, the image is first segmented to obtain a dark-light area image, and then the dark-light area image is enhanced to obtain an enhanced image, and the enhanced image is merged with the bright-light area image to restore the light and obtain the final enhanced image.

[0108] The low-light image enhancement method provided in the embodiment of the present application has at least the following technical effects: 1. Image registration: Low- and high-illumination image pairs taken in the field are often not fully registered and cannot be used for training. Image registration can greatly improve the utilization rate of images for training.

[0109] Specifically, the daytime and nighttime image pairs in real data cannot be collected at the same time. During this period, the camera operation will cause the image pair's perspective to change, and the difference in perspective will make the pixel-based learning method invalid. Therefore, for low-light image enhancement, image registration before training is necessary.

[0110] Different from ordinary image registration, the image pairs that need to be registered in low-light image enhancement are taken under different lighting conditions. We use the following two innovative methods to improve the registration effect under different lighting conditions: Occlusion information processing: When registering low-light images with high-light images, the low-light images have low light, low pixels, and unclear feature points. Before image registration, this technology pre-processes and masks labels such as text and dates that may interfere with feature matching, reducing the negative impact of such information on feature point matching and further improving matching accuracy; Feature matching based on deep learning: Low-light images and high-light images are not taken at the same time, and the perspectives are somewhat different. The embodiment of the present application introduces a feature matching method based on deep learning (such as LightGlue) to achieve efficient and accurate registration of image pairs with different perspectives. Combined with the cv2.findHomography function in OpenCV, the homography matrix is ​​calculated through at least four pairs of matching feature points, and the image is accurately projected and transformed, thereby achieving high-quality perspective alignment.

[0111] Through these innovations, the embodiments of the present application not only improve the registration accuracy of low-light images, but also enhance the quality and usability of the final images. The combination of these methods enables effective alignment of image pairs acquired under different lighting conditions, thereby providing a solid foundation for subsequent image enhancement and analysis.

[0112] 2. Light segmentation: Some dark images may contain strong light sources such as street lights and searchlights, which seriously affect the enhancement effect of the surrounding areas. Light segmentation and enhancement can effectively reduce this effect.

[0113] Specifically, in the image processing of night scenes, strong light sources such as street lamps often cause overexposure of images, affecting the image quality and processing effect. The innovative method we proposed effectively solves this problem by comprehensively applying the "light segmentation and enhancement" technology. The key innovations of this method are: Accurate light source overexposure detection: By analyzing the RGB value or grayscale value of each pixel in the image, we can accurately identify brightness anomalies caused by strong light sources such as street lamps. The set brightness threshold enables us to efficiently separate overexposed pixels from normal pixels, generate clear binary images, and accurately mark overexposed areas.

[0114] Segmented processing optimization: Traditional methods often process the entire image uniformly, but our technology achieves more refined optimization by processing normal areas and overexposed areas separately. Normal areas are processed by image enhancement, while overexposed areas are corrected by lighting, which greatly reduces the local overexposure caused by the light source while retaining the details and beauty of the image.

[0115] Adaptive adjustment: Our method can adaptively adjust processing parameters according to the characteristics of the specific image and the intensity of the light source to deal with overexposure problems in different scenes. This adaptive capability improves the flexibility and accuracy of the processing.

[0116] Overall improvement of enhancement effect: By merging the normal areas and overexposed areas that have been processed differently, we obtain the final optimized image. This method not only improves the visual effect of the image, but also improves the accuracy of subsequent image processing tasks such as image segmentation and object detection.

[0117] In short, when processing low-light images, the overexposure problem is effectively solved through innovative segmentation and enhancement strategies, the overall quality and processing effect of the image are improved, and a new solution is provided for night scene image processing.

[0118] 3. Wavelet transform: Compared with specific datasets such as LOL_v1, the actual image to be enhanced is usually very large. Wavelet transform can save feature information while scaling the image, reducing computing resource usage.

[0119] Specifically, low-light images are usually noisier and lack clear features. At the same time, the real image to be enhanced is sometimes too large in scale. Therefore, before enhancement, the improved Daubechies wavelet transform (db4 or db8) is applied to reduce the image scale, optimize image quality and suppress noise. Specific innovations include: Multi-scale wavelet transform: The db wavelet transform is used to decompose a large-size image into sub-images of multiple scales. This method can retain the low-frequency components with clear contours when scaling the image, while reducing the scale, effectively reducing the use of computing resources. The db wavelet transform can effectively capture the edge and detail information in the image, especially in low-light conditions, which is crucial for restoring the visibility of the image.

[0120] Noise suppression and detail enhancement: Under low illumination conditions, noise is often more significant. When processing noisy images, the embodiment of the present application first performs light segmentation, extracts normal areas, and then applies the db wavelet transform to these areas, thereby reducing noise and enhancing details at multiple scales. The tight support and high smoothness of the db wavelet ensure that the key features of the image are preserved, while noise is effectively suppressed, thereby better processing the various noises and details that may exist in low illumination images.

[0121] Efficient reconstruction and optimization: After multi-scale processing, the enhanced sub-images are reconstructed using inverse wavelet transform, and the image quality is finally optimized to ensure that the restored image has clear details and higher visual quality. This process improves the overall performance of the image and enables better results when processing large-size images.

[0122] 4. Model fusion: Fusion of multiple well-performing LLIE models, each model complements its own strengths and weaknesses, and improves the overall enhancement quality.

[0123] (1) MoE model selection method Different from the general model fusion that directly weights all models, when solving the low-light image enhancement problem, we innovatively insert the MoE module using the attention router method before weighting.

[0124] This method divides the low-light image into several small blocks, and uses the MoE module to select the most suitable m sub-models for each block of the image for subsequent weighted fusion. This approach is more in line with the actual needs of low-light image enhancement problems. The specific innovations are as follows: Evaluate the relationship between sub-models: Unlike the general MoE model that only calculates the pros and cons of each expert model, the attention router structure uses the self-attention mechanism to evaluate the relationship between experts. In this structure, each expert generates three feature vectors: Q (query vector), K (key vector), and V (value vector). Q and K are used to calculate the correlation coefficient matrix between experts, and then this matrix is ​​multiplied by V to obtain the final weight of each expert. Compared with the general MoE method, this method makes the selection of experts not only consider their own level, but also consider the correlation with other experts, thereby obtaining the weight of each sub-model.

[0125] This structure is very suitable for the model fusion task of low-light image enhancement. In general MoE, each expert model is good at different types of tasks, and the output variance is large; all experts in our model fusion task focus on low-light enhancement, so the output variance is small. The attention router structure can make the final output better reflect the opinions of the majority of experts, thereby enhancing image features.

[0126] Dynamic sub-model selection and weighting: The MoE architecture can dynamically select the most suitable expert sub-model instead of all expert sub-models according to the characteristics of the image area. This method can effectively capture the local features of the image and specifically enhance the quality of each image block, thereby improving the visual effect of the overall image.

[0127] Saving computing resources: The MoE model only activates some expert sub-models instead of all, which greatly reduces the computing overhead. This meets the actual needs of saving computing resources while maintaining high performance when processing complex low-light images.

[0128] Improved generalization ability: The flexibility of the MoE architecture allows it to select appropriate expert models for different image features, effectively dealing with various influencing factors in low-light images, such as noise and uneven illumination. By applying different expert sub-models to different image blocks, the MoE architecture can better generalize to unseen low-light image data, improving the robustness and stability of the model.

[0129] In summary, the embodiment of the present application innovatively applies the MoE architecture commonly used in large language models to low-light image enhancement tasks, filling the gap in the prior art and demonstrating the potential of the MoE model in improving the quality of low-light images. This method provides a new solution for the field of image processing and opens up new directions for future research and applications.

[0130] (2) Model fusion method based on self-attention mechanism The embodiment of the present application uses a new model fusion method, which significantly improves the enhancement effect of low-light images through comprehensive weighted fusion. Specific innovations include: Combining inter-block attention with inter-model attention: Upsample all small blocks after secondary block division to generate a vector containing the relationship information between different blocks and sub-model outputs. The weights of these upsampled results are calculated through self-attention transformation to optimize the contribution between different sub-models and blocks. This mechanism accurately captures the subtle differences between blocks, ensuring that the role of each image block in the final enhanced image is maximized, significantly improving the image's detail restoration ability and overall quality.

[0131] Smooth merging: All enhanced image blocks are merged and smoothed to reduce the boundary effect between blocks and ensure the coherence and naturalness of the final image. This process effectively avoids the artifact problem in traditional enhancement methods and improves the quality and visual consistency of the final enhanced image.

[0132] 5. Post-processing: further optimize the brightness and contrast of the image to achieve the best effect that meets the aesthetic taste of the human eye.

[0133] Specifically, the embodiment of the present application introduces an innovative post-processing technology that combines adaptive histogram equalization and dynamic gamma correction to significantly improve image quality. Specific innovations include: Processing flow: Although traditional histogram equalization and gamma correction techniques are effective, they are not combined. Histogram equalization can improve the global contrast of the image, but may cause local detail loss or noise enhancement. Although gamma correction can adjust the brightness of the image, it often fails to fully consider changes in local contrast. Therefore, the embodiment of the present application combines adaptive histogram equalization with dynamic gamma correction to create a method for comprehensively optimizing image quality, which can not only improve the global contrast, but also finely adjust the brightness and details, thereby achieving higher visual effects and image clarity.

[0134] Dynamic gamma correction: A dynamic gamma correction algorithm is introduced. The algorithm dynamically calculates the gamma parameters based on the local brightness characteristics and global contrast of the image, and further optimizes the brightness and contrast of the image through nonlinear adjustment. This method not only corrects the problem of uneven brightness, but also improves the visual effect and detail presentation of the image under different lighting conditions, significantly improving the overall image quality.

[0135] In general, the low-illumination image enhancement method provided in the embodiment of the present application can enhance any data to be enhanced, fully utilize the advantages of existing technologies, and achieve better results in both image quality and computational complexity.

[0136] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0137] Fig. 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig. 9 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0138] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0139] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0140] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a low-light image enhancement device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Performing light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; Performing wavelet transform on the first image to obtain a third image; Processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; Performing an inverse wavelet transform on the fourth image to obtain a fifth image; The fifth image and the second image are synthesized, and the synthesized image is post-processed to obtain an enhanced image corresponding to the low-illuminance image.

[0141] The above application Fig. 9The method performed by the low-light image enhancement device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0142] The electronic device may also perform Figure 1 Method and low illumination image enhancement device are implemented in Figure 1 The functions in the illustrated embodiments will not be described in detail in this application.

[0143] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0144] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations: Performing light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; Performing wavelet transform on the first image to obtain a third image; Processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; Performing an inverse wavelet transform on the fourth image to obtain a fifth image; The fifth image and the second image are synthesized, and the synthesized image is post-processed to obtain an enhanced image corresponding to the low-illuminance image.

[0145] Fig.10 FIG. 1 is a schematic diagram of a low-light image enhancement device 100 according to an embodiment of the present application. Fig.10 In a software implementation, the low illumination image enhancement device 100 may include: an image segmentation module 1001, a wavelet transform module 1002, an image enhancement module 1003, an inverse wavelet transform module 1004 and a post-processing module 1005, wherein: An image segmentation module 1001 performs light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; A wavelet transform module 1002 performs wavelet transform on the first image to obtain a third image; An image enhancement module 1003 processes the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; The wavelet inverse transformation module 1004 performs wavelet inverse transformation on the fourth image to obtain a fifth image; The post-processing module 1005 synthesizes the fifth image and the second image, and performs post-processing on the synthesized image to obtain an enhanced image corresponding to the low-illuminance image.

[0146] In some implementations, the image enhancement module 1003 is trained to obtain the image enhancement model in the following manner: Acquire a plurality of high and low illumination image pairs, a standard training set, and the plurality of sub-models; Performing image registration on the multiple high and low illumination image pairs to obtain a first training set; Performing independent pre-training on the multiple sub-models on the standard training set to determine pre-training weights; Based on the expert model, the multiple sub-models are integrated to obtain a fusion model; The fusion model is fine-tuned on the first training set according to the pre-training weights and the fusion weights to obtain the image enhancement model.

[0147] In some implementations, the image enhancement module 1003 performs image registration on any high- and low-illumination image pair, including: Masking text information in the high-illumination and low-illumination image pairs; Processing the high and low illumination image pairs using a deep learning-based feature matching method; The processed image is projected and transformed to obtain the registered high and low illumination image pairs.

[0148] In some implementations, the image enhancement module 1003 processes the third image according to a pre-trained image enhancement model to obtain a fourth image, including: dividing the third image into a plurality of image blocks; For each image block, input the embedding result of the image block into the image enhancement model to obtain a target sub-model corresponding to the image block, wherein the target sub-model includes at least part of the multiple sub-models, and different image blocks correspond to different target sub-models; Performing enhancement processing on the image block according to the target sub-model to obtain an image enhancement result of the image block; The image enhancement results of the multiple image blocks are combined and smoothed to obtain the fourth image.

[0149] In some implementations, the image enhancement module 1003 inputs the embedding result of the image block into the image enhancement model to obtain a target sub-model corresponding to the image block, including: Inputting the embedding result of the image block into the image enhancement model to obtain a first vector, a second vector and a third vector, wherein the first vector and the second vector are used to calculate the correlation between the multiple sub-models, and the third vector is used to represent the initial weight values ​​of the multiple sub-models; Multiplying the first vector and the second vector to obtain a correlation coefficient matrix between the multiple sub-models; Multiplying the correlation coefficient matrix by the third vector to obtain target weight values ​​of the multiple sub-models; According to the target weight values ​​of the multiple sub-models, at least some of the sub-models are selected from the multiple sub-models as the target sub-models corresponding to the image block.

[0150] In some implementations, the image enhancement module 1003 performs enhancement processing on the image block according to the target sub-model to obtain an image enhancement result of the image block, including: Performing image enhancement on the image block according to the sub-model in the target sub-model to obtain multiple image enhancement results; Perform channel fusion on the multiple image enhancement results to obtain an image enhancement fusion result; Performing block processing on the image enhancement fusion result to obtain multiple image sub-blocks; Upsampling the multiple image sub-blocks to obtain a fourth vector, where the fourth vector is used to characterize the relationship between image enhancement results of different sub-models corresponding to different image sub-blocks; Performing a self-attention transformation on the fourth vector to obtain weighted weight values ​​between different sub-models corresponding to different image sub-blocks; The multiple image sub-blocks are weightedly fused according to the weighted weight values ​​to obtain an image enhancement result of the image block.

[0151] In some embodiments, the image segmentation module 1001 performs light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image, including: Determining the brightness of each pixel in the low-light image; Determining bright pixels and dark pixels in the low-light image according to the brightness of each pixel in the low-light image and a preset brightness threshold; Binarizing the low-light image according to the bright-light pixels and the dark-light pixels to generate a binary image; A first image corresponding to a dark area and a second image corresponding to a bright area in the low-illuminance image are determined according to the binary image.

[0152] In some implementations, the post-processing module 1005 performs post-processing on the synthesized image to obtain an enhanced image corresponding to the low-illumination image, including: Performing histogram equalization on the composite image; Determining an average brightness value of the composite image according to a histogram of the composite image; Determining a gamma correction parameter according to the average brightness value; Gamma correction is performed on the synthesized image according to the parameters to obtain an enhanced image corresponding to the low-illumination image.

[0153] The low-light image enhancement device 100 provided in the present application can also perform Figure 1 The method is implemented by the low-light image enhancement device 100 Figure 1 The functions of the illustrated embodiments will not be described in detail in this application.

[0154] The present application also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the above-mentioned low-illumination image enhancement method embodiment.

[0155] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0156] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0157] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0159] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A low-light image enhancement method, characterized in that: include: Performing light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; Performing wavelet transform on the first image to obtain a third image; Processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; Performing an inverse wavelet transform on the fourth image to obtain a fifth image; The fifth image and the second image are synthesized, and the synthesized image is post-processed to obtain an enhanced image corresponding to the low-illuminance image.

2. The method according to claim 1, characterized in that The image enhancement model is trained in the following way: Acquire a plurality of high and low illumination image pairs, a standard training set, and the plurality of sub-models; Performing image registration on the multiple high and low illumination image pairs to obtain a first training set; Performing independent pre-training on the multiple sub-models on the standard training set to determine pre-training weights; Based on the expert model, the multiple sub-models are integrated to obtain a fusion model; The fusion model is fine-tuned on the first training set according to the pre-training weights and the fusion weights to obtain the image enhancement model.

3. The method according to claim 2, characterized in that For any high- and low-illumination image pair, image registration is performed on the high- and low-illumination image pair, including: Masking text information in the high-illumination and low-illumination image pairs; Processing the high and low illumination image pairs using a deep learning-based feature matching method; The processed image is projected and transformed to obtain the registered high and low illumination image pairs.

4. The method according to claim 1, characterized in that The step of processing the third image according to the pre-trained image enhancement model to obtain a fourth image includes: dividing the third image into a plurality of image blocks; For each image block, inputting the embedding result of the image block into the image enhancement model to obtain a target sub-model corresponding to the image block, wherein the target sub-model includes at least part of the multiple sub-models, and different image blocks correspond to different target sub-models; Performing enhancement processing on the image block according to the target sub-model to obtain an image enhancement result of the image block; The image enhancement results of the multiple image blocks are combined and smoothed to obtain the fourth image.

5. The method according to claim 4, characterized in that The step of inputting the embedding result of the image block into the image enhancement model to obtain a target sub-model corresponding to the image block includes: Inputting the embedding result of the image block into the image enhancement model to obtain a first vector, a second vector and a third vector, wherein the first vector and the second vector are used to calculate the correlation between the multiple sub-models, and the third vector is used to represent the initial weight values ​​of the multiple sub-models; Multiplying the first vector and the second vector to obtain a correlation coefficient matrix between the multiple sub-models; Multiplying the correlation coefficient matrix by the third vector to obtain target weight values ​​of the multiple sub-models; According to the target weight values ​​of the multiple sub-models, at least some of the sub-models are selected from the multiple sub-models as the target sub-models corresponding to the image block.

6. The method according to claim 4, characterized in that The step of performing enhancement processing on the image block according to the target sub-model to obtain an image enhancement result of the image block includes: Performing image enhancement on the image block according to the sub-model in the target sub-model to obtain multiple image enhancement results; Perform channel fusion on the multiple image enhancement results to obtain an image enhancement fusion result; Performing block processing on the image enhancement fusion result to obtain multiple image sub-blocks; Upsampling the multiple image sub-blocks to obtain a fourth vector, where the fourth vector is used to characterize the relationship between image enhancement results of different sub-models corresponding to different image sub-blocks; Performing a self-attention transformation on the fourth vector to obtain weighted weight values ​​between different sub-models corresponding to different image sub-blocks; The multiple image sub-blocks are weightedly fused according to the weighted weight values ​​to obtain an image enhancement result of the image block.

7. The method according to claim 1, characterized in that The light segmentation is performed on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image, including: Determining the brightness of each pixel in the low-light image; Determining bright pixels and dark pixels in the low-light image according to the brightness of each pixel in the low-light image and a preset brightness threshold; Binarizing the low-light image according to the bright-light pixels and the dark-light pixels to generate a binary image; A first image corresponding to a dark area and a second image corresponding to a bright area in the low-illuminance image are determined according to the binary image.

8. The method according to claim 1, characterized in that The post-processing of the synthesized image to obtain an enhanced image corresponding to the low-illumination image includes: Performing histogram equalization on the composite image; Determining an average brightness value of the composite image according to a histogram of the composite image; Determining a gamma correction parameter according to the average brightness value; Gamma correction is performed on the synthesized image according to the parameters to obtain an enhanced image corresponding to the low-illumination image.

9. A low-light image enhancement device, characterized in that: include: An image segmentation module performs light segmentation on the low-light image to be processed to obtain a first image corresponding to a dark light area and a second image corresponding to a bright light area of ​​the low-light image; A wavelet transform module, performing wavelet transform on the first image to obtain a third image; an image enhancement module, processing the third image according to a pre-trained image enhancement model to obtain a fourth image, wherein the image enhancement model is a model obtained by fusing a plurality of sub-models for image enhancement based on an expert model, and the image enhancement model is used to select at least some sub-models from the plurality of sub-models to perform enhancement processing on different regions of the third image, and different sub-models are used to perform enhancement processing on different regions; A wavelet inverse transformation module, performing a wavelet inverse transformation on the fourth image to obtain a fifth image; A post-processing module is used to synthesize the fifth image and the second image, and to post-process the synthesized image to obtain an enhanced image corresponding to the low-illuminance image.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

Images