A method, device and equipment for enhancing dark light images

Abstract

The present invention discloses a method, device and equipment for dark light image enhancement, which comprises: obtaining an input pair of images as training samples, wherein the input pair of images comprises a bright light image and a corresponding dark light image; inputting the input pair of images into a model constructed by a dark light image decomposition network, an image generator and a semantic discriminator and training based on a preset loss function to obtain a dark light enhancement model; inputting the dark light image to be enhanced into the trained dark light enhancement model to obtain an enhanced result image. The method can effectively improve the perception ability after enhancement, while retaining and restoring the semantic information of the original dark light image, so that the enhanced image is clearer and more natural, and the visual quality of the image is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a dark light image enhancement method, device and equipment. Background Art

[0002] In today's digital age, the booming development of the Internet and the popularization of smart terminal devices have greatly promoted the creation and dissemination of multimedia content. Among them, natural images, as an important carrier of information transmission, have a direct impact on the effective communication of information and the user's visual experience. In particular, images taken under low-light conditions often have problems such as poor color brightness, low perception and clarity due to the severe lack of ambient light and the limitations of the shooting equipment itself (such as sensor sensitivity, lens aperture size, etc.), and even the human eye has difficulty understanding and perceiving the image well. These problems not only reduce the visual beauty of the image, but also pose severe challenges to subsequent image processing tasks such as target detection, face recognition, image segmentation, etc., because these tasks are highly dependent on the high fidelity and rich detail information of the image.

[0003] Traditional low-light image enhancement methods, such as histogram equalization, Retinex algorithm and its variants, can improve the brightness distribution and contrast of images to a certain extent. However, when processing complex scenes, it is often difficult to balance the natural restoration of colors, the effective suppression of noise and the retention of detail information, resulting in enhanced images that are either too stiff and lose their natural feel, or still retain a lot of noise, affecting image quality. In addition, these methods are even less effective when processing dynamic scenes or images with complex textures, making it difficult to meet the growing demand for high-quality image processing. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a dark-light image enhancement method, device and equipment, aiming to solve the problems such as how to optimize the perception effect after dark-light image enhancement and improve the ability to process image details.

[0005] To achieve the above object, the present invention provides a dark light image enhancement method, the method comprising:

[0006] Acquire an input pair of images as training samples, wherein the input pair of images includes a bright light image and a corresponding dark light image;

[0007] The input pair image is input into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and trained based on a preset loss function to obtain a dark-light enhancement model; wherein,

[0008] Performing feature decomposition on the dark-light image through the dark-light image decomposition network to obtain a reflectance image and an illumination image;

[0009] The reflectance image and the illumination image are enhanced and reconstructed by the image generator, and based on the bright light image, an image correction is performed by using an adaptive jump diffusion strategy to obtain an enhanced reflectance feature map and an enhanced illumination feature map;

[0010] Extracting the semantic features of the enhanced reflectance feature map by the semantic discriminator to perform semantic discrimination and obtain a semantic feature vector;

[0011] The dark light image to be enhanced is input into the trained dark light enhancement model to obtain an enhanced result image.

[0012] Preferably, the dark-light image decomposition network is constructed based on a lightweight CBR module, and the CBR module includes a convolutional layer, batch normalization, and a ReLU activation function; the dark-light image decomposition network includes a reflectance map branch and an illumination map branch, and the reflectance map branch includes 7 of the CBR modules, and the illumination map branch includes 5 of the CBR modules.

[0013] Preferably, the preset loss function includes a first loss function and a second loss function; the inputting the input pair image into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and training based on the preset loss function to obtain a dark-light enhancement model includes:

[0014] In a first training stage, the dark-light image decomposition network is trained by using the first loss function;

[0015] In the second training stage, the dark light image decomposition network completed by pre-training is frozen, and the overall model is trained using the second loss function to obtain the dark light enhancement model; wherein,

[0016] The first loss function is , where L r represents the reconstruction loss, L ref represents the reflectivity consistency loss, L il represents the illumination smoothing loss, is the weight corresponding to each loss function;

[0017] The second loss function is , where L D represents the diffusion model denoising loss, L s1 , L s2 Represents the reflectivity image output using jump diffusion , reflectance image of adaptive diffusion output The semantic loss is sent to the semantic discriminator and calculated. , is the weight hyperparameter, , Represents the weight of semantic loss.

[0018] Preferably, the reconstruction loss is , where R high , L high represents the reflectance image and the illumination image obtained by decomposing the bright light image, R low , L low represents the reflectance image and the illumination image obtained by decomposing the dark light image, S high , S low represents the input bright light image and the dark light image, a r represents the hyperparameter, represents the Hadamard product;

[0019] The reflectivity consistency loss is ;

[0020] The illumination smoothing loss is , where , Represent the gradient of the illumination image in x and y respectively;

[0021] The diffusion model denoising loss is , where represents noise, For predicting noise, , , It increases between (0,1) as t increases;

[0022] The semantic loss is , where CLIP represents the semantic feature extraction operation, , R out Represents the enhanced reflectivity feature map.

[0023] Preferably, the image generator is constructed based on a diffusion model; and the enhanced reconstruction of the reflectivity image and the illumination image by the image generator includes:

[0024] The diffusion model is used to learn the enhanced mapping between the reflectivity image and the illumination image; the diffusion model includes a forward Gaussian noise addition process and a reverse denoising process; wherein,

[0025] The forward Gaussian noise process is expressed as , where represents the forward Gaussian noise process, represents the reflectivity image at time t, Indicates the mean , the variance is Gaussian distribution, It increases between (0,1) as t increases, and eventually approaches 1. I represents the identity matrix;

[0026] The reverse denoising process is expressed as , where represents a neural network that predicts noise, ,in, , .

[0027] Preferably, the adaptive jump diffusion strategy includes a coarse diffusion process and a fine diffusion process; using the adaptive jump diffusion strategy to perform image correction includes:

[0028] In the coarse diffusion process, the enhanced reflectivity image is generated by using a strategy of uniformly spaced G and S steps of jump diffusion. and lighting images ;

[0029] In the fine diffusion process, the perceptual features of the image are obtained through fine diffusion, and an adaptive diffusion strategy is introduced to guide the diffusion model to learn enhanced features, wherein the adaptive diffusion length set in the fine diffusion process is , T represents the full Markov chain of T steps.

[0030] Preferably, the method of acquiring the perceptual features of the image by fine diffusion and introducing an adaptive diffusion strategy to guide the diffusion model to learn enhanced features includes:

[0031] The pre-trained reward model is used to determine whether the current step is diffused to determine the decision action, which is expressed as: ,in, express The output of the module, the The module is pre-trained using the proximal strategy optimization PPO algorithm. The function represents a single diffusion operation, represents the reflectivity image of the jump diffusion output in the coarse diffusion process, Represents the reflectivity image after the decision was made and the action was executed at the last moment, Represents the reflectivity image after the current action is executed.

[0032] To achieve the above object, the present invention further provides a dark light image enhancement device, the device comprising:

[0033] A sample acquisition unit, used to acquire an input pair of images as training samples, wherein the input pair of images includes a bright light image and a corresponding dark light image;

[0034] A model training unit is used to input the input image into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and train it based on a preset loss function to obtain a dark-light enhancement model; wherein,

[0035] Performing feature decomposition on the dark-light image through the dark-light image decomposition network to obtain a reflectance image and an illumination image;

[0036] The reflectance image and the illumination image are enhanced and reconstructed by the image generator, and based on the bright light image, an image correction is performed by using an adaptive jump diffusion strategy to obtain an enhanced reflectance feature map and an enhanced illumination feature map;

[0037] Extracting the semantic features of the enhanced reflectance feature map by the semantic discriminator to perform semantic discrimination and obtain a semantic feature vector;

[0038] The dark light enhancement unit is used to input the dark light image to be enhanced into the trained dark light enhancement model to obtain an enhanced result image.

[0039] In order to achieve the above objectives, the present invention also proposes a low-light image enhancement device, including a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a low-light image enhancement method as described in the above embodiment.

[0040] In order to achieve the above-mentioned object, the present invention further proposes a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a dark-light image enhancement method as described in the above-mentioned embodiment.

[0041] In order to achieve the above-mentioned object, the present invention further proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a dark-light image enhancement method as described in the above-mentioned embodiment.

[0042] Beneficial effects:

[0043] The above scheme, by constructing a model including a dark-light image decomposition network, an image generator, and a semantic discriminator, and training and optimizing the obtained dark-light enhancement model based on a preset loss function, can effectively improve the perception ability after enhancement, while retaining and restoring the semantic information of the original dark-light image, and restoring the real light image to the greatest extent. By utilizing multiple features, the dark-light image is gradually enhanced from multiple angles to obtain the best enhancement strategy. The enhanced image is clearer and more natural, greatly improving the visual quality of the image. In addition, the number and cost of training are reduced through adaptive jump diffusion in the training stage, and the most streamlined structure is retained in the inference stage, so that the dark-light image is enhanced more lightly.

[0044] The above scheme uses a dark-light image decomposition network constructed with a lightweight CBR module to perform feature decomposition on the original dark-light image to obtain a reflectance image and an illumination image, thereby achieving refined processing of the image content and providing good feature information for subsequent enhancement. It can retain the key content of the original image and is also easy to learn the best enhancement effect.

[0045] The above scheme can accurately guide the training process of the entire model by presetting the loss function including the first loss function and the second loss function. The first loss function ensures the accuracy of the dark light image decomposition network through the combination of reconstruction loss, reflectivity consistency loss and illumination smoothness loss. By guiding the effective decomposition of dark light images, the model's ability to decompose dark light images is enhanced while improving the stability of model training. The reconstruction loss ensures the visual consistency of the enhanced image with the reference bright light image, and the reflectivity consistency loss and illumination smoothness loss ensure the naturalness and smoothness of the reflectivity image and the illumination image. The second loss function further improves the performance of the image generator through the combination of diffusion model denoising loss and semantic loss, so that the enhanced image has better semantic consistency and denoising effect while maintaining details. By introducing semantic loss, not only the difference at the pixel level of the image is paid attention to, but also the similarity of the image at the high-level semantics is considered, which helps to improve the generalization ability of the model, so that the model can better adapt to different types of dark light images and different application scenarios. At the same time, the semantic loss also promotes the model's in-depth understanding of the image content, so that it can more accurately capture and restore the key information in the image.

[0046] In the above scheme, the diffusion model introduced in the image generator further enhances the model's ability to capture and restore image details through the forward addition of Gaussian noise and the reverse denoising process, making the enhanced image richer and more realistic in details.

[0047] The above scheme can significantly reduce the model complexity and model iteration times by introducing a jump diffusion strategy during training. The adaptive jump diffusion includes a coarse diffusion process and a fine diffusion process, so that the enhanced image can obtain the best perceptual effect, and the diffusion step size can be dynamically adjusted according to the actual situation of the image, thereby further improving the enhancement effect while ensuring the enhancement efficiency. In particular, in the fine diffusion process, by introducing an adaptive diffusion strategy to guide the diffusion model to learn enhanced features, the enhanced image is more in line with the visual habits of the human eye in terms of perception.

[0048] The above scheme extracts the semantic features of the enhanced reflectance feature map through a semantic discriminator and performs semantic discrimination, thereby ensuring that the enhanced image remains semantically consistent, constraining the overall process, retaining the semantic information and semantic features of the original dark light image, and improving the model's ability to process image details.

[0049] The above scheme, this method is built based on deep learning, so it has strong learning and generalization capabilities. It can not only process different types of dark-light images, but also adapt to different lighting conditions and shooting equipment, providing an effective solution for dark-light image enhancement in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic flow chart of a dark light image enhancement method provided by an embodiment of the present invention.

[0052] Figure 2 A schematic diagram of a dark-light image decomposition network provided by an embodiment of the present invention.

[0053] Figure 3 A schematic diagram of an overall network structure provided for an embodiment of the present invention.

[0054] Figure 4 A schematic diagram of an image generator provided by an embodiment of the present invention.

[0055] Figure 5 A schematic structural diagram of a dark light image enhancement device provided by an embodiment of the present invention.

[0056] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0059] The present invention is described in detail below with reference to the embodiments.

[0060] Reference Figure 1 FIG. 1 is a flow chart of a dark light image enhancement method provided by an embodiment of the present invention.

[0061] In this embodiment, the method includes:

[0062] S11, obtaining an input pair of images as training samples, wherein the input pair of images includes a bright light image and a corresponding dark light image.

[0063] S12, inputting the input pair of images into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and training based on a preset loss function to obtain a dark-light enhancement model; wherein,

[0064] Performing feature decomposition on the dark-light image through the dark-light image decomposition network to obtain a reflectance image and an illumination image;

[0065] The reflectance image and the illumination image are enhanced and reconstructed by the image generator, and based on the bright light image, an image correction is performed by using an adaptive jump diffusion strategy to obtain an enhanced reflectance feature map and an enhanced illumination feature map;

[0066] The semantic feature of the enhanced reflectivity feature map is extracted by the semantic discriminator to perform semantic discrimination and obtain a semantic feature vector.

[0067] S13, inputting the dark light image to be enhanced into the trained dark light enhancement model to obtain an enhanced result image.

[0068] Furthermore, the dark-light image decomposition network is constructed based on a lightweight CBR module, which includes a convolutional layer, batch normalization, and a ReLU activation function; the dark-light image decomposition network includes a reflectance map branch and an illumination map branch, the reflectance map branch includes 7 of the CBR modules, and the illumination map branch includes 5 of the CBR modules.

[0069] Reference Figure 2 In this embodiment, an input pair of images including a bright light image and a corresponding dark light image is obtained as a training sample. The dark light image is feature decomposed by a dark light image decomposition network, which is expressed as:

[0070] ,in, represents the source image, which refers to the dark light image input by the model, R represents the reflectivity image, and L represents the illumination image. represents the Hadamard product, which is the matrix dot product.

[0071] The dark-light image decomposition network includes a reflectance map branch and an illumination map branch, which generate reflectance images and illumination images respectively. The dark-light image decomposition network is mainly composed of lightweight CBR modules. Each CBR module contains a convolution layer, batch normalization, and ReLU activation function; among them, the convolution kernel size of the convolution layer is 3×3, the step size is 1, and it is filled with 0. The reflectance map branch includes 7 CBR modules. At the same time, there is an average pooling layer between the first 4 CBR modules to perform 2x downsampling, and bilinear upsampling is used between the last 4 CBR modules. The output features of the first and last 3 CBR modules will be horizontally fused to provide multi-scale features. Finally, the sigmoid layer is connected at the output to obtain the normalized reflectance image R T There are 5 CBR modules in the illumination image branch, which are similar to the processing of the reflectivity image branch, and finally output the illumination image L T .

[0072] Furthermore, the preset loss function includes a first loss function and a second loss function; the inputting the image into the model constructed by the dark-light image decomposition network, the image generator and the semantic discriminator and training based on the preset loss function to obtain the dark-light enhancement model includes:

[0073] In a first training stage, the dark-light image decomposition network is trained by using the first loss function;

[0074] In the second training stage, the dark light image decomposition network completed by pre-training is frozen, and the overall model is trained using the second loss function to obtain the dark light enhancement model; wherein,

[0075] The first loss function is , where L r represents the reconstruction loss, L ref represents the reflectivity consistency loss, L il represents the illumination smoothing loss, is the weight corresponding to each loss function;

[0076] The second loss function is , where L D represents the diffusion model denoising loss, L s1 , L s2 Represents the reflectivity image output using jump diffusion , reflectance image of adaptive diffusion output The semantic loss is sent to the semantic discriminator and calculated. , is the weight hyperparameter, , Represents the weight of semantic loss.

[0077] Reference Figure 3 As shown. In this embodiment, a model including a dark light image decomposition network, an image generator and a semantic discriminator is constructed for training optimization to obtain a dark light enhancement model. During training, the reconstruction loss, reflectivity consistency loss and illumination smoothing loss are first integrated to train the dark light image decomposition network; then the diffusion model denoising loss and semantic loss are integrated and the dark light image decomposition network is frozen for overall training to obtain a dark light image enhancement model. Among them, the dark light image decomposition network is trained in the first training stage, and the corresponding loss function is defined as:

[0078] ;

[0079] In the formula, represents the total loss of the dark-light image decomposition network, L r represents the reconstruction loss, L ref represents the reflectivity consistency loss, L il represents the illumination smoothing loss, ~ are the weights corresponding to each loss function, all set to 1.

[0080] In the second training phase, the dark-light image decomposition network pre-trained in the first training phase is frozen in advance, and an image generator is added to further correct the reflectance image and the illumination image. A semantic discriminator is additionally added to the reflectance image, and finally the overall model is trained. The corresponding loss function is:

[0081] ;

[0082] in, is the total loss of the second training stage, , represents the weight hyperparameter, represents the diffusion model denoising loss, , Represents the reflectance image output using jump diffusion , reflectance image of adaptive diffusion output The semantic loss is sent to the semantic discriminator and calculated. , Represents the weights of semantic loss, which are 0.5 and 1 respectively.

[0083] The dark-light image is feature decomposed through the dark-light image decomposition network, and a reconstruction loss is calculated for the output reflectance image and the illumination image, which is expressed as:

[0084] ;

[0085] In the formula, represents the reconstruction loss, , represents the reflectance image and illumination image obtained by decomposing the bright light image, , represents the reflectance image and illumination image obtained by decomposing the dark light image, , Represents the input bright light image and dark light image, is a hyperparameter. At the same time, reflectivity consistency loss and illumination smoothness loss are introduced to guide the effective decomposition of dark light images. Since different brightness does not affect the reflectivity of the image and there will be no significant changes in the smooth area, it will present a piecewise smooth phenomenon. Reflectivity consistency loss and illumination smoothness loss are expressed as:

[0086] ;

[0087] ;

[0088] in, , Represent the reflectivity consistency loss and illumination smoothness loss respectively, , They represent the gradients of the illumination image in x and y, respectively, reflecting the brightness difference in the corresponding directions.

[0089] Furthermore, the image generator is constructed based on a diffusion model; and the enhanced reconstruction of the reflectivity image and the illumination image by the image generator includes:

[0090] The diffusion model is used to learn the enhanced mapping between the reflectivity image and the illumination image; the diffusion model includes a forward Gaussian noise addition process and a reverse denoising process; wherein,

[0091] The forward Gaussian noise process is expressed as , where represents the forward Gaussian noise process, represents the reflectivity image at time t, Indicates the mean , the variance is Gaussian distribution, It increases between (0,1) as t increases, and eventually approaches 1. I represents the identity matrix;

[0092] The reverse denoising process is expressed as , where represents a neural network that predicts noise, ,in, , .

[0093] In this embodiment, an image generator is constructed based on a diffusion model, and the diffusion model is used to learn the optimal enhancement mapping between the reflectance image and the illumination image. The diffusion model contains two Markov processes, one is a forward Gaussian noise process, and the other is a learnable reverse denoising process. The forward process is expressed as:

[0094] ;

[0095] In the formula, represents the forward process, is the reflectivity image at time t (the same applies to the illumination image), Indicates the mean , the variance is Gaussian distribution, As t increases between (0,1), it increases and eventually approaches 1. I represents the identity matrix. For the forward process, it can also be completed by single-step diffusion, which is expressed as:

[0096] ;

[0097] In the formula, , that is, all products from 1 to t, ,noise It is sampled from a Gaussian distribution with mean 0 and variance 1.

[0098] Similar to the forward process, the reverse process of the diffusion model reconstructs the original high quality by gradually denoising. The reverse denoising process of the diffusion model is expressed as:

[0099] ;

[0100] In the formula, is a neural network used to predict noise, .

[0101] The denoising loss of the entire diffusion model is expressed as:

[0102] ;

[0103] in, represents noise, Used to predict noise; the entire diffusion model optimizes the loss function to obtain high-quality enhanced feature maps, including enhanced reflectivity feature maps and enhanced illumination feature maps at different times.

[0104] Furthermore, the adaptive jump diffusion strategy includes a coarse diffusion process and a fine diffusion process; using the adaptive jump diffusion strategy to perform image correction includes:

[0105] In the coarse diffusion process, the enhanced reflectivity image is generated by using a strategy of uniformly spaced G and S steps of jump diffusion. and lighting images ;

[0106] In the fine diffusion process, an image is obtained by fine diffusion (the image obtained after jump diffusion, including the reflectivity image and lighting images ) and introduces an adaptive diffusion strategy to guide the diffusion model to learn enhanced features, where the adaptive diffusion length set by the fine diffusion process is , T represents the full Markov chain of T steps.

[0107] Furthermore, the perceptual features of the image are obtained by fine diffusion, and an adaptive diffusion strategy is introduced to guide the diffusion model to learn enhanced features, including:

[0108] The pre-trained reward model is used to determine whether the current step is diffused to determine the decision action, which is expressed as: ,in, express The output of the module, the The module is pre-trained using the proximal strategy optimization PPO algorithm. The function represents a single diffusion operation, represents the reflectivity image of the jump diffusion output in the coarse diffusion process, Represents the reflectivity image after the decision was made and the action was executed at the last moment, Represents the reflectivity image after the current action is executed.

[0109] Reference Figure 4 As shown. In this embodiment, by introducing an adaptive jump diffusion strategy, the image is further corrected from coarse diffusion to approximate the bright light image to fine diffusion, thereby improving the model perception and reducing the number of sampling times. Adaptive jump diffusion allows the model to automatically find the best diffusion path (i.e., different outputs are obtained according to different decisions of module A), so that the enhanced image can obtain the best perception effect. Specifically, by using a Markov jump process to approximate a T-step full Markov chain, in the coarse diffusion process of N time steps in the first stage of the diffusion model, a jump diffusion strategy of S steps with a uniform interval of G is used to generate the enhanced reflectance map and lighting diagram (These two images are directly used as input for fine diffusion). In the specific implementation, T is 100, G is set to 10, and S is set to 9. Compared with the ordinary diffusion model, the number of sampling times is greatly shortened, and the larger the interval G, the fewer the sampling times. In the fine diffusion process of the second stage of the diffusion model, the enhanced result of the first stage is used as input, and a finer diffusion is performed to obtain the best perception effect of the image. At the same time, an adaptive diffusion strategy is introduced through deep reinforcement learning to guide the diffusion model to learn the best enhancement effect. The adaptive diffusion strategy is reflected in the fact that each step of fine diffusion in the second stage first makes a decision action through a pre-trained reward model, that is, whether to diffuse at the current step, so that the model can learn the best perception effect. The structure of the reward model consists of three layers of CBR modules and a GRU module. The output of its reward model only includes 0 and 1, corresponding to two decision actions. The entire decision state and action are expressed as:

[0110] ;

[0111] in, express The output of the module, that is, 0 means to perform the diffusion operation, 1 means to skip, The function represents a single diffusion operation, represents the output of jump diffusion in the coarse diffusion process, Represents the reflectivity image after the decision was made and the action was executed at the last moment, Represents the reflectivity image after the current action is executed. Among them, the adaptive diffusion length set by the second stage of fine diffusion process is .

[0112] In the adaptive diffusion strategy, the proximal policy optimization (PPO) algorithm is used to optimize the The module (the module used to output the Action) is simply pre-trained, and the reward function is expressed as:

[0113] ;

[0114] Where P is NIQE, which is a measure of perceptual quality. The indicator is calculated only when , and the reward function value of the intermediate step is 0.

[0115] In this embodiment, a semantic discriminator is designed to extract semantic features to perform semantic discrimination on the enhanced image, and the model is optimized to enhance the details. The dark light image will have a certain feature loss during the decomposition process, especially the noise degradation of the reflectivity image is more serious. Therefore, the image encoder model in the pre-trained CLIP is used to extract semantic features from the enhanced image, and the loss noise of the reflectivity image is restored through semantic loss, thereby improving the semantic information reconstruction capability of the image generation and improving the details and clarity. The semantic loss function is expressed as:

[0116] ;

[0117] ;

[0118] Among them, CLIP represents the semantic feature extraction operation, CLIP is the Contrastive Language-Image Pre-training model, R out represents the enhanced reflectance image, R high Represents the reflectivity image of the decomposed bright light image; input the reflectivity image into the CLIP model with the pre-trained backbone network RN50 to obtain the corresponding semantic feature vector. This semantic loss function acts on the reflectivity image output by the jump diffusion in the first stage of the image generator , reflectance image of the second stage adaptive diffusion output , a total of two semantic losses, in order to better control the enhancement details in the whole process.

[0119] After the dark light enhancement model training is completed, the dark light image to be enhanced is used as the model input for inference to obtain an enhanced result image with better visual effect. Unlike the model training stage, the inference stage does not need to go through the semantic discriminator, and does not require paired dark light image-bright light image input. Image enhancement can be performed by inputting only one dark light image.

[0120] Reference Figure 5 FIG. 4 is a schematic structural diagram of a dark light image enhancement device provided by an embodiment of the present invention.

[0121] In this embodiment, the device 20 includes:

[0122] A sample acquisition unit 21 is used to acquire an input pair of images as training samples, wherein the input pair of images includes a bright light image and a corresponding dark light image;

[0123] The model training unit 22 is used to input the input image into the model constructed by the dark-light image decomposition network, the image generator and the semantic discriminator and train it based on a preset loss function to obtain a dark-light enhancement model; wherein,

[0124] Performing feature decomposition on the dark-light image through the dark-light image decomposition network to obtain a reflectance image and an illumination image;

[0125] The reflectance image and the illumination image are enhanced and reconstructed by the image generator, and based on the bright light image, an image correction is performed by using an adaptive jump diffusion strategy to obtain an enhanced reflectance feature map and an enhanced illumination feature map;

[0126] Extracting the semantic features of the enhanced reflectance feature map by the semantic discriminator to perform semantic discrimination and obtain a semantic feature vector;

[0127] The dark light enhancement unit 23 is used to input the dark light image to be enhanced into the trained dark light enhancement model to obtain an enhanced result image.

[0128] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0129] The embodiment of the present invention further provides a low-light image enhancement device, which includes the low-light image enhancement device as described above, wherein the low-light image enhancement device can be used Figure 5 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0130] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0131] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0132] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0133] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0134] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The dark light image enhancement method shown. The computer readable storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0135] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A dark light image enhancement method is shown.

[0136] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0138] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A dark light image enhancement method, characterized in that: The method comprises: Acquire an input pair of images as training samples, wherein the input pair of images includes a bright light image and a corresponding dark light image; The input pair image is input into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and trained based on a preset loss function to obtain a dark-light enhancement model; wherein, Performing feature decomposition on the dark-light image through the dark-light image decomposition network to obtain a reflectance image and an illumination image; The reflectance image and the illumination image are enhanced and reconstructed by the image generator, and based on the bright light image, an adaptive jump diffusion strategy is used to perform image correction to obtain an enhanced reflectance feature map and an enhanced illumination feature map; wherein the adaptive jump diffusion strategy includes a coarse diffusion process and a fine diffusion process; and image correction using the adaptive jump diffusion strategy includes: In the coarse diffusion process, the enhanced reflectivity image is generated by using a strategy of uniformly spaced G and S steps of jump diffusion. and lighting images ; In the fine diffusion process, the perceptual features of the image are obtained through fine diffusion, and an adaptive diffusion strategy is introduced to guide the diffusion model to learn enhanced features, wherein the adaptive diffusion length set in the fine diffusion process is , T represents the full Markov chain of T steps; Extracting the semantic features of the enhanced reflectance feature map by the semantic discriminator to perform semantic discrimination and obtain a semantic feature vector; The dark light image to be enhanced is input into the trained dark light enhancement model to obtain an enhanced result image.

2. A dark light image enhancement method according to claim 1, characterized in that: The dark-light image decomposition network is constructed based on a lightweight CBR module, which includes a convolutional layer, batch normalization, and a ReLU activation function; the dark-light image decomposition network includes a reflectance map branch and an illumination map branch, the reflectance map branch includes 7 CBR modules, and the illumination map branch includes 5 CBR modules.

3. The dark light image enhancement method according to claim 1, characterized in that: The preset loss function includes a first loss function and a second loss function; the inputting the input pair image into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and training based on the preset loss function to obtain a dark-light enhancement model includes: In a first training stage, the dark-light image decomposition network is trained by using the first loss function; In the second training stage, the dark light image decomposition network completed by pre-training is frozen, and the overall model is trained using the second loss function to obtain the dark light enhancement model; wherein, The first loss function is , where L r represents the reconstruction loss, L ref represents the reflectivity consistency loss, L il represents the illumination smoothing loss, is the weight corresponding to each loss function; The second loss function is , where L D represents the diffusion model denoising loss, L s1 , L s2 Represents the reflectivity image output using jump diffusion , reflectance image of adaptive diffusion output The semantic loss is sent to the semantic discriminator and calculated. , is the weight hyperparameter, , Represents the weight of semantic loss.

4. The dark light image enhancement method according to claim 3, characterized in that: The reconstruction loss is , where R high , L high represents the reflectance image and the illumination image obtained by decomposing the bright light image, R low , L low represents the reflectance image and the illumination image obtained by decomposing the dark light image, S high , S low represents the input bright light image and the dark light image, a r represents the hyperparameter, represents the Hadamard product; The reflectivity consistency loss is ; The illumination smoothing loss is , where , Represent the gradient of the illumination image in x and y respectively; The diffusion model denoising loss is , where represents noise, For predicting noise, , , It increases between (0,1) as t increases; The semantic loss is , where CLIP represents the semantic feature extraction operation, , R out Represents the enhanced reflectivity feature map.

5. The dark light image enhancement method according to claim 1, characterized in that: The image generator is constructed based on a diffusion model; The step of enhancing and reconstructing the reflectivity image and the illumination image by the image generator comprises: The diffusion model is used to learn the enhanced mapping between the reflectivity image and the illumination image; the diffusion model includes a forward Gaussian noise addition process and a reverse denoising process; wherein, The forward Gaussian noise process is expressed as , where represents the forward Gaussian noise process, represents the reflectivity image at time t, It means the mean , the variance is Gaussian distribution, It increases between (0,1) as t increases, and eventually approaches 1. I represents the identity matrix; The reverse denoising process is expressed as , where represents a neural network that predicts noise, ,in, , .

6. The dark light image enhancement method according to claim 1, characterized in that: The method of obtaining the perceptual features of the image by fine diffusion and introducing an adaptive diffusion strategy to guide the diffusion model to learn enhanced features includes: The pre-trained reward model is used to determine whether the current step is diffused to determine the decision action, which is expressed as: ,in, express The output of the module, the The module is pre-trained using the proximal strategy optimization PPO algorithm. The function represents a single diffusion operation, represents the reflectivity image of the jump diffusion output in the coarse diffusion process, Represents the reflectivity image after the decision was made and the action was executed at the last moment, Represents the reflectivity image after the current action is executed.

7. A low-light image enhancement device, characterized in that: The device comprises: A sample acquisition unit, used to acquire an input pair of images as training samples, wherein the input pair of images includes a bright light image and a corresponding dark light image; A model training unit is used to input the input image into a model constructed by a dark-light image decomposition network, an image generator, and a semantic discriminator and train it based on a preset loss function to obtain a dark-light enhancement model; wherein, Performing feature decomposition on the dark-light image through the dark-light image decomposition network to obtain a reflectance image and an illumination image; The reflectance image and the illumination image are enhanced and reconstructed by the image generator, and based on the bright light image, an adaptive jump diffusion strategy is used to perform image correction to obtain an enhanced reflectance feature map and an enhanced illumination feature map; wherein the adaptive jump diffusion strategy includes a coarse diffusion process and a fine diffusion process; and image correction using the adaptive jump diffusion strategy includes: In the coarse diffusion process, the enhanced reflectivity image is generated by using a strategy of uniformly spaced G and S steps of jump diffusion. and lighting images ; In the fine diffusion process, the perceptual features of the image are obtained through fine diffusion, and an adaptive diffusion strategy is introduced to guide the diffusion model to learn enhanced features, wherein the adaptive diffusion length set in the fine diffusion process is , T represents the full Markov chain of T steps; Extracting the semantic features of the enhanced reflectance feature map by the semantic discriminator to perform semantic discrimination and obtain a semantic feature vector; The dark light enhancement unit is used to input the dark light image to be enhanced into the trained dark light enhancement model to obtain an enhanced result image.

8. A low-light image enhancement device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a dark light image enhancement method as claimed in any one of claims 1 to 6.

9. A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by a processor to implement the steps of a dark light image enhancement method as described in any one of claims 1 to 6.

Images