Low-illumination image enhancement method and system based on optical prior and spectrum constraint

Abstract

The invention discloses a low-illumination image enhancement method and system based on optical prior and spectrum constraint. The method comprises the following steps: acquiring a low-light image to be processed; processing the low-light image through a pre-trained image enhancement network, specifically, performing preset optical prior extraction on the low-light image to obtain an optical prior image corresponding to the three illumination irrelevant quantities; taking the first optical prior image, the second optical prior image and the brightness prior image as auxiliary information, inputting the auxiliary information and the low-light image into a preset first branch network, and extracting an illumination image; taking the third optical prior image as auxiliary information, inputting the third optical prior image and the low-light image into a preset second branch network, and extracting a structure diagram; multiplying the illuminance diagram by the structure diagram, and adding the multiplied illuminance diagram and the structure diagram to obtain a preliminary enhanced image; and de-noising the preliminarily enhanced image and enhancing the color and the brightness to obtain a finally enhanced image. According to the invention, the network feature expression capability and the image enhancement effect can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a low-illumination image enhancement method and system based on optical prior and spectrum constraints. Background Art

[0002] In daily life, images cannot be captured under ideal lighting conditions due to many factors. Physical factors such as weak lighting and insufficient exposure time turn images into low-light images with dark areas in most areas, and also cause degradations such as loss of details, reduced contrast, sensor noise amplification, and color distortion. These degradations not only make the visual effect of the image worse, but also affect downstream tasks of computer vision, such as object detection, image matching, and semantic segmentation. The low-light image enhancement task aims to restore low-light input images to well-exposed images, which is not only a visual requirement, but also a key preprocessing step before downstream applications. At present, although there are some neural network methods to enhance low-light images. However, the existing low-light image enhancement algorithms still have the following problems:

[0003] (1) When using neural networks to extract low-light image features, there is a lack of extraction and use of imaging optical prior information, which will cause the neural network to lack the support and guidance of real physical models when modeling, limiting the space for further improvement of image enhancement effects;

[0004] (2) When using neural networks to achieve low-light image enhancement, learning is usually only performed in the spatial domain, and the frequency domain information is not utilized. This makes it impossible for the neural network to fully mine the useful information in the image and also makes the overall enhancement effect not good enough.

[0005] Therefore, how to increase the extraction of image illumination prior information and use frequency domain information to assist network learning are key technical issues that need to be solved urgently. Summary of the invention

[0006] The main purpose of the present invention is to provide a low-illumination image enhancement method and system based on optical prior and spectrum constraints that can simultaneously solve the above problems.

[0007] The technical solution adopted by the present invention is:

[0008] A low-light image enhancement method based on optical prior and spectrum constraint is provided, comprising the following steps:

[0009] Acquire a low-light image to be processed;

[0010] The low-light image is processed by a pre-trained image enhancement network, specifically:

[0011] A preset optical prior extraction is performed on the low-light image to obtain optical prior maps corresponding to three illumination-independent quantities, wherein the first optical prior map is an image H that is independent of illumination based on the assumption that the spectral wavelength is constant, the second optical prior map is an image S that is independent of illumination when the object surface is further assumed to be matte, and the third optical prior map is an image W that is independent of illumination when the illumination is further assumed to be uniform;

[0012] The first optical prior map, the second optical prior map and the brightness prior map are used as auxiliary information and input into a preset first branch network together with the low-light image to extract an illumination map; wherein the brightness prior map is obtained by taking the average value of the RGB three channels of the low-light image;

[0013] The third optical prior image is used as auxiliary information and input into a preset second branch network together with the low-light image to extract a structure map;

[0014] The illumination map and the structure map are multiplied and then added to the structure map to obtain a preliminary enhanced image;

[0015] Denoising and enhancing the color and brightness of the preliminary enhanced image to obtain the final enhanced image;

[0016] Among them, when training the image enhancement network, the Fourier loss function is introduced, the input image is transformed into a two-dimensional Fourier transform to obtain a spectrum map, and the texture details and brightness of the image are learned. The output image is decomposed into high-frequency images and low-frequency images using fast Fourier transform, and then pixel-by-pixel constraints are performed separately.

[0017] Following the above technical solution, when extracting the optical prior, the spectral equation of the low-light image and its first and second partial derivatives with respect to wavelength are specifically associated with the RGB spatial pixel values ​​of the low-light image to generate a mapping matrix; the spectrum of the low-light image is processed by a Gaussian derivative filter to obtain two spatial derivative maps, which are then combined with the expressions of three illumination-independent quantities to calculate the first optical prior map, the second optical prior map and the third optical prior map.

[0018] Following the above technical solution, the IGT module in the Retinexformer network is used to denoise the preliminary enhanced image, the attention module IGAB guided by illumination information is inserted in each downsampling and upsampling process, and the intermediate features of the first branch network are introduced to guide the network to learn the context information of the image.

[0019] Following the above technical solution, during training, the IGT module introduces the Fourier loss function, performs a two-dimensional Fourier transform on the input image to obtain a spectrum diagram, uses a fast Fourier transform to decompose the output image into a high-frequency image and a low-frequency image, and performs pixel-by-pixel L1 regularization constraints on the high-frequency image and the low-frequency image, respectively.

[0020] Following the above technical solution, during training, the IGT module performs the same preprocessing operation on each pair of images in the training set to expand the data set, wherein the preprocessing operation includes flipping, rotation and translation.

[0021] Following the above technical solution, when testing the IGT module, the peak signal-to-noise ratio PSNR and structural similarity SSIM are used for evaluation.

[0022] Following the above technical solution, a U-net network is specifically used to denoise the preliminary enhanced image.

[0023] The present invention also provides a low-illumination image enhancement system based on optical prior and spectrum constraint, comprising:

[0024] An image acquisition module, used for acquiring a low-light image to be processed;

[0025] The image enhancement network module is used to process the low-light image through a pre-trained image enhancement network, specifically including:

[0026] The optical prior extraction submodule is used to perform a preset optical prior extraction on the low-light image to obtain optical prior maps corresponding to three illumination-independent quantities, wherein the first optical prior map is an image H that is independent of illumination based on the assumption that the spectral wavelength is constant, the second optical prior map is an image S that is independent of illumination when the surface of the object is further assumed to be matte, and the third optical prior map is an image W that is independent of illumination when the illumination is further assumed to be uniform;

[0027] The image decomposition submodule is used to input the first optical prior map H, the second optical prior map S and the brightness prior map as auxiliary information together with the low-light image into a preset first branch network to extract an illumination map; wherein the brightness prior map is obtained by taking the average value of the RGB three channels of the low-light image; and input the third optical prior map W as auxiliary information together with the low-light image into a preset second branch network to extract a structure map;

[0028] A denoising and enhancement submodule is used to multiply the illumination map and the structure map and then add the structure map to obtain a preliminary enhanced image; and to denoise the preliminary enhanced image and enhance the color and brightness to obtain a final enhanced image;

[0029] Among them, when training the image enhancement network, the Fourier loss function is introduced, the input image is transformed into a two-dimensional Fourier transform to obtain a spectrum map, and the texture details and brightness of the image are learned. The output image is decomposed into high-frequency images and low-frequency images using fast Fourier transform, and then pixel-by-pixel constraints are performed separately.

[0030] Following the above technical solution, the optical prior extraction submodule is specifically used to associate the spectral equation of the low-light image and its first and second partial derivatives with respect to wavelength with the RGB spatial pixel values ​​of the low-light image to generate a mapping matrix; the spectrum of the low-light image is processed by a Gaussian derivative filter to obtain two spatial derivative maps, and then combined with the expressions of three illumination-independent quantities, the first optical prior map H, the second optical prior map S and the third optical prior map W are calculated.

[0031] The present invention also provides a computer storage medium, which stores a computer program that can be executed by a processor, and the computer program executes the low-illumination image enhancement method based on optical prior and spectrum constraints described in the above technical solution.

[0032] The beneficial effects of the present invention are as follows: the present invention extracts the optical prior map and inputs the optical prior map into the illumination map branch network and the structure map branch network to generate the illumination map and the structure map according to the characteristics, thereby realizing the extraction and utilization of the optical prior information, providing physical model support and guidance for the network to mine image features, and improving the network enhancement effect. Furthermore, the present invention introduces Fourier loss in the low-light image enhancement network, allowing the network to better pay attention to the high-frequency information and low-frequency information of the image, and further improving the network feature expression capability and image enhancement effect by combining spatial domain information and frequency domain information.

[0033] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a schematic diagram of the improvement idea of ​​the present invention based on the traditional Retinex theoretical method;

[0036] Figure 2 is a flow chart of a low illumination image enhancement method based on optical prior and spectrum constraint according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the method framework of the embodiment of the present invention Figure 1 ;

[0038] Figure 4 This is a schematic diagram of the method framework of the embodiment of the present invention Figure 2 ;

[0039] Figure 5 is a schematic diagram of an optical prior image extraction process according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the IGT module structure of an embodiment of the present invention;

[0041] Figure 7 Schematic diagram of low-light image enhancement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.

[0044] In the present invention, it is also necessary to explain that if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance. In addition, it is also necessary to explain that the features of various embodiments of the present invention may be combined or combined in part or in whole, and as can be understood by those skilled in the art, may interact and operate in different ways. Each embodiment may be implemented independently of each other, or in an associated relationship.

[0045] According to Retinex theory, the low-light image I can be decomposed into the product of the reflectance map R and the illuminance map L:

[0046]

[0047] The classic Retinex theory simulates the color perception mechanism of the human eye and restores the true color and contrast of the image by separating illumination and reflection.

[0048] However, the classical Retinex theory lacks the guidance of physical models, which restricts the space for further improvement of image enhancement effect. In addition, the lack of utilization of frequency domain information will also make it impossible for the neural network to fully mine useful information in the image. Figure 1 As shown, in order to solve the above two technical problems, the present invention is mainly implemented by introducing optical prior information and Fourier loss.

[0049] like Figure 2-4 As shown, the low-light image enhancement method based on optical prior and spectrum constraint in the embodiment of the present invention includes the following steps:

[0050] S1, obtaining a low-light image to be processed;

[0051] S2. Processing the low-light image through a pre-trained image enhancement network, specifically including the following steps:

[0052] S21, performing a preset optical prior extraction on the low-light image to obtain optical prior maps corresponding to three illumination-independent quantities, wherein the first optical prior map is an image H that is independent of illumination based on the assumption that the spectral wavelength is constant, the second optical prior map is an image S that is independent of illumination when the object surface is further assumed to be matte, and the third optical prior map is an image W that is independent of illumination when the illumination is further assumed to be uniform;

[0053] S22, inputting the first optical prior map, the second optical prior map and the brightness prior map as auxiliary information together with the low-light image into a preset first branch network to extract an illumination map; wherein the brightness prior map is obtained by taking the average value of the RGB three channels of the low-light image;

[0054] S23, using the third optical prior image as auxiliary information and the low-light image together with the low-light image to input into a preset second branch network to extract a structure map;

[0055] S24, multiplying the illumination map and the structure map and then adding the result to the structure map to obtain a preliminary enhanced image;

[0056] S25, denoising the preliminary enhanced image and enhancing the color and brightness to obtain a final enhanced image.

[0057] In an embodiment of the present invention, when training an image enhancement network, a Fourier loss function is introduced, a two-dimensional Fourier transform is performed on the input image to obtain a spectrum map, the image gradient constraint is used to enhance the network's learning of the texture details and brightness of the image, the output image is decomposed into a high-frequency image and a low-frequency image using a fast Fourier transform, and pixel-by-pixel constraints are performed on each of them.

[0058] In step S21, the optical prior extraction process is as follows:

[0059] According to the Kubelka-Munk theory describing color image imaging, given the wavelength λ, the incident spectrum E at the spatial position x on the imaging plane can be expressed as:

[0060] E(λ,x)=e(λ,x)((1-i(x)) 2 R ∞ (λ,x)+i(x))

[0061] Among them, e(λ,x) represents the light source spectrum, i(x) represents the specular reflectivity, R ∞ (λ,x) represents the material reflectivity that is independent of illumination. For this theory, three assumptions are proposed, and the illumination-independent quantities under three conditions can be obtained.

[0062] Assumption 1: When the light is of equal energy, that is, the spectral wavelength is constant, the light source spectrum e(λ,x) can be simplified to be wavelength-independent

[0063]

[0064] From this we can deduce:

[0065]

[0066] Among them, E λ and E λλ represents the first and second partial derivatives of E with respect to λ, and Represents R ∞ The first and second partial derivatives of λ. The result of formula x is only related to the material reflectivity and has nothing to do with the illumination. Therefore, the first illumination-independent quantity v can be obtained. h :

[0067]

[0068] Assumption 2: Further assuming that the surface of the object is matte, that is, the specular reflectivity i(x)≈0, we can deduce the second illumination-independent quantity v s :

[0069]

[0070] Assumption 3: Based on the above assumption, if the illumination is uniform, the spectrum of the light source can be simplified to a parameter that is independent of the spatial position. We can deduce the third illumination-independent quantity v w :

[0071]

[0072] After deriving the three illumination-independent quantities from the spectral equation, it is necessary to associate the variables of the spectral equation with the RGB space. According to the Gaussian color model, the spectrum E and its partial derivative with respect to wavelength E λ and E λλ It can be linearly mapped using RGB space pixel values:

[0073]

[0074] In order to achieve better mapping effect, such as Figure 5 As shown, the present invention can choose to use a convolution layer as an extraction module of the illumination prior map to learn and generate the above 3×3 mapping matrix. Through the mapping matrix, the spectrum map E and its partial derivative E with respect to wavelength can be obtained. λ and E λλ Then, the Gaussian derivative filter is used to process the spectrum image E to obtain the spatial derivative image E of E. x and E y , change E, E λ 、E λλ 、E x 、E y By substituting the three derived illumination-independent formulas, we can obtain three optical prior maps H, S, and W.

[0075] After extracting the optical prior maps H, S, and W, the structure map I is extracted from the low-light image I in the first branch network (structure map branch) and the second branch network (illuminance map branch) according to the previously extracted optical prior maps. str And illumination diagram I illu . The structure diagram I str And illumination diagram I illu Multiply and then structure diagram I str After adding the initial enhanced image I', the intermediate feature F of the initial enhanced image I' and the illumination map branch is illu Input them together into the denoising enhancement module to obtain the final enhanced image I en .

[0076] Specifically, according to the Retinex theory, an image can be decomposed into an illumination map and a reflectance map, and the product of the two is the original image. Based on this theory, this patent extracts a structure map with good image details and an illumination map reflecting illumination information from a low-light image through the guidance of an optical prior map and a brightness prior map. To this end, the present invention designs two groups of structure map branches and illumination map branches with the same network structure. These two groups of branches extract the structure map I from the low-light image I respectively. str And illumination diagram I illu In addition to the low-light image I, the input of the two branches also introduces the extracted optical prior maps H, S, W and the brightness prior map L pAs auxiliary information, the brightness prior map L p It is obtained by averaging the three RGB channels of the low-light image I. The optical prior map is derived based on properties that are independent of illumination. According to observations, different components of the optical prior map reflect different information. The optical prior maps H and S reflect brightness information more, while the optical prior map W reflects structure information more. Therefore, H and S are input into the illumination map branch as auxiliary information, and W is input into the structure map branch as auxiliary information. At the same time, in order to help the network better learn brightness information, the brightness prior map L is additionally p As input to the illumination map branch.

[0077] The structures of the illumination map branch and the structure map branch can use Resnet block and depthwise separable convolution. This patent uses depthwise separable convolution to extract illumination maps and structure maps from low-light images.

[0078] After obtaining the structure map and illumination map through image decomposition, in order to better retain the structural information, the structure map I str And illumination diagram I illu Multiply and then structure diagram I str The initial enhanced image I' is added together to obtain the initial enhanced image I'. The initial enhanced image I' will be improved in image details, but it still needs to be enhanced in color, brightness and noise suppression. In order to solve the above problems, the initial enhanced image I' is input into the denoising enhancement module for further enhancement, and the intermediate feature F of the illumination map branch is illu The input is processed together as auxiliary information and the final enhanced image I is output en .like Figure 7 As shown, the left column is the input low-light image, and the right column of images are enhanced after being processed by the method of the present invention.

[0079] Preferably, the denoising enhancement module can be implemented using an IGT (Illumination-Guided Multi-head Self-Attention) module to enhance the color, brightness and noise suppression of the preliminary enhanced image. The IGT module adopts a U-net-like structure (such as Figure 6 As shown in Figure 2, the overall module can be divided into a downsampling branch and an upsampling branch. The input image is first sent to the downsampling branch. In the downsampling branch, the input image is first convolved by 3×3 to obtain the feature F 0 , and then generate 1 / 2 scale and 1 / 4 scale down-sampled features F by one IGAB+4×4 stride convolution with a step size of 2 and two IGAB+4×4 stride convolutions with a step size of 2. 1 and F 2 , then let F 2 After two IGAB treatments, F 2'. The up-sampling branch is symmetrical with respect to the down-sampling branch. 2 'By repeating two groups of 2×2 deconvolution with a step size of 2 + 1×1 convolution + two IGAB upsampling, the upsampled feature F is obtained 1 ' and F 0 ', set up skip connections between up-sampled features and down-sampled features of the same scale to prevent information loss. Finally, F 0 'Get the residual image I through 3×3 convolution re , the residual image is added to the input image to obtain the output image, which is the final enhanced image. The illumination-guided attention block IGAB introduces the intermediate features of the illumination map branch to better guide the network to learn the context information of the image. IGAB first reconstructs the three-dimensional input features, and then uses three fully connected layers to linearly project the reconstructed features to obtain m groups of Q i , K i 、V i At the same time, the illumination features are reconstructed and linearly projected to obtain m IL i matrix, and the corresponding V i Matrix multiplication gives V i ', to introduce illumination information. For each group of matrices, the K matrix is ​​transposed and multiplied with the Q matrix, and the multiplication result is multiplied with the V' matrix to achieve self-attention. The results of the m groups of self-attention are connected, processed by the fully connected layer, and the output features are obtained after adding the position encoding.

[0080] In order to allow the image enhancement network to better grasp the low-frequency information representing brightness and color and the high-frequency information representing texture and details, the Fourier loss L is introduced. fre The frequency of an image is an indicator of the intensity of grayscale changes in the image, and is the gradient of grayscale in the plane space. The frequency spectrum obtained by performing a two-dimensional Fourier transform on the image is the distribution map of the image gradient. The constraint of the image gradient can enhance the network's learning of the texture details and brightness of the image. For the enhanced image I output by the network en and the corresponding normal light image I gt , using fast Fourier transform to enhance the image I en Decompose into high frequency images and low frequency images Similarly, the normal light image I gt Decompose into high frequency images and low frequency images Then, L1 constraints are applied pixel by pixel to the high-frequency image and the low-frequency image respectively, so that the high-frequency and low-frequency images decomposed by the enhanced image and the normal light image are as similar as possible, which helps the network to better learn the frequency domain information:

[0081]

[0082] In addition, the conventional reconstruction loss is used to enhance the image I en and the corresponding normal light image I gt Perform pixel-by-pixel L1 constraints to make the pixel values ​​of each corresponding position of the enhanced image and the normal light image as similar as possible:

[0083] L rec =||I en -I gt || 1

[0084] The total loss function is as follows:

[0085] L total =L rec +L fre

[0086] To use the image enhancement network designed by the present invention to enhance low-light images, it is necessary to build a data set to train the model. This patent trains the network on an Nvidia 3060 GPU, using pytorch implementation, batchsize set to 8, and using the Adam optimizer. The LOL_v2_synthetic data set is used as the initial data set. LOL_v2_synthetic synthesizes low-light images from RAW images by analyzing the illumination distribution of low-light images. The data set contains 1,000 pairs of low-light / normal images, 900 pairs of images are divided for training, and 100 pairs of images are used for testing.

[0087] The specific training steps are as follows:

[0088] (1) Perform the same preprocessing operations such as flipping, rotating, and translating on each pair of images in the training set to expand the dataset and enhance the generalization of the model for low-light image enhancement;

[0089] (2) The model is trained using the training set for a total of 150,000 iterations;

[0090] (3) According to the changes in the model's accuracy on the validation set, adjust the model's hyperparameters and finally save the network model that performs best on the test set.

[0091] Model testing: Use the saved model to enhance the images in the test set and output the enhancement results for the low-light images. Evaluate the enhanced images with the corresponding normal-light images in the test set, using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure) as evaluation indicators. PSNR: Peak signal-to-noise ratio (PSNR) is often used to measure image distortion or noise levels. The larger the PSNR value between two images, the more similar they are. Given a clean image x and a noisy image y of size m×n, the PSNR is calculated as follows:

[0092]

[0093] Among them, MAX I Represents the possible maximum value of pixels of image I, which is 255 in the present invention.

[0094] SSIM: Structural similarity (SSIM) measures the similarity between two images based on brightness, contrast, and structure. The larger the SSIM value between two images, the more similar they are. Given a clean image x and a noisy image y of size m×n, SSIM is calculated as follows:

[0095]

[0096] Among them, μ x and μ y are the means of x and y respectively, and are the variances of x and y, respectively, is the variance of xy. 1 =(k 1 L) 2 , c 2 =(k 2 L) 2 , L is set to 255, k 1 Set to 0.01, k 2 Set to 0.03.

[0097] The final test results are shown in Table 1 below:

[0098] Table 1 Comparison of PSNR and SSIM evaluation indicators of various models

[0099]

[0100] The present invention extracts the optical prior map and inputs the optical prior map into the illumination map branch and the structure map branch according to the characteristics to generate the illumination map and the structure map, thereby realizing the extraction and utilization of the optical prior information, providing physical model support and guidance for the network to mine image features, and improving the network enhancement effect. In addition, Fourier loss is introduced into the low-light image enhancement network to allow the network to better pay attention to the high-frequency and low-frequency information of the image, and the network feature expression capability and image enhancement effect are further improved by combining spatial domain information and frequency domain information.

[0101] In order to implement the above method embodiment, the present invention further provides a low-illumination image enhancement system based on optical prior and spectrum constraint, comprising:

[0102] An image acquisition module, used for acquiring a low-light image to be processed;

[0103] The image enhancement network module is used to process the low-light image through a pre-trained image enhancement network, specifically including:

[0104] The optical prior extraction submodule is used to perform a preset optical prior extraction on the low-light image to obtain optical prior maps corresponding to three illumination-independent quantities, wherein the first optical prior map is an image H that is independent of illumination based on the assumption that the spectral wavelength is constant, the second optical prior map is an image S that is independent of illumination when the surface of the object is further assumed to be matte, and the third optical prior map is an image W that is independent of illumination when the illumination is further assumed to be uniform;

[0105] The image decomposition submodule is used to input the first optical prior map H, the second optical prior map S and the brightness prior map as auxiliary information together with the low-light image into a preset first branch network to extract an illumination map; wherein the brightness prior map is obtained by taking the average value of the RGB three channels of the low-light image; and input the third optical prior map W as auxiliary information together with the low-light image into a preset second branch network to extract a structure map;

[0106] The denoising and enhancement submodule is used to multiply the illumination map and the structure map and then add the structure map to obtain a preliminary enhanced image; and to denoise the preliminary enhanced image and enhance the color and brightness to obtain a final enhanced image.

[0107] Among them, when training the image enhancement network, the Fourier loss function is introduced, and the input image is subjected to a two-dimensional Fourier transform to obtain a spectrum map. The image gradient constraint is used to enhance the network's learning of the image's texture details and brightness. The output image is decomposed into high-frequency and low-frequency images using a fast Fourier transform, and pixel-by-pixel constraints are performed separately.

[0108] Among them, the optical prior extraction submodule is specifically used to associate the spectral equation of the low-light image and its first and second partial derivatives with respect to wavelength with the RGB spatial pixel values ​​of the low-light image to generate a mapping matrix; the spectrum of the low-light image is processed by a Gaussian derivative filter to obtain two spatial derivative maps, and then combined with the expressions of three illumination-independent quantities, the first optical prior map H, the second optical prior map S and the third optical prior map W are calculated.

[0109] Each module is mainly used to implement each step of the method embodiment, which will not be described in detail here.

[0110] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and when the program is executed by a processor, a corresponding function is realized. When the computer-readable storage medium of this embodiment is executed by a processor, the low-light image enhancement method based on optical prior and spectral constraints of the method embodiment is realized.

[0111] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0112] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0113] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A low-light image enhancement method based on optical prior and spectral constraints, characterized in that: The following steps are involved: Acquire a low-light image to be processed; The low-light image is processed by a pre-trained image enhancement network, specifically: A preset optical prior extraction is performed on the low-light image to obtain optical prior maps corresponding to three illumination-independent quantities, wherein the first optical prior map is an image H that is independent of illumination based on the assumption that the spectral wavelength is constant, the second optical prior map is an image S that is independent of illumination when the object surface is further assumed to be matte, and the third optical prior map is an image W that is independent of illumination when the illumination is further assumed to be uniform; The first optical prior map, the second optical prior map and the brightness prior map are used as auxiliary information and input into a preset first branch network together with the low-light image to extract an illumination map; wherein the brightness prior map is obtained by taking the average value of the RGB three channels of the low-light image; The third optical prior image is used as auxiliary information and input into a preset second branch network together with the low-light image to extract a structure map; The illumination map and the structure map are multiplied and then added to the structure map to obtain a preliminary enhanced image; Denoising and enhancing the color and brightness of the preliminary enhanced image to obtain the final enhanced image; Among them, when training the image enhancement network, the Fourier loss function is introduced, the input image is transformed into a two-dimensional Fourier transform to obtain a spectrum map, and the texture details and brightness of the image are learned. The output image is decomposed into high-frequency images and low-frequency images using fast Fourier transform, and then pixel-by-pixel constraints are performed separately.

2. The low-light image enhancement method based on optical prior and spectrum constraint according to claim 1, characterized in that: When extracting the optical prior, the spectral equation of the low-light image and its first and second partial derivatives with respect to wavelength are specifically associated with the RGB spatial pixel values ​​of the low-light image to generate a mapping matrix; the spectrum of the low-light image is processed by a Gaussian derivative filter to obtain two spatial derivative maps, which are then combined with the expressions of three illumination-independent quantities to calculate the first optical prior map, the second optical prior map, and the third optical prior map.

3. The low-light image enhancement method based on optical prior and spectrum constraint according to claim 1, characterized in that: Specifically, the IGT module in the Retinexformer network is used to denoise the preliminary enhanced image, and the attention module IGAB guided by illumination information is inserted in each downsampling and upsampling process, and the intermediate features of the first branch network are introduced to guide the network to learn the contextual information of the image.

4. The low-light image enhancement method based on optical prior and spectrum constraint according to claim 3, characterized in that: During the training of the IGT module, the decomposed high-frequency image and low-frequency image are subjected to pixel-by-pixel L1 regularization constraints respectively, so that the high- and low-frequency images decomposed by the enhanced image and the normal light image remain consistent.

5. The low-light image enhancement method based on optical prior and spectrum constraint according to claim 3, characterized in that: When training, the IGT module performs the same preprocessing operation on each pair of images in the training set to expand the data set, where the preprocessing operations include flipping, rotation, and translation.

6. The low-light image enhancement method based on optical prior and spectrum constraint according to claim 3, characterized in that: When testing the IGT module, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used for evaluation.

7. The low-light image enhancement method based on optical prior and spectrum constraint according to claim 1, characterized in that: Specifically, the U-net network is used to denoise the preliminary enhanced image.

8. A low-light image enhancement system based on optical priors and spectral constraints, characterized in that: include: An image acquisition module, used for acquiring a low-light image to be processed; The image enhancement network module is used to process the low-light image through a pre-trained image enhancement network, specifically including: The optical prior extraction submodule is used to perform a preset optical prior extraction on the low-light image to obtain optical prior maps corresponding to three illumination-independent quantities, wherein the first optical prior map is an image H that is independent of illumination based on the assumption that the spectral wavelength is constant, the second optical prior map is an image S that is independent of illumination when the surface of the object is further assumed to be matte, and the third optical prior map is an image W that is independent of illumination when the illumination is further assumed to be uniform; The image decomposition submodule is used to input the first optical prior map H, the second optical prior map S and the brightness prior map as auxiliary information together with the low-light image into a preset first branch network to extract an illumination map; wherein the brightness prior map is obtained by taking the average value of the RGB three channels of the low-light image; and input the third optical prior map W as auxiliary information together with the low-light image into a preset second branch network to extract a structure map; A denoising and enhancement submodule is used to multiply the illumination map and the structure map and then add the structure map to obtain a preliminary enhanced image; and to denoise the preliminary enhanced image and enhance the color and brightness to obtain a final enhanced image; Among them, when training the image enhancement network, the Fourier loss function is introduced, the input image is transformed into a two-dimensional Fourier transform to obtain a spectrum map, and the texture details and brightness of the image are learned. The output image is decomposed into high-frequency images and low-frequency images using fast Fourier transform, and then pixel-by-pixel constraints are performed separately.

9. The low-light image enhancement system based on optical prior and spectrum constraint according to claim 8, characterized in that: The optical prior extraction submodule is specifically used to associate the spectral equation of the low-light image and its first and second partial derivatives with respect to wavelength with the RGB spatial pixel values ​​of the low-light image to generate a mapping matrix; the spectrum of the low-light image is processed by a Gaussian derivative filter to obtain two spatial derivative maps, and then combined with the expressions of three illumination-independent quantities to calculate the first optical prior map H, the second optical prior map S and the third optical prior map W.

10. A computer storage medium, characterized in that: A computer program executable by a processor is stored therein, and the computer program executes the low-illumination image enhancement method based on optical prior and spectrum constraint as described in any one of claims 1 to 7.

Images