Image enhancement method based on double-domain illumination prior and electronic equipment

Abstract

The invention provides an image enhancement method based on double-domain illumination prior and electronic equipment, and relates to the field of image processing. The method comprises the steps of obtaining a to-be-processed illumination image; according to local illumination distribution characteristics of the to-be-processed illumination image, multi-scale enhancement is carried out on the to-be-processed illumination image after global illumination enhancement, a first illumination image after spatial domain enhancement is obtained, and the local illumination distribution characteristics comprise a brightness mean value and a brightness standard deviation; amplitude information of the to-be-processed illumination image in the frequency domain is enhanced, and a second illumination image enhanced in the frequency domain is obtained; and fusing the to-be-processed illumination image, the first illumination image and the second illumination image to obtain an enhanced target illumination image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, more particularly, to an image enhancement method based on dual-domain light prior and an electronic device. BACKGROUND

[0002] In actual scenes, under the conditions of weak light environment, night shooting or backlight, the intensity of light entering the imaging device is seriously insufficient, resulting in problems such as low overall brightness of the image, loss of dark details, enhanced noise, color distortion, etc. The signal-to-noise ratio of the image under low light conditions decreases sharply, and the contrast between bright and dark parts is unbalanced, which significantly reduces the visual quality of the image. Although related image enhancement methods have made certain progress, there are still three shortcomings: inaccurate light estimation, resulting in over-bright or over-dark enhanced regions; relying only on single-domain features in the spatial domain or frequency domain, ignoring the complementarity of the two; and simple feature fusion mechanism, which cannot adaptively select effective information. Therefore, an efficient enhancement method that can jointly model the light distribution in the frequency domain and the spatial domain is needed. SUMMARY

[0003] Therefore, the present application provides an image enhancement method based on dual-domain light prior and an electronic device.

[0004] One aspect of the present application provides an image enhancement method based on dual-domain light prior, comprising: obtaining a to-be-processed light image; performing multi-scale enhancement on the to-be-processed light image that has been globally enhanced in light, according to the local light distribution characteristics of the to-be-processed light image, to obtain a first light image that has been enhanced in the spatial domain; the local light distribution characteristics include the brightness mean and the brightness standard deviation; enhancing the amplitude information of the to-be-processed light image in the frequency domain to obtain a second light image that has been enhanced in the frequency domain; and fusing the to-be-processed light image, the first light image and the second light image to obtain a target light image that has been enhanced.

[0005] According to the embodiment of the present application, the to-be-processed light image that has been globally enhanced in light is enhanced according to the local light distribution characteristics of the to-be-processed light image, to obtain a first light image that has been enhanced in the spatial domain, comprising: enhancing the to-be-processed light image that has been globally enhanced in light, according to the local light distribution characteristics of the to-be-processed light image, to obtain a first intermediate light image; enhancing the first intermediate light image that has been globally enhanced in light, according to the local light distribution characteristics of the first intermediate light image, to obtain a second intermediate light image; wherein the scale of the local light distribution characteristics of the first light image is smaller than the scale of the local light distribution characteristics of the to-be-processed light image; enhancing the second intermediate light image that has been globally enhanced in light, according to the overall light distribution characteristics of the second intermediate light image, to obtain the first light image.

[0006] According to an embodiment of the present application, the image to be processed is enhanced according to the local illumination distribution characteristics of the image to be processed, and the image to be processed is enhanced according to the local illumination distribution characteristics of the first intermediate image, and the second intermediate image is obtained by enhancing the first intermediate image subjected to global illumination enhancement, and the enhanced target image is obtained by fusing the image to be processed, the first illumination image and the second illumination image.

[0007] According to an embodiment of the present application, the image to be processed is enhanced according to the local illumination distribution characteristics of the image to be processed, and the second intermediate image is obtained by enhancing the first intermediate image subjected to global illumination enhancement according to the local illumination distribution characteristics of the first intermediate image, and the enhanced target image is obtained by fusing the image to be processed, the first illumination image and the second illumination image.

[0008] According to an embodiment of the present application, the image to be processed is enhanced according to the local illumination distribution characteristics of the image to be processed, and the second intermediate image is obtained by enhancing the first intermediate image subjected to global illumination enhancement according to the local illumination distribution characteristics of the first intermediate image, and the enhanced target image is obtained by fusing the image to be processed, the first illumination image and the second illumination image.

[0009] According to an embodiment of the present application, the image to be processed is enhanced according to the local illumination distribution characteristics of the image to be processed, and the second intermediate image is obtained by enhancing the first intermediate image subjected to global illumination enhancement according to the local illumination distribution characteristics of the first intermediate image, and the enhanced target image is obtained by fusing the image to be processed, the first illumination image and the second illumination image.

[0010] According to an embodiment of the present application, the first weight, the second weight and the third weight are generated based on an attention mechanism according to the aggregated feature map, including: performing global pooling and average pooling on the aggregated feature map respectively, and splicing the results of the global pooling and the average pooling to obtain spliced features; inputting the spliced features into three feature extraction branches respectively, and outputting first feature values, second feature values and third feature values for respectively enhancing features of the first illumination image, the to-be-processed illumination image and the second illumination image; and processing the first feature values, the second feature values and the third feature values by using a normalization function respectively to obtain the first weight, the second weight and the third weight.

[0011] According to an embodiment of the present application, the amplitude information of the to-be-processed illumination image in the frequency domain is enhanced to obtain the second illumination image enhanced in the frequency domain, including: performing Fourier transform on the to-be-processed illumination image of each color channel to obtain frequency domain information; performing feature enhancement on the amplitude information separated from the frequency domain information to obtain enhanced amplitude information; performing inverse Fourier transform on the enhanced amplitude information and phase information in the frequency domain information to obtain a third illumination image of each color channel enhanced in the frequency domain; and splicing the third illumination images of each color channel to obtain the second illumination image.

[0012] According to an embodiment of the present application, the illumination enhancement model is configured to generate a target illumination image according to a to-be-processed illumination image; the illumination enhancement model is trained in the following manner: a sample illumination image is input into a to-be-trained illumination enhancement model to generate a target sample illumination image; a loss function is used to adjust the to-be-trained illumination enhancement model according to the target sample illumination image and a label illumination image corresponding to the sample illumination image, until a training condition is met, to obtain a trained illumination enhancement model; the loss function includes: a pixel reconstruction loss representing a pixel difference between the target sample illumination image and the label illumination image, and an amplitude consistency loss representing an amplitude difference between the target sample illumination image and the label illumination image.

[0013] Another aspect of the present application provides an electronic device, including: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as above.

[0014] Another aspect of the present application provides a computer-readable storage medium, storing computer-executable instructions, the instructions being used to implement the method as above when executed.

[0015] Another aspect of the present application provides a computer program product, the computer program product including computer-executable instructions, the instructions being used to implement the method as above when executed.

[0016] The application provides an image enhancement method based on a dual-domain light illumination prior, aiming to maintain image structural details and color information while realizing adaptive recovery of light distribution of a low-illumination image. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A flowchart of the image enhancement method based on the dual-domain light illumination prior according to an embodiment of the present application is shown.

[0019] Figure 2 A spatial domain image enhancement principle diagram of the image enhancement method based on the dual-domain light illumination prior according to an embodiment of the present application is shown.

[0020] Figure 3 A frequency domain image enhancement principle diagram of the image enhancement method based on the dual-domain light illumination prior according to an embodiment of the present application is shown.

[0021] Figure 4 A principle diagram of obtaining a second light image of the image enhancement method based on the dual-domain light illumination prior according to an embodiment of the present application is shown.

[0022] Figure 5 A selective core feature fusion principle diagram of the image enhancement method based on the dual-domain light illumination prior according to an embodiment of the present application is shown.

[0023] Figure 6 An image restoration result diagram of the present model combined with multiple models according to an embodiment of the present application is shown.

[0024] Figure 7 A block diagram of an electronic device suitable for implementing the image enhancement method based on the dual-domain light illumination prior according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, mean "comprising."

[0027] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein are defined as having meanings that are consistent with the context of the specification, and should not be interpreted in an idealized or overly formal way.

[0028] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally to be interpreted as including one or more of the same unless otherwise specified. For example, "a system having at least one of A, B, and C" is to be interpreted as including a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C together, etc.

[0029] The signal-to-noise ratio in low-light conditions decreases sharply, and the contrast between bright and dark parts is unbalanced, which significantly reduces the visual quality of the image. In addition, the light distribution of different channels is inconsistent, which is prone to color cast phenomenon; at the same time, the camera sensor will amplify the noise in the high gain mode, making it difficult for traditional enhancement algorithms to balance brightness enhancement and detail preservation. Although the image enhancement methods based on Retinex theory or deep learning have made some progress, they mostly model the light in a single domain (spatial domain or frequency domain), and cannot balance the global light rule and local structure details; the lack of bidirectional feature interaction and prior constraint between the frequency domain and the spatial domain makes the model unable to balance between brightness recovery and texture preservation; at the same time, the feature fusion method is mostly static convolution or simple splicing, which cannot adaptively adjust the weight of each branch according to the light distribution of the image.

[0030] Therefore, the present application provides an image enhancement method based on dual-domain light prior and electronic equipment, which at least partially solves the above technical problems.

[0031] According to an embodiment of the present application, the image enhancement method based on dual-domain illumination prior includes: obtaining a to-be-processed illumination image; performing multi-scale enhancement on the to-be-processed illumination image that has been subjected to global illumination enhancement according to local illumination distribution characteristics of the to-be-processed illumination image, to obtain a first illumination image that has been subjected to spatial domain enhancement, the local illumination distribution characteristics including brightness mean value and brightness standard deviation; enhancing the amplitude information of the to-be-processed illumination image in the frequency domain to obtain a second illumination image that has been subjected to frequency domain enhancement; and fusing the to-be-processed illumination image, the first illumination image and the second illumination image to obtain a target illumination image that has been subjected to enhancement.

[0032] In actual shooting scenes, in a low-light, night or backlight environment, the imaging device is insufficient in light, which can cause problems such as low image brightness, loss of dark details, color distortion, and reduced signal-to-noise ratio, resulting in poor image visual effects. The to-be-processed illumination image can be a low-illumination image obtained in the above-mentioned scene.

[0033] Figure 1 A flowchart of the image enhancement method based on dual-domain illumination prior according to an embodiment of the present application is shown. As shown in Figure 1 The to-be-processed illumination image is subjected to illumination enhancement based on spatial domain illumination distribution prior and frequency domain illumination distribution prior, respectively, and the first illumination image and the second illumination image that have been subjected to enhancement are selectively fused with the to-be-processed illumination image to obtain a target illumination image that has been subjected to enhancement.

[0034] The spatial domain can directly reflect the spatial distribution of pixels in the image. The brightness of the image is restored based on the correlation characteristics in the spatial domain, and the light is supplemented on the premise of preserving the local structure, so that the image content is clearer. Therefore, the spatial domain illumination modeling method is constructed based on the local statistical characteristics. First, the to-be-processed illumination image is subjected to global illumination enhancement, that is, the illumination distribution of the global pixels is adaptively optimized through multi-stage convolution-activation cascade and other technical means. On the premise of considering the brightening of the dark area, the retention of details in the bright area, the suppression of noise and color distortion, the global brightness balance of the low-illumination image is realized, and the hidden information is restored, so that the image visual effect is natural and the information is complete. To improve the stability and consistency of the brightness enhancement, the brightness of each local part of the to-be-processed illumination image that has been subjected to global illumination enhancement is enhanced according to the corresponding local illumination distribution characteristics of the to-be-processed illumination image, and then the first illumination image is obtained. The local illumination distribution characteristics include brightness mean value and brightness standard deviation.

[0035] The frequency domain can intuitively present the essential characteristics of image brightness distribution, detail richness and noise interference, and the overall brightness is optimized in the frequency domain, which can balance the brightness without interfering with the texture and edge details, so as to realize the natural improvement of global brightness of the image and the complete retention of core information. Since the brightness of the image is mainly reflected in the amplitude information, and the structure and texture are mainly maintained by the phase information, the amplitude information is only enhanced in the frequency domain, so that the brightness is improved while the image structure is stable. Specifically, the features of a to-be-processed illumination image in red (Red), green (Green) and blue (Blue) color channels are converted from the spatial domain to the frequency domain, and the amplitude information and the phase information of each color channel are obtained, wherein the three color channels can be referred to as R, G and B channels. The amplitude information is processed by convolution and an activation function to realize light-weight amplitude enhancement, then the image is restored from the frequency domain to the spatial domain based on the enhanced amplitude and the original phase, and the results of the three color channels are spliced to obtain a second illumination image enhanced in the frequency domain.

[0036] Finally, a selective feature fusion strategy based on a self-attention mechanism is used to adaptively fuse the to-be-processed illumination image, the first illumination image and the second illumination image, so as to realize information fusion of the to-be-processed illumination image after brightness enhancement in the spatial and frequency domains, and finally obtain an enhanced target illumination image.

[0037] The present application provides an image enhancement method based on dual-domain illumination prior, which aims to maintain the image structure details and color information while realizing the adaptive recovery of low-illumination image illumination distribution. The present application establishes illumination distribution prior in the frequency domain and the spatial domain respectively, realizes the unity of global illumination compensation and local detail optimization through the bidirectional prior collaborative mechanism of the frequency domain and the spatial domain, significantly improves the brightness performance and visual quality of the low-illumination image while ensuring the structure fidelity and color naturalness. At the same time, based on the self-attention mechanism, the frequency domain enhancement result, the spatial domain correction result and the original feature are dynamically weighted and fused, which maintains the detail structure while improving the overall brightness and color balance.

[0038] According to an embodiment of the present application, the image to be processed is enhanced according to the local illumination distribution characteristics of the image to be processed, and a first intermediate image is obtained. The first intermediate image is enhanced according to the local illumination distribution characteristics of the first intermediate image, and a second intermediate image is obtained. The scale of the local illumination distribution characteristics of the first intermediate image is smaller than the scale of the local illumination distribution characteristics of the image to be processed. The second intermediate image is enhanced according to the overall illumination distribution characteristics of the second intermediate image, and the first image is obtained.

[0039] According to an embodiment of the present application, the first image is obtained by performing three-stage image brightness enhancement on the image to be processed, and the first intermediate image and the second intermediate image are intermediate quantities in the three-stage image enhancement process.

[0040] It can be understood that the image to be processed or the image to be processed enhanced by global illumination can be divided into a plurality of sub-regions, and each sub-region can be referred to as a "local part". For each local part of the image to be processed enhanced by global illumination, the brightness mean value and the brightness standard deviation (i.e., the local illumination distribution characteristics) of each image local part of the image to be processed are obtained, and the brightness mean value and the brightness standard deviation of the enhanced image are obtained based on this. The image to be processed enhanced by global illumination is enhanced based on the brightness enhancement, and the first intermediate image is obtained.

[0041] Subsequently, the first intermediate image is globally enhanced in brightness by convolution. For example, two-stage "convolution and activation function" is used for global brightness enhancement. In the first stage operation, the convolution captures the local illumination distribution characteristics of the image by a local receptive field, the activation function filters the effective illumination information and suppresses the noise, and preliminarily distinguishes the region to be enhanced from the normal region. In the second stage operation, the convolution further extracts the global illumination correlation, and the activation function further strengthens the non-linear mapping capability. Through the above cascade operation, the global illumination enhancement of the image to be processed is finally realized.

[0042] For each local part of the first intermediate illumination image which has been globally brightness enhanced, the brightness is enhanced according to the local illumination distribution feature of the corresponding position of the first intermediate illumination image, to obtain a second intermediate illumination image. Similarly, the global brightness enhancement manner of the second intermediate illumination image, and the brightness enhancement manner by using the local illumination distribution feature are similar to the operation of the first illumination image, and will not be described here again. For the second intermediate illumination image which has been globally brightness enhanced, the second intermediate illumination image which has been globally brightness enhanced can be directly brightness enhanced according to the overall illumination distribution feature of the second intermediate illumination image, to obtain the first illumination image. Wherein, the subsequent two-stage global brightness enhancement manner, and the brightness enhancement manner by using the local / global illumination distribution feature are similar to the operation of the first illumination image, and will not be described here again.

[0043] The scale of the local illumination distribution feature of the first illumination image is smaller than the scale of the local illumination distribution feature of the to-be-processed illumination image. Wherein, the scale of the local illumination distribution feature can refer to the size of the sub-region to which the local illumination distribution feature is directed, for example, the size of the sub-region to which the local illumination distribution feature adopted in the first enhancement is smaller than the size of the sub-region to which the local illumination distribution feature adopted in the second enhancement is directed.

[0044] In the embodiment of the application, in addition, the multi-scale enhancement manner of the to-be-processed image by the medium-scale local illumination distribution feature, the small-scale local illumination distribution feature, and the original-scale overall illumination distribution feature can ensure the maintenance of the enhancement of the real illumination environment through the medium-scale enhancement, and on this basis, the accuracy of the low-light recovery of each region is ensured through the small-scale enhancement; and on this basis, the original-scale enhancement is performed to ensure the rationality and accuracy of the real illumination distribution of the entire image recovery. In summary, the embodiment of the application can realize multi-scale progressive enhancement by enhancing the to-be-processed image by using illumination distribution features of different scales and overall illumination distribution features, and balance the illumination authenticity and illumination accuracy.

[0045] Figure 2 The spatial image enhancement principle diagram of the image enhancement method based on the dual-domain illumination prior according to the embodiment of the application is shown. According to the local illumination distribution feature of the to-be-processed illumination image, the to-be-processed illumination image which has been globally brightness enhanced is enhanced to obtain a first intermediate illumination image, including: performing region division on the to-be-processed illumination image which has been globally brightness enhanced to obtain a plurality of first sub-regions of a first size; according to the local illumination distribution feature of each first sub-region and the corresponding sub-region of the to-be-processed illumination image, performing illumination distribution adjustment on the first sub-region to obtain an adjusted first sub-region; and according to the spatial position of the first sub-region in the to-be-processed illumination image, splicing a plurality of adjusted first sub-regions into the first intermediate illumination image.

[0046] As Figure 2As shown, first, the region division is performed on the global light enhanced to-be-processed light image 201 with the size of height x width (H x W), to obtain 16 first sub-regions with a first size, and the first size is The specific division manner is shown as 202. According to the brightness mean value and the brightness standard deviation of each first sub-region of the to-be-processed light image, the light distribution adjustment is performed on the first sub-region corresponding to the first intermediate light image after global light enhancement, to obtain a plurality of adjusted first sub-regions. The adjusted first sub-regions are spliced according to the positions of the first sub-regions in the to-be-processed image, to restore the first intermediate light image with the size of H x W. In the above splicing process, the difference between the brightness mean value and the brightness standard deviation of the to-be-processed image before global enhancement and the real light distribution can be used to selectively adjust the light distribution of the to-be-processed image after global enhancement. The difference between the brightness mean value and the brightness standard deviation of the to-be-processed image before global enhancement and the real light distribution can be obtained by using the parameter adjustment coefficient learned during the training.

[0047] According to the local light distribution characteristics of the first intermediate light image, the global light enhanced first intermediate light image is enhanced to obtain a second intermediate light image, including: performing region division on the global light enhanced first intermediate light image to obtain a plurality of second sub-regions with a second size, wherein the second size is smaller than the first size; according to the local light distribution characteristics of each second sub-region and the corresponding sub-region of the first intermediate light image, adjusting the light distribution of the second sub-region to obtain an adjusted second sub-region; and splicing the plurality of adjusted second sub-regions into the second intermediate light image according to the spatial positions of the second sub-regions in the to-be-processed light image.

[0048] First, the obtained first intermediate light image is globally enhanced in brightness. As shown in Figure 2 , the region division is performed on the global light enhanced first intermediate light image with the size of height x width (H x W), to obtain 256 second sub-regions with a second size, and the second size is . According to the first intermediate light image, the light distribution adjustment is performed on the corresponding sub-region of the global light enhanced first intermediate light image, to obtain a plurality of adjusted second sub-regions. After splicing, the second intermediate light image with the size of H x W is obtained.

[0049] For the convenience of understanding the specific operation of the light distribution adjustment, the light distribution adjustment of the first sub-region is taken as an example for description.

[0050] Based on the local illumination distribution characteristics of each first sub-region and the corresponding sub-region of the illumination image to be processed, the illumination distribution of the first sub-region is adjusted to obtain the adjusted first sub-region. This includes: acquiring the local illumination distribution characteristics of each first sub-region corresponding to the sub-region extracted from the illumination image to be processed; correcting the mean luminance and standard deviation luminance using adaptively learned parameter adjustment coefficients to obtain the corrected mean luminance and standard deviation luminance; and adjusting the illumination distribution of the first sub-region based on the corrected mean luminance, the corrected standard deviation luminance, and the mean luminance and standard deviation luminance to obtain the adjusted first sub-region.

[0051] The illumination image to be processed is also divided into sizes of [size]. After dividing the image into 16 sub-regions, the average brightness of the corresponding sub-region in the illumination image to be processed is obtained for each first sub-region. With the standard deviation of brightness and This is used to accurately characterize the current illumination level and contrast distribution characteristics. Based on adaptive learning of the differences between the real illumination distribution and the low illumination distribution during training, parameter adjustment coefficients can be determined, such as parameter factors and bias terms. After learning the parameter adjustment coefficients, the mean brightness and standard deviation of the current illumination image to be processed can be corrected using formulas (1) and (2) formed by the parameter adjustment coefficients, respectively:

[0052] (1)

[0053] (2)

[0054] In the formula, , These are the parameter factors for the mean and standard deviation of brightness, adaptively learned during training. , This refers to the bias terms for the mean and standard deviation of brightness learned adaptively during training. The average brightness value after correction. This represents the corrected standard deviation of brightness.

[0055] according to , as well as , For the first sub-region The illumination distribution is adjusted according to formula (3) to obtain the adjusted first sub-region. :

[0056] (3)

[0057] It should be noted that the light distribution adjustment of the second sub-region, the light distribution adjustment of the entire second intermediate light image can refer to the first sub-region, and the three are different in scale but the same or similar in operation.

[0058] In the embodiments of the present application, the local light distribution characteristics (brightness mean and brightness standard deviation) of each sub-region can measure the overall distribution of local light in the dimension of the sub-region; similarly, finally the overall light distribution characteristics (such as brightness mean and brightness standard deviation) of the entire second intermediate light image can measure the overall distribution of local light in the final overall image dimension. In the entire light enhancement process, by correcting the brightness mean and the brightness standard deviation which can measure the light distribution, and using the corrected brightness mean and the brightness standard deviation to adjust the image, the authenticity and accuracy of the light distribution adjustment can be improved.

[0059] Figure 3 A frequency domain image enhancement principle diagram of the image enhancement method based on the dual-domain light prior according to the embodiments of the present application is shown. The amplitude information of the to-be-processed light image in the frequency domain is enhanced to obtain a second light image enhanced in the frequency domain, including: performing Fourier transform on the to-be-processed light image of each color channel to obtain frequency domain information; performing feature enhancement on the amplitude information separated from the frequency domain information to obtain enhanced amplitude information; performing inverse Fourier transform on the enhanced amplitude information and the phase information in the frequency domain information to obtain a third light image of each color channel enhanced in the frequency domain; and splicing the third light image of each color channel to obtain the second light image.

[0060] As Figure 3 shown, the features of each single color channel of the to-be-processed light image R, G, B can be subjected to Fourier transform to obtain the frequency domain information of each color channel, to determine the amplitude information of each color channel based on the parameters of the frequency domain information of each color channel, and to realize amplitude acquisition. For the amplitude information of each color channel, the amplitude enhancement can be performed on it in a convolution manner, and the enhanced result is fused and restored to the spatial domain through inverse Fourier transform to obtain the second light image .

[0061] Figure 4 A principle diagram of obtaining the second light image according to the image enhancement method based on the dual-domain light prior according to the embodiments of the present application is shown.

[0062] As Figure 4 shown, the specific process of obtaining the second light image includes: first, performing fast Fourier transform on the to-be-processed light image through formula (4) to obtain the frequency domain information :

[0063] (4)

[0064] where h, w represent spatial coordinates, u, v represent frequency domain coordinates, j is an imaginary unit, represents a to-be-processed image in a certain color channel in the spatial domain.

[0065] After transformation, the real part and the imaginary part of the to-be-processed image are obtained. According to formulas (5) and (6), the amplitude information and the phase information can be calculated respectively.

[0066] (5)

[0067] (6)

[0068] where u, v represent frequency domain coordinates, is represented by an inverse tangent function.

[0069] In the experiment, the brightness of the image is mainly reflected in the amplitude information, and the structure and texture are mainly maintained by the phase information. Therefore, the amplitude information is only enhanced in the present application, so as to improve the brightness while maintaining the stability of the image structure. Preferably, in order to avoid the color deviation phenomenon caused by the difference between color channels, the present application respectively performs independent frequency domain enhancement processing on the R, G and B three color channels of the to-be-processed illumination image. For each color channel, the amplitude information and the phase information are obtained by fast Fourier transform, and then 1x1 convolution, activation function 1x1 convolution are sequentially performed for lightweight amplitude enhancement to obtain the enhanced amplitude information , and the calculation process is shown in formula (7):

[0070] (7)

[0071] where represents 1x1 convolution, is the amplitude information of any color channel before enhancement.

[0072] Subsequently, the enhanced amplitude information is combined with the original unenhanced phase information, and inverse Fourier transform is performed to restore to the spatial domain to obtain the third illumination image enhanced in the frequency domain as shown in formula (8): :

[0073]

[0074] where j is an imaginary unit, represents inverse Fourier transform, sin is a function, and P is phase information.

[0075] After the above operations on the three color channels R, G and B, the third illumination image of each color channel after frequency domain enhancement is obtained The third illumination images of the three color channels are spliced to obtain the second illumination image as shown in formula (9)

[0076]

[0077] Wherein, Cat() represents splicing the third illumination images of the three color channels.

[0078] According to the embodiment of the application, the to-be-processed illumination image, the first illumination image and the second illumination image are fused to obtain an enhanced target illumination image, including: performing pixel-by-pixel summation on the to-be-processed illumination image, the first illumination image and the second illumination image to obtain an aggregated feature map; generating a first weight, a second weight and a third weight based on an attention mechanism according to the aggregated feature map; fusing the superimposition result of the first weight and the first illumination image, the superimposition result of the second weight and the to-be-processed illumination image, and the superimposition result of the third weight and the second illumination image to obtain a fusion feature map; and connecting the fusion feature map after feature alignment with the to-be-processed illumination image residual to obtain the enhanced target illumination image.

[0079] In the fusion stage, first, the first illumination image The original to-be-processed image The second illumination image Element-by-element addition is performed to obtain the aggregated feature map L as shown in formula (10):

[0080]

[0081] Based on the attention mechanism, the first weight s1, the second weight s2 and the third weight s3 are generated according to the aggregated feature map L to realize dynamic adjustment of the contribution degree of each input branch. Then, the superimposition result of the first weight s1 and the first illumination image, the superimposition result of the second weight s2 and the to-be-processed image, and the superimposition result of the third weight s3 and the second illumination image are fused to obtain the fusion feature map U to strengthen effective features, suppress redundant information, and improve the image quality of the fusion feature map. Wherein, the expression of the fusion feature map U is shown in formula (11):

[0082]

[0083] ​In addition, a residual connection is further introduced after the fusion of the feature maps, and the features of the to-be-processed illumination image are directly fused with the enhanced fusion feature maps. Specifically, first, the to-be-processed illumination image and the fusion feature maps are made consistent under a unified distribution through matching, mapping and calibration operations, so as to realize feature alignment. The to-be-processed illumination image and the fusion feature maps generated after multi-stage processing such as convolution and activation function are cooperatively fused, so that the core information such as image basic structure and inherent brightness distribution contained in the to-be-processed illumination image and the illumination optimization features in the fusion feature maps form a complement, and finally the features with complete basic information integrity and illumination enhancement effect are output. Through the above operations, the system can dynamically allocate weights according to the illumination conditions of the input image, rely on the frequency domain enhancement branch in the area with insufficient illumination, and strengthen the spatial domain enhancement branch and the original feature branch in the area with rich texture, so as to realize optimal fusion and complement of information. Both the basic structure and the effective information of the to-be-processed illumination image are retained, and the problems of feature loss or gradient dissipation in multi-stage processing are avoided, and the illumination optimization features learned in the enhancement process are superimposed, so that the effect of not losing basic information and accurately superimposing enhanced features is realized. At the same time, the enhancement amplitude can be dynamically adjusted to suppress the side effects such as overexposure and noise amplification, so that the final target illumination image with global illumination enhancement is more consistent with the illumination distribution properties of the image authenticity, and the brightness balance and detail integrity are considered, and the adaptability and output stability of the model to complex low-light scenes are improved.

[0084] According to the embodiment of the application, based on the attention mechanism, the first weight, the second weight and the third weight are generated according to the aggregated feature map, including: performing global pooling and average pooling on the aggregated feature map respectively, and splicing the results of global pooling and average pooling to obtain spliced features; inputting the spliced features into three feature extraction branches respectively, and outputting first feature values, second feature values and third feature values for respectively enhancing the features of the first illumination image, the to-be-processed illumination image and the second illumination image; and using a normalization function to process the first feature values, the second feature values and the third feature values respectively to obtain the first weight, the second weight and the third weight.

[0085] First, the global average pooling (GAP) and global max pooling (GMP) operations are performed on the aggregated feature map L respectively, and the results of GAP and GMP are spliced to obtain the spliced feature s. The global average pooling takes the arithmetic mean of all spatial position pixel values of L as the global representation, which can smoothly integrate the overall information, weaken the local noise interference, and highlight the global distribution trend of the feature. The global maximum pooling selects the maximum value of the pixel in the spatial dimension of L as the global representation of the color channel, focusing on capturing the most significant and most discriminative local features in the color channel, and strengthening the sparseness of the feature and the recognition of the key information. The spliced feature s is input into the first illumination image, the to-be-processed illumination image and the second illumination image feature extraction branch, and after 1x1 convolution and PReLU activation, the first feature value v1, the second feature value v2 and the third feature value v3 are obtained. Then, the first weight s1, the second weight s2 and the third weight s3 are generated by using the normalization function softmax for normalization, so as to adaptively adjust the importance of each input branch (L1, L2, L3) using the above three weights.

[0086] Figure 5 The selective core feature fusion principle diagram of the image enhancement method based on the dual-domain illumination prior according to the embodiment of the application is shown.

[0087] As shown in Figure 5 , first, L1, L2 and L3 are added element by element to obtain the aggregated feature map L. Based on L, the global average pooling and global max pooling operations are performed to generate the spliced feature s. The spliced feature s is input into the first illumination image, the to-be-processed illumination image and the second illumination image feature extraction branch, and after 1x1 convolution and PReLU activation, the first feature value v1, the second feature value v2 and the third feature value v3 are obtained. Then, the first weight s1, the second weight s2 and the third weight s3 are generated by using the normalization function softmax for normalization, so as to adaptively adjust the importance of each input branch. L1, L2 and L3 are spliced with the output of each branch respectively, and then the spliced results are added element by element, that is, the fused feature map U after feature alignment is obtained.

[0088] According to an embodiment of the present application, the illumination enhancement model is configured to generate a target illumination image according to a to-be-processed illumination image; the illumination enhancement model is trained in the following manner: a sample illumination image is input into a to-be-trained illumination enhancement model to generate a target sample illumination image; a loss function is used to adjust the to-be-trained illumination enhancement model according to the target sample illumination image and a label illumination image corresponding to the sample illumination image, until a training condition is met, to obtain a trained illumination enhancement model; the loss function includes a pixel reconstruction loss representing a pixel difference between the target sample illumination image and the label illumination image, and an amplitude consistency loss representing an amplitude difference between the target sample illumination image and the label illumination image.

[0089] For example, the illumination enhancement model includes a frequency domain enhancement branch, a spatial domain enhancement branch and a selective core feature fusion module (SCFF), the spatial domain enhancement branch is configured to perform multi-scale enhancement on the to-be-processed illumination image that has been globally enhanced, according to local illumination distribution characteristics of the to-be-processed illumination image, to obtain a first illumination image that has been enhanced in the spatial domain; the frequency domain enhancement branch is configured to enhance the amplitude information of the to-be-processed illumination image in the frequency domain, to obtain a second illumination image that has been enhanced in the frequency domain; and the selective core feature fusion module is configured to fuse the to-be-processed illumination image, the first illumination image and the second illumination image, to obtain an enhanced target illumination image.

[0090] In the model training phase, the loss function of the joint constraint of the pixel domain and the frequency domain is used to ensure that the enhanced target sample illumination image approximates the label illumination image of the real illumination in brightness and structure. Specifically, the pixel reconstruction loss is expressed as formula (12):

[0091] (12)

[0092] wherein GT represents a label illumination image under a real illumination condition, I GT represents a target sample illumination image recovered by the model, and ||·|| represents an L1 norm. To constrain the amplitude consistency of the frequency domain illumination distribution, the amplitude consistency loss as shown in formula (13) is introduced :

[0093] (13)

[0094] wherein I GT represents amplitude information of the label illumination image, and I GT represents amplitude information of the sample illumination image.

[0095] The two losses are combined to obtain a loss function as shown in equation (14) :

[0096] (14)

[0097] wherein λ is a weight coefficient, preferably 0.01. Through joint optimization, the model can align the real brightness at the pixel level while accurately maintaining the light distribution characteristics in the frequency domain, thereby achieving the enhancement effect of global brightness consistency and local detail authenticity.

[0098] Further, the overall network framework of the present application can be combined with various existing structures (such as U-Net, Restormer, Retinexformer, etc.) to form an end-to-end low-light enhancement system. The frequency domain and spatial domain priori modules can be flexibly embedded in different levels of the backbone network, and the fusion module can be used as a unified output end. Through this architecture, the present application realizes the dual optimization of light distribution modeling and structural information preservation, which is significantly better than related methods in terms of visual quality, color restoration and noise suppression.

[0099] Table 1 is the quantitative comparison results of embedding the DIP model of the present application in multiple models on LOL-v1, LOL-v2-Real and LOL-v2-Syn data sets:

[0100] Table 1

[0101]

[0102] The quantitative comparison on the LSRW-Huawei and LSRW-Nikon image data sets is shown in Table 2:

[0103] Table 2

[0104]

[0105] Figure 6 The image restoration results of the present model combined with multiple models according to an embodiment of the present application are shown. As shown in Figure 6 (a) is a to-be-processed light image obtained in a low-light environment, (b), (c), (e), (g), (i) are image processing results obtained by separately using FECNet, MIRNet-v2, Retinexformer, EvLight and GT model for brightness recovery; (d), (f), (h) are target light images obtained by combining the present model (DIP) with MIRNet-v2, Retinexformer, EvLight, respectively. As shown in Figure 6It can be known that by using the DIP model of the application, the brightness and color balance of the whole image to be processed can be improved while the detail structure is maintained.

[0106] Figure 7 A block diagram of an electronic device suitable for implementing the image enhancement method based on dual-domain illumination prior according to an embodiment of the application is shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the function and use range of the embodiments of the application.

[0107] As shown in Figure 7 The electronic device 700 according to an embodiment of the application includes a processor 701 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the application.

[0108] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The processor 701 performs various operations of the method processes according to embodiments of the application by executing programs in the ROM 702 and / or the RAM 703. It should be noted that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method processes according to embodiments of the application by executing programs stored in the one or more memories.

[0109] According to an embodiment of the present application, the electronic device 700 can further include an input / output (I / O) interface 705 that is also connected to the bus 704. The electronic device 700 can further include one or more of the following components connected to the input / output (I / O) interface 705: an input part 706 including a keyboard, a mouse, etc.; an output part 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 708 including a hard disk, etc.; and a communication part 709 including a network interface card such as a LAN card, a modem, etc. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read therefrom is installed into the storage part 708 as necessary.

[0110] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, the embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0111] The embodiments of the present application have been described above. However, these embodiments are merely for the purpose of illustration and not for the purpose of limiting the scope of the present application. Although each of the embodiments has been described above, this does not mean that measures in each of the embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the present application, and such substitutions and modifications are to be included within the scope of the present application.

Claims

1. An image enhancement method based on dual-domain lightness prior, characterized in that, The method comprises: obtaining a to-be-processed illumination image; performing multi-scale enhancement on the to-be-processed illumination image that has been globally enhanced in illumination according to local illumination distribution characteristics of the to-be-processed illumination image, to obtain a first illumination image that has been enhanced in spatial domain, wherein the local illumination distribution characteristics comprise a brightness mean value and a brightness standard deviation; performing enhancement on amplitude information of the to-be-processed illumination image in frequency domain, to obtain a second illumination image that has been enhanced in frequency domain; and fusing the to-be-processed illumination image, the first illumination image and the second illumination image, to obtain a target illumination image that has been enhanced.

2. The method of claim 1, wherein, The method comprises: performing enhancement on the to-be-processed illumination image that has been globally enhanced in illumination according to local illumination distribution characteristics of the to-be-processed illumination image, to obtain a first intermediate illumination image; performing enhancement on the first intermediate illumination image that has been globally enhanced in illumination according to local illumination distribution characteristics of the first intermediate illumination image, to obtain a second intermediate illumination image; wherein the local illumination distribution characteristics of the first intermediate illumination image have a smaller scale than the local illumination distribution characteristics of the to-be-processed illumination image; performing enhancement on the second intermediate illumination image that has been globally enhanced in illumination according to overall illumination distribution characteristics of the second intermediate illumination image, to obtain the first illumination image.

3. The method of claim 2, wherein, The method comprises: performing regional division on the to-be-processed illumination image that has been globally enhanced in illumination, to obtain a plurality of first sub-regions of a first size; performing illumination distribution adjustment on each first sub-region according to local illumination distribution characteristics of a corresponding sub-region of the to-be-processed illumination image, to obtain an adjusted first sub-region; and splicing a plurality of adjusted first sub-regions into the first intermediate illumination image according to spatial positions of the first sub-regions in the to-be-processed illumination image.

4. The method of claim 3, wherein, The method comprises: performing regional division on the first intermediate illumination image that has been globally enhanced in illumination, to obtain a plurality of second sub-regions of a second size, wherein the second size is smaller than the first size; performing illumination distribution adjustment on each second sub-region according to local illumination distribution characteristics of a corresponding sub-region of the first intermediate illumination image, to obtain an adjusted second sub-region; and splicing a plurality of adjusted second sub-regions into the second intermediate illumination image according to spatial positions of the second sub-regions in the to-be-processed illumination image.

5. The method of claim 3, wherein, The method comprises: performing illumination distribution adjustment on each first sub-region according to local illumination distribution characteristics of a corresponding sub-region of the to-be-processed illumination image, to obtain an adjusted first sub-region; and obtaining a local illumination distribution feature of a corresponding sub-region of each first sub-region extracted from the to-be-processed illumination image; correcting the brightness mean value and the brightness standard deviation by using the adaptive learned parameter adjustment coefficient to obtain a corrected brightness mean value and a corrected brightness standard deviation; adjusting the illumination distribution of the first sub-region according to the corrected brightness mean value, the corrected brightness standard deviation, the brightness mean value and the brightness standard deviation to obtain an adjusted first sub-region.

6. The method of claim 1, wherein, The fusing the to-be-processed illumination image, the first illumination image and the second illumination image to obtain an enhanced target illumination image comprises: pixel-by-pixel summing the to-be-processed illumination image, the first illumination image and the second illumination image to obtain an aggregated feature map; generating a first weight, a second weight and a third weight according to the aggregated feature map based on an attention mechanism; fusing the superimposition result of the first weight and the first illumination image, the superimposition result of the second weight and the to-be-processed illumination image, and the superimposition result of the third weight and the second illumination image to obtain a fused feature map; and connecting the fused feature map after feature alignment with the to-be-processed illumination image residual to obtain the enhanced target illumination image.

7. The method of claim 6, wherein, The generating a first weight, a second weight and a third weight according to the aggregated feature map based on an attention mechanism comprises: performing global pooling and average pooling on the aggregated feature map respectively, and splicing the results of global pooling and average pooling to obtain a spliced feature; inputting the spliced feature into three feature extraction branches respectively to output first feature values, second feature values and third feature values for respectively enhancing the features of the first illumination image, the to-be-processed illumination image and the second illumination image; processing the first feature values, the second feature values and the third feature values respectively by using a normalization function to obtain the first weight, the second weight and the third weight.

8. The method of claim 1, wherein, The enhancing the amplitude information of the to-be-processed illumination image in the frequency domain to obtain a second illumination image enhanced in the frequency domain comprises: performing Fourier transform on the to-be-processed illumination image of each color channel to obtain frequency domain information; performing feature enhancement on the amplitude information separated from the frequency domain information to obtain enhanced amplitude information; performing inverse Fourier transform on the enhanced amplitude information and the phase information in the frequency domain information to obtain a third illumination image of each color channel enhanced in the frequency domain; and channel splicing the third illumination image of each color channel to obtain the second illumination image.

9. The method according to any one of claims 1 to 8, characterized in that, The illumination enhancement model is configured to generate the target illumination image according to the to-be-processed illumination image; and the illumination enhancement model is obtained by training in the following manner: inputting a sample illumination image into a to-be-trained illumination enhancement model to generate a target sample illumination image; adjusting the to-be-trained illumination enhancement model according to the target sample illumination image and a label illumination image of a real illumination corresponding to the sample illumination image by using a loss function until a training condition is met to obtain a trained illumination enhancement model; and The loss function comprises: a pixel reconstruction loss representing a pixel difference between a target sample illumination image and the label illumination image, and an amplitude consistency loss representing an amplitude difference between the target sample illumination image and the label illumination image. 10.An electronic device, comprising: one or more processors; memory storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more programs cause the one or more processors to implement the method of any one of claims 1 to 9.

Images