Low-light image enhancement method and device

Abstract

The application provides a low-illumination image enhancement method and device. The method comprises the following steps: acquiring a sequence of T low-illumination images containing a current frame, wherein each low-illumination image is a Bayer image; calculating a global average illumination field of the sequence of low-illumination images based on the sequence of T low-illumination images; calculating a local gradient coupling factor atlas based on the global average illumination field; calculating a detail residual term based on the low-illumination image of the current frame, the global average illumination field and the local gradient coupling factor atlas; and performing image fusion reconstruction on the low-illumination image of the current frame under the constraint of spatiotemporal continuity based on the global average illumination field, the local gradient coupling factor atlas and the detail residual term, and outputting an enhanced image of the current frame. The method and device of the application realize high-quality enhancement of low-illumination images through multi-level decoupling and fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a low-illumination image enhancement method and device. BACKGROUND

[0002] In the field of image processing, image enhancement in low-illumination environments has always been a key challenge. Existing technologies mainly include histogram equalization-based methods, Retinex (retina cortex theory) decomposition methods, and data-driven methods based on deep learning. However, these methods have obvious limitations in practical applications: histogram equalization methods are prone to loss of details and over-enhancement; Retinex methods have difficulty in accurately decomposing illumination and reflection components under complex lighting conditions, and have high computational complexity; data-driven methods based on deep learning rely on a large amount of training data, have limited generalization ability in different scenarios, and have poor model interpretability. In particular, when processing raw Bayer format data from image sensors, existing methods often ignore the wavelength response characteristics specific to the sensor, resulting in color distortion, noise amplification, and detail blurring in the enhanced image. In addition, most existing methods only process single-frame images and fail to effectively utilize temporal information in video sequences, which can easily produce flicker artifacts and motion blur when enhancing consecutive frames. Therefore, existing technologies lack a low-illumination image enhancement method and device that can balance physical interpretability, computational efficiency, and visual naturalness. SUMMARY

[0003] Therefore, it is necessary to provide a low-illumination image enhancement method and device to solve the above technical problems.

[0004] In a first aspect, the present application provides a low-illumination image enhancement method, comprising:

[0005] obtaining a sequence of T consecutive low-illumination images containing a current frame, wherein each low-illumination image in the sequence of T consecutive low-illumination images is a Bayer image;

[0006] based on the sequence of T consecutive low-illumination images, calculating a global average illumination field of the sequence of low-illumination images, wherein the global average illumination field is used to represent the basic light intensity distribution of the scene;

[0007] based on the global average illumination field, calculating a local gradient coupling factor atlas, wherein the local gradient coupling factor atlas is used to represent the cooperativity of gradient direction and geometric structure features in the image;

[0008] based on the low-illumination image of the current frame, the global average illumination field, and the local gradient coupling factor atlas, calculating a detail residual term;

[0009] Based on the global average light field, local gradient coupling factor map and detail residual term, the low-illumination image of the current frame is fused and reconstructed under the spatiotemporal continuity constraint, and the enhanced image of the current frame is output.

[0010] Optionally, based on the T-frame low-illumination image sequence, the global average light field of the low-illumination image sequence is calculated, comprising:

[0011] For each low-illumination image in the T-frame low-illumination image sequence, multi-scale decomposition and reconstruction are performed by the improved Gaussian pyramid respectively, and a single-frame candidate light field corresponding to each low-illumination image is obtained.

[0012] According to the dynamic weight mechanism, the T single-frame candidate light fields are weighted and averaged to obtain the final global average light field.

[0013] Optionally, for each low-illumination image in the T-frame low-illumination image sequence, multi-scale decomposition and reconstruction are performed by the improved Gaussian pyramid respectively, and a single-frame candidate light field corresponding to each low-illumination image is obtained, comprising:

[0014] Multi-scale decomposition operation is performed on each low-illumination image in the T-frame low-illumination image sequence to construct a corresponding Gaussian pyramid, wherein the multi-scale decomposition operation comprises: taking the down-sampled image generated by the previous level as the input image of the current level, iteratively performing Gaussian blur filtering and down-sampling operation to generate a pyramid structure containing multiple resolution levels;

[0015] The fusion reconstruction operation is performed on the Gaussian pyramid constructed for each low-illumination image, wherein the fusion reconstruction operation comprises: starting from the lowest resolution image of the highest layer of the Gaussian pyramid, the up-sampling operation is performed layer by layer and the up-sampling result is fused with the image of the previous layer to reconstruct a single-frame candidate light field with the same size as the original low-illumination image.

[0016] Optionally, according to the dynamic weight mechanism, the T single-frame candidate light fields are weighted and averaged to obtain the final global average light field, comprising:

[0017] Based on the dynamic weight mechanism, the weight coefficient of each single-frame candidate light field in the T single-frame candidate light fields is calculated, and the calculation formula is: , wherein, is the weight coefficient of the tth single-frame candidate light field, is the time correlation decay hyperparameter, is the brightness information of the tth low-illumination image, is the brightness information of the reference low-illumination image;

[0018] Based on the weight coefficient of each single-frame candidate illumination field, the T single-frame candidate illumination fields are weighted and averaged to obtain a final global average illumination field, and a calculation formula thereof is: ; wherein, is a normalization coefficient, ; is a weight coefficient of the tth single-frame candidate illumination field; is the tth single-frame candidate illumination field.

[0019] Optionally, based on the global average illumination field, a local gradient coupling factor graph is calculated, comprising:

[0020] A horizontal direction Sobel operator and a vertical direction Sobel operator are respectively applied to the global average illumination field to generate a corresponding first horizontal direction gradient graph and a first vertical direction gradient graph;

[0021] A Markov random field model is used to optimize the first horizontal direction gradient graph and the first vertical direction gradient graph to obtain a second horizontal direction gradient graph and a second vertical direction gradient graph;

[0022] The high-frequency energy region and the low-frequency energy region of the second horizontal direction gradient graph and the second vertical direction gradient graph are respectively divided, and a non-uniform response mechanism is applied to different regions to obtain a third horizontal direction gradient graph and a third vertical direction gradient graph;

[0023] Based on the third horizontal direction gradient graph and the third vertical direction gradient graph, a local gradient coupling factor graph is calculated; wherein, for each position in the local gradient coupling factor graph , the value is calculated by the following normalization formula: , wherein, and are pixel values of the position in the third horizontal direction gradient graph and the third vertical direction gradient graph, is a predetermined normal number for avoiding division by zero error.

[0024] Optionally, the high-frequency energy region and the low-frequency energy region of the second horizontal direction gradient graph and the second vertical direction gradient graph are respectively divided, and a non-uniform response mechanism is applied to different regions to obtain a third horizontal direction gradient graph and a third vertical direction gradient graph, comprising:

[0025] Based on whether the gradient amplitude in the second horizontal direction gradient graph and the second vertical direction gradient graph is greater than a preset energy threshold, it is divided into a high-frequency energy region or a low-frequency energy region;

[0026] A first gain coefficient for enhancing the contour and texture is applied to the positions divided into the high-frequency energy region, and / or a second gain coefficient for suppressing noise is applied to the positions divided into the low-frequency energy region, to obtain a third horizontal direction gradient map and a third vertical direction gradient map, wherein the first gain coefficient is greater than the second gain coefficient.

[0027] Optionally, based on the low-illumination image of the current frame, the global average illumination field and the local gradient coupling factor map, a detail residual term is calculated, including:

[0028] Based on the low-illumination image of the current frame, the global average illumination field and the local gradient coupling factor map, a residual signal map is calculated;

[0029] The residual signal map is corrected by an optical correction factor to obtain a corrected residual signal map, wherein the optical correction factor is based on a wavelength attenuation curve to deduce the medium transmittance characteristics to compensate for wavelength-related physical attenuation;

[0030] Total variation regularization processing is applied to the corrected residual signal map to maintain texture sharpness and eliminate artifacts to obtain a detail residual term.

[0031] Optionally, based on the global average illumination field, the local gradient coupling factor map and the detail residual term, the low-illumination image of the current frame is image fusion reconstructed under the spatio-temporal continuity constraint, and an enhanced image of the current frame is output, including:

[0032] A weighted fusion model is established, and the input of the model is the global average illumination field, the local gradient coupling factor map, the detail residual term and the low-illumination image of the current frame;

[0033] The spatio-temporal continuity constraint is realized by minimizing a double energy function, and the double energy function includes:

[0034] A first energy function is used to implement a time dimension constraint, which penalizes the difference between adjacent frame illumination components to ensure time smoothness;

[0035] A second energy function is used to implement a spatial dimension constraint, which penalizes gradient jumps in the reconstructed image to ensure boundary continuity;

[0036] An adaptive step adjuster is used to control the optimization process of the double energy function to seek the optimal weight combination of the weighted fusion model within the physically feasible range;

[0037] Based on the optimal weight combination, the input is weighted and fused to output an enhanced image of the current frame.

[0038] Optionally, the weighted fusion model is represented as: wherein, is the reconstructed enhanced image, G is a global average illumination field, G is a local gradient coupling factor map, is a detail residual term, is a low-illumination image of a current frame, is a corresponding weight coefficient.

[0039] In a second aspect, the present application provides a low-illumination image enhancement device, comprising:

[0040] An acquisition module is configured to acquire a sequence of T consecutive low-illumination images containing a current frame, wherein each low-illumination image in the sequence of T low-illumination images is a Bayer image;

[0041] A global average illumination field calculation module is connected to the acquisition module and is configured to calculate a global average illumination field of the sequence of low-illumination images based on the sequence of T low-illumination images, wherein the global average illumination field is used to represent the basic light intensity distribution of a scene;

[0042] A local gradient calculation module is connected to the global average illumination field calculation module and is configured to calculate a local gradient coupling factor map based on the global average illumination field, wherein the local gradient coupling factor map is used to represent the synergy and geometric structure features of the gradient direction in the image;

[0043] A detail residual calculation module is connected to the acquisition module, the global average illumination field calculation module and the local gradient calculation module, and is configured to calculate a detail residual term based on the low-illumination image of the current frame, the global average illumination field and the local gradient coupling factor map;

[0044] A fusion module is connected to the acquisition module, the global average illumination field calculation module, the local gradient calculation module and the detail residual calculation module, and is configured to perform image fusion reconstruction on the low-illumination image of the current frame under the constraint of spatiotemporal continuity based on the global average illumination field, the local gradient coupling factor map and the detail residual term, and output an enhanced image of the current frame.

[0045] The low-illumination image enhancement method and device provided by the present application realize high-quality enhancement of low-illumination images through multi-level decoupling and fusion of the global average illumination field, the local gradient coupling factor map and the detail residual term. Firstly, the improved Gaussian pyramid and the dynamic weight mechanism effectively suppress inter-frame flicker and motion interference, and improve the stability of illumination estimation. Secondly, in combination with the Markov random field and the non-uniform response mechanism, the noise in the flat area is effectively suppressed while the outline and texture are strengthened, and the visual naturalness and detail preservation ability of the image are significantly improved. In addition, the optical correction factor introduced for the Bayer sensor characteristics compensates for the wavelength-dependent attenuation from the physical level, and improves the color restoration accuracy. The method has robustness and computational efficiency in low-illumination environments. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1a A flowchart of a low-illumination image enhancement method provided by an embodiment of the present application is shown in FIG. 1.

[0047] Figure 1b Another flowchart of a low-illumination image enhancement method provided by an embodiment of the present application is shown in FIG. 2.

[0048] Figure 1c Still another flowchart of a low-illumination image enhancement method provided by an embodiment of the present application is shown in FIG. 3.

[0049] Figure 2 A circuit module structure diagram of a low-illumination image enhancement device provided by an embodiment of the present application is shown in FIG. 4.

[0050] Figure 3 An internal structure diagram of a computer device in an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer and more comprehensible, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0052] As shown in FIG. 1, the present application provides a low-illumination image enhancement method, which comprises: Figure 1a

[0053] Step S11: acquiring a continuous T-frame low-illumination image sequence containing a current frame, wherein each frame of the T-frame low-illumination image sequence is a Bayer image;

[0054] The Bayer image is a mode in which red, green and blue filters are arranged above the photodiodes of an image sensor. Its purpose is to allow each pixel point to capture only one color (one of red, green or blue). Through a subsequent "demosaicing" algorithm, the Bayer image can calculate the complete RGB color of each pixel according to the color information of the surrounding pixels, thereby generating a full-color image.

[0055] Step S12: calculating a global average illumination field of the low-illumination image sequence based on the T-frame low-illumination image sequence, wherein the global average illumination field is used to represent the basic light intensity distribution of the scene;

[0056] In an optional embodiment of the present application, as shown in FIG. 2, step S12 specifically comprises: Figure 1b

[0057] ​​Step S121: for each low-illumination image in the T-frame low-illumination image sequence, respectively perform multi-scale decomposition and reconstruction on each low-illumination image by the improved Gaussian pyramid to obtain a single-frame candidate illumination field corresponding to each low-illumination image;

[0058] wherein the improved Gaussian pyramid can be a Laplacian pyramid in the prior art, and a person skilled in the art can select according to actual needs, which is not limited here. For step S121, OpenCV in the prior art, for example, OpenCV 4, can be used, and a person skilled in the art can also select according to actual needs, which is not limited here.

[0059] Optionally, step S121 specifically comprises:

[0060] Step S1211: performing a multi-scale decomposition operation on each low-illumination image in the T-frame low-illumination image sequence to build a corresponding Gaussian pyramid, wherein the multi-scale decomposition operation comprises: taking a down-sampled image generated in a previous level as an input image of a current level, iteratively performing a Gaussian blur filtering and a down-sampling operation to generate a pyramid structure containing multiple resolution levels;

[0061] In addition, the levels of the Gaussian pyramid are usually 3 to 6, and a person skilled in the art can flexibly select according to actual needs, which is not limited here.

[0062] Taking the first low-illumination Bayer image (i.e., the first low-illumination image, with a resolution of 1920x1080) as an example to describe the construction process of the Gaussian pyramid.

[0063] Layer 0 (the bottom layer): input the first low-illumination Bayer image (with a resolution of 1920x1080);

[0064] Layer 1: perform 5x5 Gaussian blur filtering on the first low-illumination Bayer image, and then perform 2 times down-sampling to obtain a 960x540 image;

[0065] Layer 2: continue to perform Gaussian blur filtering and 2 times down-sampling on the 960x540 image obtained in layer 1 to obtain a 480x270 image;

[0066] Layer 3: continue to perform Gaussian blur filtering and 2 times down-sampling on the 480x270 image obtained in layer 2 to obtain a 240x135 image;

[0067] Layer 4: continue to perform Gaussian blur filtering and 2 times down-sampling on the 240x135 image obtained in layer 3 to obtain a 120x68 image;

[0068] Layer 5 (the highest layer): continue to perform Gaussian blur filtering and 2 times down-sampling on the 120x68 image obtained in layer 4 to obtain a 60x34 image;

[0069] The Gaussian blur filter and downsampling operation are performed for 5 times to generate a pyramid structure including 1920x1080, 960x540, 480x270, 240x135, 120x68, and 60x34 resolution levels. The construction process of the Gaussian pyramid of the low-illumination image of other frames is similar, and is not described here.

[0070] Step S1212: The Gaussian pyramid constructed for each low-illumination image is respectively subjected to a fusion reconstruction operation, wherein the fusion reconstruction operation includes: starting from the lowest resolution image of the highest layer of the Gaussian pyramid, a single-frame candidate illumination field with the same size as the original low-illumination image is reconstructed by performing an upsampling operation layer by layer and fusing the upsampling result with the image of the previous layer.

[0071] Continuing the above example, starting from the highest layer (the 5th layer, 60x34) of the Gaussian pyramid, the first upsampling result is obtained by performing bilinear interpolation to the 4th layer resolution (120x68), and the first fusion image is obtained by performing weighted fusion on the first upsampling result and the 4th layer (120x68) image.

[0072] The first fusion image is upsampled to the 3rd layer resolution (240x135) to obtain the second upsampling result, and the second fusion image is obtained by fusing the second upsampling result with the 3rd layer (240x135) image.

[0073] This process is repeated to sequentially fuse with the 2nd layer, the 1st layer, and the 0th layer, and finally a single-frame candidate illumination field with the same resolution (1920x1080) as the 1st low-illumination Bayer image is reconstructed.

[0074] The fusion process of the low-illumination image of other frames is similar, and is not described here.

[0075] It should be noted that the single-frame candidate illumination field directly comes from the data of the Bayer image output by the image sensor, and the data is a pure digital value after analog-to-digital conversion and stripping of physical units. Subsequently, the multi-scale decomposition and reconstruction by the improved Gaussian pyramid are only a series of mathematical transformations without introducing any new units, so the output result "single-frame candidate illumination field" must be a dimensionless numerical matrix.

[0076] Step S122: According to a dynamic weight mechanism, the T single-frame candidate illumination fields are weighted and averaged to obtain a final global average illumination field.

[0077] Optionally, step S122 specifically includes:

[0078] Step S1221: based on the dynamic weight mechanism, the weight coefficient of each single-frame candidate illumination field in the T single-frame candidate illumination fields is calculated, and the calculation formula is: wherein, is the weight coefficient of the tth single-frame candidate illumination field, is the time correlation decay hyperparameter, is the luminance information of the tth low-illumination image, is the luminance information of the reference low-illumination image;

[0079] wherein, the time correlation decay hyperparameter is a system parameter preset through experiments, such as the time correlation decay hyperparameter is 0.1, which controls the decay speed of the influence of the historical frame, The greater the value is, the more severe the penalty for the luminance difference is, and the faster the weight decay is. Those skilled in the art can flexibly set it according to actual needs, which is not limited here.

[0080] Suppose a sequence of 5 consecutive low-illumination images (i.e., 5 consecutive low-illumination Bayer images, T=5), each with a resolution of 1920x1080, arranged in a standard RGGB array.

[0081] Based on the special arrangement structure of the Bayer image, the green channel information is directly extracted through the pixel position index. For the Bayer image arranged in an RGGB array, the distribution of green pixels follows a fixed pattern: in the even rows (0, 2, 4,...), the green pixels are located in the odd columns (1, 3, 5,...); in the odd rows (1, 3, 5,...), the green pixels are located in the even columns (0, 2, 4,...). The coordinate positions of the green pixels can be directly calculated by this rule.

[0082] According to the calculated coordinate positions of the green pixels, the original sensor data at these specific positions are read as the green channel luminance information;

[0083] The arithmetic mean of all the extracted green channel luminance information is calculated to obtain the luminance information of the corresponding low-illumination image. For example, the luminance information of the 0th low-illumination image (which is usually the reference frame), the luminance information of the 1st low-illumination image , the luminance information of the 2nd low-illumination image , the luminance information of the 3rd low-illumination image (the 3rd frame appears a transient light source), and the luminance information of the 4th low-illumination image .

[0084] According to the formula , the weight coefficient of each single-frame candidate illumination field is calculated as follows:

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] .

[0090] The weight coefficient calculation result effectively identifies and suppresses the abnormal frame (the third frame, ), while maintaining the reasonable weight of the normal frame, providing a reliable basis for subsequent weighted fusion, and ensuring the time stability of the global illumination field.

[0091] It should be noted that the green channel brightness information read from the image sensor data is dimensionless, because it is essentially a discrete digital value after analog-to-digital conversion (ADC), representing a relative intensity level rather than an absolute luminosity with physical units. Similarly, the arithmetic mean of all extracted green channel brightness information is obtained, which is the brightness information of the corresponding frame low-illumination image. In addition, since the weight coefficient of each single-frame candidate illumination field is calculated based on the brightness information of the corresponding frame low-illumination image and the time correlation decay hyperparameter, and the brightness information of the corresponding frame low-illumination image and the time correlation decay hyperparameter are dimensionless, the weight coefficient of each single-frame candidate illumination field is also dimensionless.

[0092] Step S1222: Based on the weight coefficient of each single-frame candidate illumination field, the T single-frame candidate illumination fields are weighted and averaged to obtain the final global average illumination field, and the calculation formula is as follows: ; wherein, is a normalization coefficient, ; is the weight coefficient of the tth single-frame candidate illumination field; is the single-frame candidate illumination field of the tth frame.

[0093] Continuing the above example, according to the formula and the weight coefficient of each single-frame candidate illumination field , , , and , the five single-frame candidate illumination fields are weighted and averaged to obtain the global average illumination field .

[0094] The global average illumination field calculated through steps S121 and S122 effectively filters out instantaneous changes in light (such as brightness fluctuations caused by people passing by), forming a stable basic illumination estimate and providing a reliable foundation for subsequent processing. It should be noted that since the weight coefficients of each single-frame candidate illumination field and each single-frame candidate illumination field are dimensionless, the global average illumination field is also dimensionless.

[0095] Step S13: Calculate the local gradient coupling factor map based on the global average illumination field, whereby the local gradient coupling factor map is used to characterize the synergy of gradient directions and geometric structure features in the image;

[0096] Optionally, such as Figure 1c As shown, step S13 specifically includes:

[0097] Step S131: Apply the horizontal Sobel operator and the vertical Sobel operator to the global average illumination field respectively to generate the corresponding first horizontal gradient map and first vertical gradient map.

[0098] Both the horizontal Sobel operator and the vertical Sobel operator can be existing Sobel operators. Those skilled in the art can choose flexibly according to actual needs, and will not be elaborated here.

[0099] Assuming a global average illumination field The matrix is ​​4×4, and its values ​​are as follows: Sobel operator in the horizontal direction for Sobel operator in the vertical direction for .

[0100] Due to the global average illumination field The global average illumination field is a 4×4 matrix, while the horizontal Sobel operator is a 3×3 matrix. Therefore, it is necessary to first calculate the global average illumination field. Expanded to a 6×6 matrix The details are as follows: .

[0101] Using the horizontal Sobel operator For the extended global average illumination field After performing convolution and removing padding, the first horizontal gradient map is obtained, as follows: .

[0102] Using the vertical Sobel operator For the extended global average illumination field After convolution and padding removal, the first vertical direction gradient map is obtained, as follows: .

[0103] It should be noted that if the size of the horizontal Sobel operator and the vertical Sobel operator matches the size of the global average illumination field , there is no need to expand and fill the global average illumination field . When expanding and filling, BORDER_CONSTANT or BORDER_REPLICATE in OpenCV can be used to ensure the rationality of the boundary processing.

[0104] Step S132: The first horizontal direction gradient map and the first vertical direction gradient map are respectively optimized by using a Markov random field model to obtain a second horizontal direction gradient map and a second vertical direction gradient map.

[0105] The Markov random field model is a Markov random field model in the prior art. The Markov random field model can capture the spatial dependence relationship between adjacent elements (such as pixels) in two-dimensional data (such as a two-dimensional matrix) by defining an energy function and a local potential function. That is, the state of each element only depends on its neighborhood (such as the adjacent pixels above, below, left and right), which is highly consistent with the grid structure of the two-dimensional matrix.

[0106] Step S133: The high-frequency energy region and the low-frequency energy region of the second horizontal direction gradient map and the second vertical direction gradient map are respectively divided, and a non-uniform response mechanism is applied to different regions to obtain a third horizontal direction gradient map and a third vertical direction gradient map.

[0107] Optionally, step S133 specifically includes:

[0108] Step S1331: Based on whether the gradient amplitude in the second horizontal direction gradient map and the second vertical direction gradient map is greater than a preset energy threshold, it is divided into a high-frequency energy region or a low-frequency energy region.

[0109] Wherein, for the preset energy threshold, those skilled in the art can flexibly set it according to actual needs, which is not limited here. For example: the preset energy threshold is 10.

[0110] Specifically, it is judged whether the gradient amplitude in the second horizontal direction gradient map and the second vertical direction gradient map is greater than the preset energy threshold; if it is greater than the preset energy threshold, the region corresponding to the gradient amplitude in the second horizontal direction gradient map and the second vertical direction gradient map is a high-frequency energy region; if it is less than or equal to the preset energy threshold, the region corresponding to the gradient amplitude in the second horizontal direction gradient map and the second vertical direction gradient map is a low-frequency energy region.

[0111] Step S1332: applying a first gain coefficient for strengthening the contour and texture to the position divided into the high-frequency energy region, and / or applying a second gain coefficient for suppressing noise to the position divided into the low-frequency energy region, to obtain a third horizontal direction gradient map and a third vertical direction gradient map, wherein the first gain coefficient is greater than the second gain coefficient.

[0112] Wherein, for the first gain coefficient and the second gain coefficient, the person skilled in the art can flexibly set according to actual needs, which is not limited here. For example, the first gain coefficient is 1.5, and the second gain coefficient is 0.8.

[0113] Step S134: calculating a local gradient coupling factor map based on the third horizontal direction gradient map and the third vertical direction gradient map; wherein for each position in the local gradient coupling factor map , the value is calculated by the following normalization formula: , wherein and are pixel values of the position in the third horizontal direction gradient map and the third vertical direction gradient map respectively, is a preset normal number for avoiding division by zero error.

[0114] Wherein, is a dimensionless scalar, and the physical meaning thereof reflects the coupling degree and cooperativity of the horizontal and vertical direction gradients of the image at the position. The absolute value of is closer to 0.5, indicating that the point is more likely to be a corner point or a complex texture region; the absolute value thereof is closer to 0, indicating that the point is more likely to be located in a flat region or an edge with a single direction. In addition, the person skilled in the art can flexibly set , which is not limited here.

[0115] Suppose that the preset energy threshold is 10, the second horizontal direction gradient map is , the second vertical direction gradient map is , the first gain coefficient is 1.5, the second gain coefficient is 0.8, the third horizontal direction gradient map and the third vertical direction gradient map are respectively: and .

[0116] Suppose , the local gradient coupling factor map is calculated based on the above normalization formula. Specifically, the maps and Each corresponding position in Retrieve the scalar value at that location. and Substitute the values ​​into the formula for calculation, and use the result as a local gradient coupling factor map. In position value .

[0117] At position (4,2) (corresponding to the matrix row and column indices, where, , ): For all other positions, because and All are 0.0, therefore, the calculation result is... It is also 0.0. Therefore, the complete local gradient coupling factor map is calculated. for: .

[0118] Step S14: Calculate the detail residuals based on the low-light image of the current frame, the global average illumination field, and the local gradient coupling factor map;

[0119] Optionally, step S14 specifically includes:

[0120] Step S141: Calculate the residual signal spectrum based on the low-light image, global average illumination field, and local gradient coupling factor map of the current frame; wherein, for each location... Calculate the margin signal value at that location. The calculation formula is as follows: ;

[0121] in, For the current frame low-light image at location pixel values, The global average illumination field at location pixel values, For the local gradient coupling factor map at location The value, A preset, adjustable intensity weight coefficient is used to control the contribution of the gradient coupling factor to detail separation; at all locations Constructing the residual signal spectrum .

[0122] Assuming the low-light image of the current frame Global average illumination field Local gradient coupling factor map Intensity weighting coefficient .

[0123] According to the formula , the residual signal atlas is calculated . Specifically, for most positions, Therefore, For example: position (1, 1): ; position (3, 3): ; for position (4, 2), where, : . Calculate all positions, get the residual signal atlas : .

[0124] Step S142: correcting the residual signal atlas by an optical correction factor to obtain a corrected residual signal atlas, wherein the optical correction factor is based on the wavelength attenuation curve to deduce the transmittance characteristics of the medium to compensate for the wavelength-related physical attenuation;

[0125] Wherein, for the optical correction factor, those skilled in the art can set it flexibly according to actual needs, which is not limited here. For example, the optical correction factor is 1.05.

[0126] In an optional embodiment of the present application, the optical correction factor can be set by the following method:

[0127] ① Wavelength attenuation curve analysis

[0128] Different wavelengths of light have selective attenuation when penetrating the filter and the medium. The R, G, B filters in the Bayer array have different transmittance characteristics due to their material and structural differences. In low-illumination environments, the signal is weak, and this wavelength-related attenuation has a more significant impact on image quality. By calibrating the sensor, the typical response characteristics and preliminary correction factors are as shown in Table 1:

[0129] Table 1 Typical response characteristics and preliminary correction factors

[0130]

[0131] ② Medium transmittance deduction and correction factor calculation

[0132] Sensor calibration: under standard light conditions (such as using D65 light source), collect the image of a standard color card (such as X-Rite ColorChecker) with known reflectance; measure the actual response value of each color channel (R, G, B) sensor ; establish a wavelength-response curve database.

[0133] Transmittance modeling and deduction: based on the physical model of Lambert-Beer law: , wherein, is the original light intensity, i.e. the ideal signal that we want to recover, without attenuation; is the original light intensity, i.e. the ideal signal that we want to recover, without attenuation; is the absorption coefficient, is the wavelength, is the medium thickness. From which we can deduce the equivalent transmittance of a specific color channel: Then the optical correction factor of this color channel is the inverse of the transmittance, used to compensate the attenuation: Take the green channel as an example, if the transmittance of the green channel is calculated as , then the optical correction factor of the green channel .

[0134] ③ Determination of the optical correction factor in the luminance domain

[0135] Since the present embodiment processes in the Bayer domain, and usually takes the green channel as the approximation of the luminance information, the optical correction factor in the luminance domain directly adopts the correction factor of the green channel, i.e. the luminance correction factor = the green channel optical correction factor = 1.00.

[0136] ④ Empirical optimization in low-illumination environment

[0137] The above theoretical calculation value (1.00) is calibrated based on standard illumination conditions. Considering that the signal is weaker and the signal-to-noise ratio is lower in low-illumination environment, an empirical gain can be introduced on the basis of the theory to more effectively enhance the details and suppress the noise. Therefore, the final luminance correction factor is adjusted as: the final luminance correction factor = the theoretical luminance correction factor + the empirical increment = 1.00 + 0.08 = 1.08. Wherein, the empirical increment (0.08 in this example) can be flexibly set by the person skilled in the art according to the actual enhancement requirement of the low-illumination scene, which is not limited here.

[0138] Step S143: The modified residual signal spectrum is processed by a total variation regularization method to maintain the texture sharpness and eliminate artifacts, to obtain a detail residual term.

[0139] Wherein, the total variation regularization method is the total variation regularization method in the prior art, which will not be described here.

[0140] It should be noted that the pixel value of the current frame low-illumination image at the position is derived from the data of the Bayer graph after analog-to-digital conversion of the image sensor, and the pixel value is only a discrete digital value representing the relative light intensity, which does not carry any physical unit. In addition, since , and are dimensionless, therefore, is also dimensionless, in other words, the residual signal map is dimensioned.

[0141] Step S15: based on the global average illumination field, the local gradient coupling factor map and the detail residual term, under the spatio-temporal continuity constraint, performing image fusion reconstruction on the low-illumination image of the current frame to output an enhanced image.

[0142] Optionally, step S15 specifically comprises:

[0143] establishing a weighted fusion model, wherein the input of the model is the global average illumination field, the local gradient coupling factor map, the detail residual term and the low-illumination image of the current frame;

[0144] the spatio-temporal continuity constraint is realized by minimizing a double energy function, wherein the double energy function comprises:

[0145] a first energy function for implementing a time dimension constraint, which ensures time smoothness by penalizing the difference of illumination components between adjacent frames;

[0146] a second energy function for implementing a space dimension constraint, which ensures boundary coherence by penalizing gradient jumps in the reconstructed image;

[0147] an adaptive step adjuster is used to control the optimization process of the double energy function to seek the optimal weight combination of the weighted fusion model within a physically feasible range;

[0148] based on the optimal weight combination, the input is weighted and fused to output the enhanced image of the current frame.

[0149] wherein the double energy function can be realized by using the optimization algorithm in the prior art, including but not limited to: energy minimization method based on variational method, graph cut optimization based on Markov random field, alternating direction multiplier method based on convex optimization, or end-to-end optimization network based on deep learning. Those skilled in the art can flexibly choose according to actual needs, which are not limited here.

[0150] wherein the adaptive step adjuster can use the step control strategy in the prior art, including but not limited to: linear search, Armijo criterion, AdaGrad algorithm or adaptive matrix estimation in Adam optimizer. Those skilled in the art can flexibly choose according to actual needs, which are not limited here.

[0151] Optionally, the weighted fusion model is represented as: wherein, is the reconstructed enhanced image, is the global average illumination field, and G is the local gradient coupling factor map. ​a detail residual term, a low-illumination image of the current frame, a corresponding weight coefficient.

[0152] In order to make the skilled in the art can clearly understand the step S15, the following examples are described in detail. Specifically as follows:

[0153] ① Establish a weighted fusion model, which is expressed as wherein, is the corresponding initial weight coefficient respectively , , , For the initial weight coefficient, the skilled in the art can be set flexibly according to the actual situation, not limited here.

[0154] ② Dual energy function constraint optimization

[0155] The first energy function (time constraint): wherein, is the global average illumination field of the t-1 frame; is the global average illumination field of the t frame; is the weight coefficient corresponding to the global average illumination field.

[0156] The second energy function (spatial constraint): wherein, is the gradient operator, which adopts Sobel operator; is the local gradient coupling factor map; is the weight coefficient corresponding to the local gradient coupling factor map; is the detail residual term; is the weight coefficient corresponding to the detail residual term; is the low-illumination image of the current frame; is the weight coefficient corresponding to the low-illumination image of the current frame.

[0157] Total energy function: wherein, the hyperparameter is set as: , . Wherein, the hyperparameter , , the skilled in the art can be set flexibly according to the actual situation, not limited here.

[0158] ③ Adaptive step optimization

[0159] Armijo criterion is used as an adaptive step size adjustment strategy, the criterion ensures that each iteration meets the sufficient descent condition through backtracking linear search, convergence is ensured while computational efficiency is taken into account, and it is particularly suitable for real-time processing requirements of low-illumination image enhancement. Among them, the parameter setting: initial step size: 0.01; Armijo parameter: 0.1 (standard value); Step size scaling factor: 1.2 (expansion) and 0.5 (reduction).

[0160] After optimization iteration, the optimal weight coefficient combination is obtained, assuming 、 、 、 .

[0161] ④Image reconstruction output

[0162] Based on the optimal weight coefficient, weighted fusion is carried out: , to obtain the enhanced image of the current frame.

[0163] The low-illumination image enhancement method provided by the application realizes high-quality enhancement of low-illumination images through multi-level decoupling and fusion of global average illumination field, local gradient coupling factor atlas and detail residual term. First, by improving the Gaussian pyramid and the dynamic weight mechanism, the inter-frame flicker and motion interference are effectively suppressed, and the stability of the illumination estimation is improved; secondly, combined with Markov random field and non-uniform response mechanism, the noise in the flat area is effectively suppressed while the outline and texture are strengthened, and the visual naturalness and detail preservation ability of the image are significantly improved; in addition, the optical correction factor introduced for the Bayer sensor characteristics compensates the wavelength-dependent attenuation from the physical layer, and improves the color restoration accuracy. The method has robustness and computational efficiency in low-illumination environment.

[0164] Based on the same inventive concept, the embodiments of the application also provide a low-illumination image enhancement device for implementing the low-illumination image enhancement method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more low-illumination image enhancement device embodiments provided below can refer to the limitations of the low-illumination image enhancement method described above, and will not be repeated here.

[0165] As shown in Figure 2 , the application provides a low-illumination image enhancement device, comprising: an acquisition module 21, a global average illumination field calculation module 22, a local gradient calculation module 23, a detail residual calculation module 24 and a fusion module 25; wherein

[0166] The acquisition module 21 is used for acquiring a continuous T-frame low-illumination image sequence containing a current frame, wherein each low-illumination image in the T-frame low-illumination image sequence is a Bayer image.

[0167] The global average illumination field calculation module 22 is connected with the acquisition module 21, and is configured to calculate a global average illumination field of the low-illumination image sequence based on the T-frame low-illumination image sequence, wherein the global average illumination field is used to represent the basic light intensity distribution of the scene.

[0168] The local gradient calculation module 23 is connected with the global average illumination field calculation module 22, and is configured to calculate a local gradient coupling factor atlas based on the global average illumination field, wherein the local gradient coupling factor atlas is used to represent the synergy of the gradient direction and the geometric structure feature in the image.

[0169] The detail residual calculation module 24 is connected with the acquisition module 21, the global average illumination field calculation module 22 and the local gradient calculation module 23 respectively, and is configured to calculate a detail residual term based on the low-illumination image of the current frame, the global average illumination field and the local gradient coupling factor atlas.

[0170] The fusion module 25 is connected with the acquisition module 21, the global average illumination field calculation module 22, the local gradient calculation module 23 and the detail residual calculation module 24 respectively, and is configured to perform image fusion reconstruction on the low-illumination image of the current frame under the spatiotemporal continuity constraint based on the global average illumination field, the local gradient coupling factor atlas and the detail residual term, and output an enhanced image of the current frame.

[0171] Optionally, the global average illumination field calculation module 22 is specifically configured to:

[0172] The improved Gaussian pyramid is respectively used for multi-scale decomposition and reconstruction on each low-illumination image in the T-frame low-illumination image sequence, to obtain a single-frame candidate illumination field corresponding to each low-illumination image.

[0173] The T single-frame candidate illumination fields are weighted and averaged according to a dynamic weight mechanism, to obtain a final global average illumination field.

[0174] Optionally, the improved Gaussian pyramid is respectively used for multi-scale decomposition and reconstruction on each low-illumination image in the T-frame low-illumination image sequence, to obtain a single-frame candidate illumination field corresponding to each low-illumination image, including:

[0175] The multi-scale decomposition operation is performed on each low-illumination image in the T-frame low-illumination image sequence to construct a corresponding Gaussian pyramid, wherein the multi-scale decomposition operation includes: taking a down-sampled image generated in a previous level as an input image of a current level, iteratively performing Gaussian blur filtering and down-sampling operation, and generating a pyramid structure including multiple resolution levels.

[0176] The Gaussian pyramid constructed for each frame of low-illumination image is respectively subjected to a fusion reconstruction operation, wherein the fusion reconstruction operation comprises: starting from the lowest resolution image of the highest layer of the Gaussian pyramid, a single-frame candidate illumination field with the same size as the original low-illumination image is reconstructed by performing an up-sampling operation layer by layer and fusing the up-sampling result with the image of the previous layer.

[0177] Optionally, according to the dynamic weight mechanism, the T single-frame candidate illumination fields are weighted and averaged to obtain a final global average illumination field, comprising:

[0178] Based on the dynamic weight mechanism, a weight coefficient of each of the T single-frame candidate illumination fields is calculated, and the calculation formula is: , wherein, is the weight coefficient of the tth single-frame candidate illumination field, is a time correlation decay hyperparameter, is the brightness information of the tth frame of low-illumination image, is the brightness information of the reference frame of low-illumination image;

[0179] Based on the weight coefficient of each single-frame candidate illumination field, the T single-frame candidate illumination fields are weighted and averaged to obtain a final global average illumination field, and the calculation formula is: , wherein, is a normalization coefficient, ; is the weight coefficient of the tth single-frame candidate illumination field; is the tth single-frame candidate illumination field.

[0180] Optionally, the local gradient calculation module 23 is specifically configured to:

[0181] The global average illumination field is respectively subjected to a horizontal Sobel operator and a vertical Sobel operator to generate corresponding first horizontal direction gradient atlas and first vertical direction gradient atlas;

[0182] The first horizontal direction gradient atlas and the first vertical direction gradient atlas are respectively optimized by using a Markov random field model to obtain a second horizontal direction gradient atlas and a second vertical direction gradient atlas;

[0183] The high-frequency energy region and the low-frequency energy region of the second horizontal direction gradient atlas and the second vertical direction gradient atlas are respectively divided, and a non-uniform response mechanism is applied to different regions to obtain a third horizontal direction gradient atlas and a third vertical direction gradient atlas;

[0184] Based on the third horizontal direction gradient atlas and the third vertical direction gradient atlas, a local gradient coupling factor atlas is calculated; wherein for each position in the local gradient coupling factor atlas a value of is calculated by the following normalization formula: wherein, and are pixel values of positions in the third horizontal direction gradient map and the third vertical direction gradient map respectively, is a preset normal number for avoiding division by zero error.

[0185] Optionally, the high-frequency energy region and the low-frequency energy region of the second horizontal direction gradient map and the second vertical direction gradient map are divided respectively, and a non-uniform response mechanism is applied to different regions to obtain the third horizontal direction gradient map and the third vertical direction gradient map, comprising:

[0186] Based on whether each gradient amplitude in the second horizontal direction gradient map and the second vertical direction gradient map is greater than a preset energy threshold, it is divided into a high-frequency energy region or a low-frequency energy region;

[0187] For the positions divided into the high-frequency energy region, a first gain coefficient for strengthening the contour and the texture is applied, and / or for the positions divided into the low-frequency energy region, a second gain coefficient for suppressing noise is applied to obtain the third horizontal direction gradient map and the third vertical direction gradient map, wherein the first gain coefficient is greater than the second gain coefficient.

[0188] Optionally, the detail residual calculation module 24 is specifically configured to:

[0189] Based on the low-illumination image of the current frame, the global average illumination field and the local gradient coupling factor map, a residual signal map is calculated;

[0190] The residual signal map is corrected by an optical correction factor to obtain a corrected residual signal map, wherein the optical correction factor is based on the wavelength attenuation curve to deduce the medium transmittance characteristics to compensate for the wavelength-related physical attenuation;

[0191] Total variation regularization processing is applied to the corrected residual signal map to maintain the texture sharpness and eliminate artifacts to obtain a detail residual term.

[0192] Optionally, the fusion module 25 is specifically configured to:

[0193] A weighted fusion model is established, and the input of the model is the global average illumination field, the local gradient coupling factor map, the detail residual term and the low-illumination image of the current frame;

[0194] The spatiotemporal continuity constraint is realized by minimizing a double energy function, and the double energy function comprises:

[0195] a first energy function for implementing time dimension constraint by penalizing the difference of illumination components between adjacent frames to ensure temporal smoothness;

[0196] a second energy function for implementing spatial dimension constraint by penalizing gradient jumps in the reconstructed image to ensure boundary coherence;

[0197] an adaptive step regulator is adopted to control the optimization process of the dual energy functions to seek the optimal weight combination of the weighted fusion model within the physically feasible range;

[0198] the input is weighted and fused based on the optimal weight combination to output the enhanced image of the current frame.

[0199] Optionally, the weighted fusion model is represented as: wherein, is the reconstructed enhanced image, is a global average illumination field, G is a local gradient coupling factor atlas, is a detail residual term, is a low-light image of the current frame, is a corresponding weight coefficient.

[0200] The low-light image enhancement device provided by the application realizes high-quality enhancement of the low-light image through multi-level decoupling and fusion of the global average illumination field, the local gradient coupling factor atlas and the detail residual term. Firstly, the improved Gaussian pyramid and the dynamic weight mechanism are used to effectively suppress inter-frame flicker and motion interference and improve the stability of illumination estimation. Secondly, the Markov random field and the non-uniform response mechanism are combined to effectively suppress noise in flat areas while strengthening outlines and textures, thereby significantly improving the visual naturalness and detail retention capability of the image. In addition, the optical correction factor introduced for the Bayer sensor characteristics compensates for wavelength-dependent attenuation from the physical layer and improves the color restoration accuracy.

[0201] It should be noted that the plurality in the application includes two or more.

[0202] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0203] Each module in each device in the present application can be realized by software, hardware and a combination thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to the above modules by the processor.

[0204] In one embodiment, a computer device is provided, which can be a server, and the internal structure diagram thereof can be as shown in Figure 3 The computer device includes a processor, a memory and a network interface connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required for or generated by the execution of the above low-illumination image enhancement method. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a low-illumination image enhancement method.

[0205] In one embodiment, a computer device is provided, which can be a terminal, and the internal structure diagram thereof can be as shown in Figure 3As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a low-illumination image enhancement method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad provided on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0206] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0207] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps in each of the above method embodiments.

[0208] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0209] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0210] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0211] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0212] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0213] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A low-light image enhancement method, characterized in that, include: Obtain a sequence of T consecutive low-light images containing the current frame, where each low-light image in the T-frame low-light image sequence is a Bayer image; Based on a T-frame low-light image sequence, the global average illumination field of the low-light image sequence is calculated. This includes: for each low-light image in the T-frame low-light image sequence, multi-scale decomposition and reconstruction are performed using an improved Gaussian pyramid to obtain the single-frame candidate illumination field corresponding to each low-light image; according to a dynamic weighting mechanism, the T single-frame candidate illumination fields are weighted and averaged to calculate the final global average illumination field; wherein, the global average illumination field is used to characterize the basic light intensity distribution of the scene. Based on the global average illumination field, the local gradient coupling factor map is calculated, whereby the local gradient coupling factor map is used to characterize the synergy of gradient directions and geometric structure features in the image. Based on the low-light image, global average illumination field, and local gradient coupling factor spectrum of the current frame, the detail residue term is calculated. This includes: calculating the residual signal spectrum based on the low-light image, global average illumination field, and local gradient coupling factor spectrum of the current frame; correcting the residual signal spectrum using an optical correction factor to obtain a corrected residual signal spectrum, wherein the optical correction factor is derived from the wavelength attenuation curve to infer the medium transmittance characteristics in order to compensate for wavelength-related physical attenuation; and applying total variation regularization to the corrected residual signal spectrum to maintain texture sharpness and eliminate artifacts, thus obtaining the detail residue term. Based on the global average illumination field, local gradient coupling factor map, and detail residue, under the constraint of spatiotemporal continuity, image fusion reconstruction is performed on the low-light image of the current frame, outputting an enhanced image of the current frame. This includes: establishing a weighted fusion model, the input of which is the global average illumination field, local gradient coupling factor map, detail residue, and the low-light image of the current frame; implementing the spatiotemporal continuity constraint by minimizing a bienergy function, which includes: a first energy function for implementing temporal dimension constraints, penalizing differences in illumination components between adjacent frames to ensure temporal smoothness; and a second energy function for implementing spatial dimension constraints, penalizing gradient jumps in the reconstructed image to ensure boundary coherence; using an adaptive step size adjuster to control the optimization process of the bienergy function to find the optimal weight combination of the weighted fusion model within physically feasible limits; and performing weighted fusion on the input based on the optimal weight combination to output the enhanced image of the current frame.

2. The method according to claim 1, characterized in that, For each low-light image in the T-frame low-light image sequence, multi-scale decomposition and reconstruction are performed using an improved Gaussian pyramid to obtain a single-frame candidate illumination field corresponding to each low-light image, including: A multi-scale decomposition operation is performed on each low-light image in the T-frame low-light image sequence to construct the corresponding Gaussian pyramid. The multi-scale decomposition operation includes: using the downsampled image generated in the previous level as the input image of the current level, iteratively performing Gaussian blur filtering and downsampling operations to generate a pyramid structure containing multiple resolution levels. For each Gaussian pyramid constructed for a low-light image, a fusion reconstruction operation is performed. The fusion reconstruction operation includes: starting from the lowest resolution image of the highest layer of the Gaussian pyramid, performing upsampling operations layer by layer and fusing the upsampling results with the image of the previous layer to reconstruct a single-frame candidate illumination field with the same size as the original low-light image.

3. The method according to claim 1 or 2, characterized in that, The step of calculating the final global average illumination field by weighting the T candidate illumination fields of a single frame according to the dynamic weighting mechanism includes: Based on the dynamic weighting mechanism, the weight coefficient of each of the T candidate illumination fields in a single frame is calculated, and the calculation formula is as follows: ,in, Let be the weighting coefficient of the t-th single-frame candidate illumination field. For time-dependent decay hyperparameter, The brightness information of the low-light image in frame t. Brightness information for the low-light image of the reference frame; Based on the weight coefficients of each single-frame candidate illumination field, a weighted average is calculated on the T single-frame candidate illumination fields to obtain the final global average illumination field. The calculation formula is as follows: ;in, The normalization coefficient is... ; is the weighting coefficient for the t-th candidate illumination field in a single frame; Let be the candidate illumination field for the t-th frame.

4. The method according to claim 1, characterized in that, The calculation of the local gradient coupling factor map based on the global average illumination field includes: The Sobel operator in the horizontal direction and the Sobel operator in the vertical direction are applied to the global average illumination field to generate the corresponding first horizontal gradient map and first vertical gradient map. The first horizontal gradient map and the first vertical gradient map are optimized using a Markov random field model to obtain the second horizontal gradient map and the second vertical gradient map. The high-frequency energy region and low-frequency energy region of the second horizontal gradient spectrum and the second vertical gradient spectrum are divided respectively, and a non-uniform response mechanism is applied to different regions to obtain the third horizontal gradient spectrum and the third vertical gradient spectrum. Based on the third horizontal gradient map and the third vertical gradient map, the local gradient coupling factor map is calculated; where, for each position in the local gradient coupling factor map... Its value Calculated using the following normalization formula: ,in, and The positions in the third horizontal gradient map and the third vertical gradient map are respectively. pixel values, This is a preset positive number used to avoid division by zero errors.

5. The method according to claim 4, characterized in that, The process of dividing the second horizontal gradient map and the second vertical gradient map into high-frequency energy regions and low-frequency energy regions, and applying a non-uniform response mechanism to different regions to obtain the third horizontal gradient map and the third vertical gradient map includes: Based on whether the gradient magnitudes in the second horizontal gradient map and the second vertical gradient map are greater than a preset energy threshold, they are divided into high-frequency energy regions or low-frequency energy regions. For locations classified as high-frequency energy regions, a first gain coefficient is applied to enhance the contour and texture, and / or, for locations classified as low-frequency energy regions, a second gain coefficient is applied to suppress noise, resulting in a third horizontal gradient map and a third vertical gradient map, wherein the first gain coefficient is greater than the second gain coefficient.

6. The method according to claim 1, characterized in that, The weighted fusion model is expressed as follows: ,in, For the reconstructed enhanced image, G represents the global average illumination field, and G represents the local gradient coupling factor map. For details remaining, This is the low-light image of the current frame. These are the corresponding weighting coefficients.

7. A low-light image enhancement device, characterized in that, include: The acquisition module is used to acquire a continuous T-frame low-light image sequence containing the current frame, wherein each low-light image in the T-frame low-light image sequence is a Bayer image. The global average illumination field calculation module, connected to the acquisition module, is used to calculate the global average illumination field of a T-frame low-light image sequence. This includes: performing multi-scale decomposition and reconstruction on each low-light image in the T-frame low-light image sequence using an improved Gaussian pyramid to obtain the single-frame candidate illumination field corresponding to each low-light image; and calculating the final global average illumination field by weighted averaging of the T single-frame candidate illumination fields according to a dynamic weighting mechanism. The global average illumination field is used to characterize the basic light intensity distribution of the scene. The local gradient calculation module is connected to the global average illumination field calculation module. It is used to calculate the local gradient coupling factor map based on the global average illumination field. The local gradient coupling factor map is used to characterize the synergy of gradient directions and geometric structure features in the image. The detail retention calculation module, connected to the acquisition module, the global average illumination field calculation module, and the local gradient calculation module, is used to calculate detail retention terms based on the low-light image, global average illumination field, and local gradient coupling factor map of the current frame. This includes: calculating the residual signal map based on the low-light image, global average illumination field, and local gradient coupling factor map of the current frame; correcting the residual signal map using an optical correction factor to obtain a corrected residual signal map, where the optical correction factor is derived from the wavelength attenuation curve to infer the medium's transmittance characteristics, compensating for wavelength-related physical attenuation; and applying total variation regularization to the corrected residual signal map to maintain texture sharpness and eliminate artifacts, thus obtaining the detail retention terms. The fusion module, connected to the acquisition module, the global average illumination field calculation module, the local gradient calculation module, and the detail residue calculation module, is used to perform image fusion reconstruction on the low-light image of the current frame based on the global average illumination field, the local gradient coupling factor map, and the detail residue term, under the constraint of spatiotemporal continuity, and output an enhanced image of the current frame. The fusion module includes: establishing a weighted fusion model, the input of which is the global average illumination field, the local gradient coupling factor map, the detail residue term, and the low-light image of the current frame; implementing the spatiotemporal continuity constraint by minimizing a bienergy function, the bienergy function comprising: a first energy function, used to implement the temporal dimension constraint by penalizing the difference in illumination components between adjacent frames to ensure temporal smoothness; a second energy function, used to implement the spatial dimension constraint by penalizing gradient jumps in the reconstructed image to ensure boundary coherence; using an adaptive step size adjuster to control the optimization process of the bienergy function to find the optimal weight combination of the weighted fusion model within the physically feasible range; and performing weighted fusion on the input based on the optimal weight combination to output the enhanced image of the current frame.

Images