A learnable tensor low-rank enhancement method for low-illumination near-infrared images

Abstract

This invention belongs to the field of deep learning and image processing technology, and discloses a learnable tensor low-rank enhancement method for near-infrared images under low illumination. Based on the estimation results of the local brightness mean and noise variance of the input image, the image is non-uniformly adaptively divided, and a three-dimensional local feature tensor containing spatial height, width, and brightness channels is constructed on each image block using a lightweight convolutional feature extraction network. Subsequently, based on the brightness energy of the image block, the tensor decomposition rank of each block is adaptively determined; factor generation networks and kernel tensor prediction networks are used to generate directional factor matrices and kernel tensors, respectively; threshold suppression is applied to the singular values ​​of the kernel tensor to remove noise subspaces, and scaling of high-frequency directional factors is combined to enhance texture details, effectively restoring degraded edge structures under low illumination conditions. Finally, a weighted fusion method based on minimizing the brightness difference in overlapping regions is used to reconstruct the entire image from the restored image blocks.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and image processing technology, and specifically relates to a learnable tensor low-rank enhancement method for low-light near-infrared images. Background Technology

[0002] Near-infrared (NIR) images are widely used in night vision surveillance, security patrols, vehicle-mounted driver assistance systems, field observation, pedestrian detection, and biometric identification due to their strong anti-fog capabilities and low-light visibility. Compared to visible light imaging, NIR imaging typically relies on a specific wavelength (e.g., 850nm, 940nm) supplementary lighting source and the near-infrared response capability of the image sensor. In low-light environments, the actual acquired NIR images generally suffer from the following problems:

[0003] (1) Low signal-to-noise ratio and complex noise types: Under conditions of extremely weak ambient light or no visible light illumination, the effective number of photons in near-infrared images is significantly reduced. The photosensitive chip operates in high-gain mode, which easily generates strong random noise, including shot noise, readout noise, and dark current noise. These noises exhibit different intensities and statistical characteristics in different spatial regions, resulting in granular backgrounds, submerged textures, and even local stripes and spot artifacts in the image.

[0004] (2) Uneven brightness and local underexposure: Near-infrared illumination is usually provided by point or array infrared light sources, resulting in uneven light distribution. At the same time, the material and surface reflectivity of target objects in the scene vary greatly, which can easily lead to problems such as "bright center and dark periphery" or severe underexposure in local areas and loss of details in dark areas. Uneven brightness distribution not only affects the visual experience of the human eye, but also interferes with subsequent high-level tasks such as detection and recognition.

[0005] (3) Significant degradation of edge and texture information: Under low-light near-infrared imaging conditions, the modulation transfer function decreases due to the optical system and sensor operating in high-gain mode, resulting in blunted edge contours and difficulty in distinguishing fine textures. Especially in low-contrast, complex background scenes, the grayscale difference between the target contour and the background is very limited, and traditional sharpening or contrast enhancement operations are prone to introducing false edges and noise amplification.

[0006] (4) Limitations of traditional enhancement methods: Existing near-infrared or low-light image enhancement methods mainly include several categories such as histogram transformation, Retinex theory, filtering and regularization, and deep learning. Methods based on histogram equalization or adaptive histogram equalization (such as AHE and CLAHE) can enhance the overall contrast to a certain extent, but they cannot distinguish between structural information and noise components, and are prone to amplifying noise at the same time, resulting in a deterioration in image quality. Illumination reflection decomposition methods based on Retinex theory assume that the image can be decomposed into illumination components and reflection components, and improve the brightness of dark areas by constraining illumination smoothness. However, in near-infrared low-light scenes, noise is often mixed in the reflection component, and simply amplifying the reflection component can easily introduce strong noise and pseudo-texture. Denoising and deblurring methods based on filtering or regularization, such as anisotropic diffusion, nonlocal means, BM3D, and low-rank matrix recovery, can reduce noise to a certain extent, but they are usually designed for visible light RGB images and are not adaptable to non-uniform noise and complex degradation models in near-infrared images. Moreover, they often lose edges and details while smoothing noise. In recent years, low-light enhancement and denoising methods based on convolutional neural networks (CNNs), attention mechanisms, or Transformers have attracted attention. By driving the learning of complex nonlinear mapping relationships through large-scale data, they can achieve good enhancement results under specific data distributions. However, these methods generally use the entire image or fixed-size image patches as input, lacking local brightness adaptive modeling mechanisms for low-light near-infrared images. They are prone to producing overly smoothed textures or unstable pseudo-textures in noisy regions. At the same time, most of them do not explicitly introduce low-rank tensor priors, making it difficult to constrain the feature space at the structural level, resulting in the model being sensitive to noise.

[0007] (5) Shortcomings of traditional tensor low-rank methods: Tensor low-rank decomposition (such as Tucker decomposition, CP decomposition, etc.) is used to mine the low-rank structure of high-dimensional data in tasks such as image denoising, super-resolution, and completion, and has certain advantages in structural representation. However, existing methods generally have the following problems: Most of them adopt the tensor decomposition method of global or large area, and apply a uniform low-rank constraint to the entire image or large area, which makes it difficult to perform fine-grained processing on the difference between low light area and normal area; Traditional Tucker decomposition is based on deterministic linear algebra calculation, and the factor matrix and kernel tensor are solved at once by fixed algorithms such as SVD and HOSVD, which do not have the ability to be learned and cannot fully adapt to the distribution of near-infrared low-light noise and structural characteristics through end-to-end training; In complex low-light environment, noise distribution and structural information are often coupled together, and traditional low-rank models have difficulty distinguishing between noise subspace and effective structural subspace, which easily leads to over-smoothing or residual noise.

[0008] In summary, existing technologies either lack the ability to characterize the local illumination and noise structure of near-infrared images under low illumination, or lack the ability to trainable tensor low-rank modeling. There is still a lack of a near-infrared image enhancement method and device that introduces a learnable tensor low-rank enhancement mechanism at the image patch level, which can both reflect the local characteristics of low illumination and adaptively optimize the decomposition and reconstruction process through deep networks. Summary of the Invention

[0009] To address the aforementioned problems, this invention proposes a learnable tensor low-rank enhancement method for near-infrared images under low illumination. First, based on the estimation results of the local brightness mean and noise variance of the input image, the image is non-uniformly adaptively divided. For each image block, a lightweight convolutional feature extraction network is used to construct a three-dimensional local feature tensor containing spatial height, width, and brightness feature channels. Subsequently, based on the brightness energy of the image block, a preset monotonic mapping rule adaptively determines the tensor decomposition rank of each block to control the expressive power of subsequent decompositions. Building upon this, a learnable tensor decomposition module is introduced, generating directional factor matrices and kernel tensors through a factor generation network and a kernel tensor prediction network, respectively, thereby obtaining the structural attenuation components and noise amplification components unique to low-illumination conditions. By applying threshold suppression to the singular values ​​of the kernel tensor to remove the noise subspace, and combining this with scale adjustment of high-frequency directional factors to enhance texture details, the degraded edge structures under low-illumination conditions can be effectively restored. Finally, a weighted fusion method based on minimizing the brightness difference in overlapping regions is used to reconstruct the entire image from the restored image blocks.

[0010] This invention differs from existing technologies in that:

[0011] Existing near-infrared image enhancement methods for low-light conditions mostly employ fixed block division, fixed-rank tensor decomposition, or direct convolution / Transformer for whole-image prediction. These methods generally suffer from problems such as inability to adapt to degradation differences in different low-light regions, sensitivity to noise, and significant block artifacts. This invention differs from existing technologies in the following substantial ways:

[0012] (1) The adaptive block division and adaptive rank prediction mechanism driven by "brightness-noise" is introduced for the first time.

[0013] Existing technologies use only fixed block sizes and fixed decomposition ranks, failing to dynamically adjust modeling complexity based on brightness attenuation and noise amplification in low-light regions. This invention determines the block size and decomposition rank by jointly using local brightness energy and noise estimation results, allowing for a more complete structural representation of extremely low-brightness regions while avoiding over-decomposition of higher-brightness regions. This mechanism is not disclosed in existing technologies, and there is no technical motivation to use brightness energy as the rank selection criterion.

[0014] (2) A joint learnable decomposition structure of factor matrix and kernel tensor is proposed.

[0015] Traditional low-rank tensor methods typically rely on fixed priors or predict only one side of the factor matrix or kernel tensor, lacking a complete learnable decomposition framework. This invention achieves synchronous learnable modeling of the three-directional factor matrix and the three-dimensional kernel tensor through the collaborative design of a factor generation network and a kernel tensor prediction network. This enables explicit capture of the structural attenuation patterns and noise amplification patterns unique to low-light near-infrared images. Existing technologies do not provide such a jointly learnable tensor decomposition structure, nor do they reveal its application value in low-light enhancement.

[0016] (3) Combining kernel tensor noise subspace suppression with brightness consistency fusion to overcome noise amplification and block artifact problems.

[0017] Existing block-based enhancement methods are prone to problems such as drastic noise amplification or discontinuous block boundaries under low-light conditions. This invention effectively removes noise subspaces by performing singular value thresholding suppression in the kernel tensor domain, and introduces a fusion strategy that minimizes brightness differences during the reconstruction stage, ensuring that the enhanced image maintains continuity and consistency in brightness, structure, and texture. Existing technologies do not design "kernel tensor domain denoising" and "brightness consistency fusion" as an integrated framework, and therefore cannot achieve the technical effects of this invention.

[0018] The technical solution of the present invention:

[0019] A learnable tensor low-rank enhancement method for low-light near-infrared images, comprising the following steps:

[0020] Step 1: Based on the mean local brightness and variance local noise of the low-light near-infrared image, perform non-uniform adaptive division of the low-light near-infrared image to obtain multiple image blocks.

[0021] Step 2: Use a lightweight convolutional feature extraction network to perform feature mapping on each image patch, representing it as a three-dimensional local feature tensor containing spatial structure information and brightness-related features;

[0022] Step 3: Construct a rank mapping function, and adaptively determine the tensor decomposition rank of each image block based on the brightness energy of the image block using a monotonic mapping rule;

[0023] Step 4: Generate factor matrices and three-dimensional kernel tensors for each direction using the factor generation network and kernel tensor prediction network in the learnable tensor decomposition module;

[0024] Step 5: Apply singular value thresholding suppression to the three-dimensional kernel tensor to reduce the noise subspace components under low illumination conditions;

[0025] Step 6: Apply high-frequency direction enhancement to the factor matrix to improve the reconstruction capability of edge and texture details;

[0026] Step 7: Based on the brightness consistency constraint, perform weighted fusion on each enhanced image block to generate an enhanced image with uniform overall brightness and clear details.

[0027] Furthermore, the specific implementation process of step 1 is as follows:

[0028] First, analyze the low-light near-infrared image. Perform local statistical estimation, where, H The height of a near-infrared image in low illumination. W The width of the near-infrared image in low illumination; using A sliding window, by controlling the pixels within the sliding window. Calculate the local brightness mean and local noise variance to obtain the brightness mean map and noise variance map of the sliding window. The local brightness mean... and local noise variance The calculation is as follows:

[0029] Local average brightness:

[0030]

[0031] Local noise variance:

[0032]

[0033] in, Represented by pixels Centered Local neighborhood; Represents local neighborhood The inner position is Pixel intensity value;

[0034] Subsequently, based on the local brightness average and local noise variance Joint distribution defines a difficulty index :

[0035]

[0036] in, This represents the maximum image brightness. , These are the weighting coefficients;

[0037] According to the difficulty index The area in question will be a low-light near-infrared image. Divide into image patches of different sizes; threshold , Set to low-light near-infrared images respectively All difficulty indicators The lower and upper quantiles of the numerical distribution, and the threshold. :

[0038] like Then the determination is based on the pixel point. The central sliding window region is a high-difficulty region, and this sliding window region is divided into small image blocks for processing;

[0039] like If the sliding window region is determined to be a medium-difficulty region, then the sliding window region will be divided into blocks using medium image blocks.

[0040] like If the sliding window region is determined to be a low-difficulty region, then the sliding window region will be divided into large image blocks for processing.

[0041] A set of image patches The spatial size of each image patch is represented as... , , The height and width of the image block are specified. The image blocks are divided into small, medium and large image blocks according to their size. The small, medium and large image blocks correspond to the preset image block size, and their side lengths range from 8 to 16 pixels, 16 to 32 pixels and 32 to 64 pixels, respectively.

[0042] Furthermore, the specific implementation process of step 2 is as follows:

[0043] No. The original pixels of each image patch are represented as For each image block Configure a lightweight convolutional feature extraction network with identical structure and shared parameters. Lightweight convolutional feature extraction network It contains 2 to 4 convolutional layers, each using a 3×3 convolutional kernel, and sequentially connects non-linear activation functions (ReLU or LeakyReLU) after the convolution operation; layer normalization is introduced between adjacent convolutional layers; each image patch The final representation is a size of 3D local feature tensor :

[0044]

[0045] in, This represents a lightweight convolutional feature extraction network consisting of 2 to 4 layers of 3×3 convolutional layers, nonlinear activation functions, and layer normalization operations. This represents the number of feature channels after image patch compression, and its value ranges from 8 to 32.

[0046] Furthermore, the specific implementation process of step 3 is as follows:

[0047] 3D local feature tensor Aggregate along the height, width, and channel directions respectively, calculate the corresponding directional energy, and obtain the height direction energy. Energy in the width direction Energy in the direction of the channel The directional energy is defined in the form of mean energy as follows:

[0048]

[0049]

[0050]

[0051] Using a monotonically increasing rank mapping function The decomposition rank in each direction is adaptively determined based on the energy in each direction:

[0052]

[0053] in, The rank mapping function It adopts an exponential form, and its decomposition rank monotonically does not decrease with increasing energy; it is defined as:

[0054]

[0055] in, Represents the rank in any direction , The minimum rank is 4. The maximum rank is 32. To control the constant of the growth rate, This indicates the floor function.

[0056] Furthermore, the specific implementation process of step 4 is as follows:

[0057] (1) For each three-dimensional feature local tensor First, construct three matrix sequences in different directions: height direction sequence. Width direction sequence and brightness channel direction sequence ;

[0058] (2) The height direction sequence, width direction sequence, and brightness channel direction sequence are processed by a factor generation network respectively. Obtain the height orientation factor matrix Width directional factor matrix and brightness channel direction factor matrix Among them, the height orientation factor matrix Describes the structural variation features of image patches in the height direction, and the width direction factor matrix. Describes the structural variation characteristics of image patches in the width direction, and the brightness channel orientation factor matrix. Describes the correlation characteristics of image patches along the brightness feature channel direction; height orientation factor matrix Width directional factor matrix and brightness channel direction factor matrix They are respectively recorded as:

[0059]

[0060]

[0061]

[0062] The FactorNet (·) network consists of a convolutional feature mapping module, a channel attention module, and an axial self-attention mechanism; for any input sequence ,in, Where L is the sequence length and D is the feature dimension, the sequence is first processed by the convolution operator. Map the feature dimension of the input sequence from D to the corresponding decomposition rank dimension. and after nonlinear activation With channel attention module Perform feature recalibration to obtain intermediate feature representations. :

[0063]

[0064] Subsequently, the intermediate features were represented. An axial self-attention mechanism is applied to model long-range dependencies only along a single axis corresponding to the input sequence, while keeping other dimensions unchanged; query vectors, key vectors, and value vectors are generated through linear mapping.

[0065]

[0066] in, For learnable parameters, Embedded dimension for attention;

[0067] Attention weights are calculated based on the similarity between the query vector and the key vector. :

[0068]

[0069] value vector We perform a weighted summation to obtain the axial self-attention output Y:

[0070]

[0071] Axial self-attention output Y and intermediate feature representation The residuals are added together to form the final output of the factor generation network. ;

[0072]

[0073] (3) Convert the three-dimensional feature local tensor The input is fed into a kernel tensor prediction network, where feature compression and global aggregation are used to generate a 3D kernel tensor containing key information. Its size is , is represented as:

[0074]

[0075] Kernel tensor prediction network It includes a dimension transformation module, a convolutional feature extraction module, a residual stacking module, a global feature aggregation module, and a kernel tensor mapping and reconstruction module. The convolutional feature extraction module includes at least a convolutional head and a convolutional tail.

[0076] First, the three-dimensional local feature tensor From the perspective of spatial priority representation Adjusted to channel-priority representation To obtain the initial feature representation ;

[0077] Subsequently, the initial feature representation The overall calculation process for convolutional feature extraction is as follows:

[0078]

[0079]

[0080] in, This represents a convolution head composed of convolution operators, used to extract primary features; This represents a stacked residual module consisting of multiple dense residual blocks, used to model local structure and multi-scale texture information; This represents the convolution tail composed of convolution operators; As an intermediate feature; This indicates a global average pooling operation; This is the global feature vector;

[0081] Finally, the global feature vector Generate a length of [length] using a fully connected mapping. The feature vectors are reconstructed into a form of size . 3D kernel tensor ;in, To pass through the rank mapping function The decomposition rank, determined by the adaptive luminance energy, ranges from 4 to 32.

[0082] Furthermore, the specific implementation process of step 5 is as follows:

[0083] No. The three-dimensional kernel tensor corresponding to each image patch is: , along the three-dimensional kernel tensor Modular expansion, n∈{1,2,3}, yields the kernel tensor expansion matrix. :

[0084]

[0085] in, Modal expansion This represents the dimension of the kernel tensor along the nth modulus, corresponding to the decomposition rank determined by the rank mapping function, when n=1, 2, 3. Corresponding to ;

[0086] Expansion matrix of kernel tensor Perform singular value decomposition:

[0087]

[0088] in, and It is an orthogonal matrix. It is a singular value matrix, which is a diagonal matrix, and its diagonal elements are... It is a singular value;

[0089] Thresholding is applied to outliers; thresholding includes hard thresholding and soft thresholding.

[0090] (1) Hard threshold suppression method;

[0091] Hard threshold suppression defines the singular values ​​after threshold processing. for:

[0092]

[0093] in, For the first The threshold parameter corresponding to the modulus is used to remove the low-energy subspace fraction dominated by noise;

[0094] When singular values Greater than or equal to the threshold parameter When singular values ​​are present, they are retained; when singular values ​​are present, they are retained. Less than threshold parameter Set it to zero at that time;

[0095] (2) Soft threshold suppression method;

[0096] Soft threshold suppression defines the singular values ​​after threshold processing. for:

[0097]

[0098] (3) Three-dimensional kernel tensor reconstruction;

[0099] The singular value matrix after thresholding Substituting into the singular value decomposition and reconstructing, we obtain the noise-suppressed expansion matrix. :

[0100]

[0101] in, This indicates that while preserving the singular value matrix Without changing the structure, change its diagonal elements Replace with singular values ​​after thresholding The resulting diagonal matrix;

[0102] Subsequently, the inverse unrolling operation is used to restore the three-dimensional kernel tensor to a noise subspace-suppressed state. :

[0103]

[0104] in, This indicates the reverse unrolling operation.

[0105] Furthermore, the specific implementation process of step 6 is as follows:

[0106] For the height direction factor matrix respectively Width directional factor matrix and brightness channel direction factor matrix High-frequency enhancement processing is applied using a high-pass filter operator. The high-frequency components are extracted to obtain the enhanced height orientation factor matrix. Enhanced width direction factor matrix and the enhanced luminance channel orientation factor matrix ;

[0107]

[0108]

[0109]

[0110] in, , , They may be the same or different, and both can be taken from the scale factor range, with a value range of 0.1 to 0.5;

[0111] High-pass filter operator Using a one-dimensional difference form, for any directional factor matrix Its definition is:

[0112] , ;

[0113] ,

[0114] in, Representation of factor matrix The Middle line, number The element values ​​of the column, where This is a direction index used to indicate the position of the factor matrix in the corresponding space or channel direction; This is a rank-dimensional index used to represent different low-rank basis vector components; Representation of factor matrix In position The high-frequency components at a given location are obtained by calculating the first-order difference between elements at adjacent locations.

[0115] Subsequently, the kernel tensor, after noise subspace suppression, is used. With the enhanced height orientation factor matrix Enhanced width direction factor matrix and the enhanced luminance channel orientation factor matrix Tensor reconstruction is performed to obtain the enhanced local feature tensor. The reconstruction process is represented as:

[0116]

[0117] in, For the first Modular tensor-matrix multiplication.

[0118] Furthermore, the specific implementation process of step 7 is as follows:

[0119] During the fusion stage, for low-light near-infrared images The same pixel in Due to the adoption Sliding window processing will result in multiple block-level enhancement results at the pixel level. The corresponding candidate pixel values ​​are given at this location. ;

[0120] Calculate the brightness difference :

[0121]

[0122] in, It is a pixel. The reference brightness value is the average of all candidate values;

[0123] Final merged pixel values It is obtained through a weighted average method, and its calculation formula is as follows:

[0124]

[0125] Among them, weight The principle that the smaller the brightness difference, the greater the weight is, is defined as:

[0126]

[0127] in, To prevent positive constants with a denominator of zero, its value is 10. -3 .

[0128] The beneficial effects of this invention are:

[0129] (1) Realize adaptive structural modeling and automatic optimization of decomposition rank under weak illumination conditions.

[0130] This invention enables tensor low-rank decomposition to automatically adjust the model capacity for different low-light regions by using brightness-noise driven adaptive image block partitioning and an adaptive decomposition rank determination mechanism based on brightness energy. This avoids the under-decomposition and over-decomposition problems caused by fixed rank, thereby obtaining a more accurate local structure representation.

[0131] (2) A learnable tensor low-rank enhancement framework is proposed to effectively recover the structure and texture under low illumination.

[0132] This invention combines learnable tensor decomposition, kernel tensor noise subspace suppression, and high-frequency factor enhancement techniques to simultaneously model the structural attenuation and noise amplification characteristics in low-light images, significantly improving the recovery quality of high-frequency details such as edges and textures in low-light regions.

[0133] (3) A brightness consistency fusion strategy is adopted to significantly reduce block artifacts and improve the overall visual consistency of the image.

[0134] This invention achieves image block fusion based on a weighted strategy of brightness difference in overlapping regions, effectively eliminating common block boundary artifacts and brightness discontinuities, resulting in a final enhanced image with higher consistency and visual quality in terms of global brightness, structure, and texture.

[0135] (4) The overall method can learn end-to-end, which is significantly better than traditional reinforcement methods based on fixed priors.

[0136] This invention integrates block segmentation, adaptive rank prediction, tensor decomposition, kernel tensor denoising, and fusion reconstruction into a learnable framework, achieving end-to-end optimization and exhibiting superior brightness enhancement, structure fidelity, and noise suppression performance under various low-light conditions. Attached Figure Description

[0137] Figure 1 This is a diagram illustrating the method steps of an embodiment of the present invention;

[0138] Figure 2 This is a network structure diagram of an embodiment of the present invention;

[0139] Figure 3 This is a schematic diagram of the factor generation network according to an embodiment of the present invention;

[0140] Figure 4 This is a schematic diagram of the kernel tensor prediction network according to an embodiment of the present invention. Detailed Implementation

[0141] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0142] like Figure 1 As shown, this invention provides a learnable tensor low-rank enhancement method for low-light near-infrared images.

[0143] like Figure 2 As shown, the specific implementation process of a learnable tensor low-rank enhancement method for low-light near-infrared images is as follows:

[0144] Step 1: Based on the mean local brightness and variance local noise of the low-light near-infrared image, perform non-uniform adaptive division of the low-light near-infrared image to obtain multiple image blocks.

[0145] Based on the spatial distribution characteristics of near-infrared images under low illumination, different block division granularities are used for different regions;

[0146] First, analyze the low-light near-infrared image. Perform local statistical estimation, where, H The height of a near-infrared image in low illumination. W The width of the near-infrared image in low illumination; using A sliding window, by controlling the pixels within the sliding window. Calculate the local brightness mean and local noise variance to obtain the brightness mean map and noise variance map of the sliding window.

[0147] Subsequently, a difficulty index was defined based on the joint distribution of local brightness mean and local noise variance. Then, based on the interval where the difficulty index was located, the near-infrared images with low illumination were... Divide the image into different size levels;

[0148] Let the input near-infrared image be , H For height, W For width.

[0149] use A window (e.g., w = 7 or 9) for each pixel. calculate:

[0150] Local average brightness:

[0151]

[0152] Local noise variance:

[0153]

[0154] in, Represented by pixels Centered Local neighborhood; Represents local neighborhood The inner position is Pixel intensity value;

[0155] Based on local brightness average and local noise variance Joint distribution defines a difficulty index :

[0156]

[0157] in, This represents the maximum image brightness. , These are the weighting coefficients;

[0158] According to the difficulty index The area in question will be a low-light near-infrared image. Divide into image patches of different sizes; threshold , Set to low-light near-infrared images respectively All difficulty indicators The lower and upper quantiles of the numerical distribution, and the threshold. :

[0159] like Then the determination is based on the pixel point. The central sliding window area is a high-difficulty area, and this sliding window area is divided into small image blocks (such as 16×16).

[0160] like If the sliding window region is determined to be a medium difficulty region, then the sliding window region is divided into blocks using medium image blocks (such as 24×24 or 32×32).

[0161] like If the sliding window region is determined to be a low-difficulty region, then the sliding window region is divided into large image blocks (such as 48×48 or 64×64).

[0162] A set of image patches The spatial size of each image patch is represented as... , , This represents the height and width of the image block.

[0163] Step 2: Use a lightweight convolutional feature extraction network to perform feature mapping on each image patch, representing it as a three-dimensional local feature tensor containing spatial structure information and brightness-related features;

[0164] No. The original pixels of each image patch are represented as For each image block Configure a lightweight convolutional feature extraction network with identical structure and shared parameters. Lightweight convolutional feature extraction network It contains 2 to 4 convolutional layers, each using a 3×3 convolutional kernel. After the convolution operation, a non-linear activation function (ReLU or LeakyReLU) is sequentially applied. Layer normalization is introduced between adjacent convolutional layers to enhance network stability. Each image patch... The final representation is a size of 3D local feature tensor :

[0165]

[0166] in, This represents a lightweight convolutional feature extraction network consisting of 2 to 4 layers of 3×3 convolutional layers, non-linear activation functions, and layer normalization operations. For example: Layer 1: 3×3 convolutional layer, ReLU activated; Layer 2: 3×3 convolutional layer, ReLU activated. This represents the number of feature channels after image patch compression, and its value ranges from 8 to 32.

[0167] Step 3: Construct a rank mapping function, and adaptively determine the tensor decomposition rank of each image block based on the brightness energy of the image block using a monotonic mapping rule;

[0168] In low-light near-infrared images, the structural complexity and noise levels vary significantly across different regions. Applying a uniform tensor rank to all regions can easily lead to overfitting or underfitting. This invention correlates the rank with the brightness energy of image blocks, assigning a higher rank to blocks with higher brightness and richer structural information, and a lower rank to blocks with lower brightness and predominant noise, thereby reducing the dimensionality of the noise subspace.

[0169] 3D local feature tensor Aggregate along the height, width, and channel directions respectively, calculate the corresponding directional energy, and obtain the height direction energy. Energy in the width direction Energy in the direction of the channel The directional energy is defined in the form of mean energy as follows:

[0170]

[0171]

[0172]

[0173] Using a monotonically increasing rank mapping function The decomposition rank in each direction is adaptively determined based on the energy in each direction:

[0174]

[0175] in, The rank mapping function It adopts an exponential form, and its decomposition rank monotonically does not decrease with increasing energy; it is defined as:

[0176]

[0177] in, Represents the rank in any direction , The minimum rank is 4. The maximum rank is 32. To control the constant of the growth rate, This indicates the floor function.

[0178] Step 4: Generate factor matrices and three-dimensional kernel tensors for each direction using the factor generation network and kernel tensor prediction network in the learnable tensor decomposition module;

[0179] (1) For each three-dimensional feature local tensor First, construct three matrix sequences in different directions: height direction sequence. Width direction sequence and brightness channel direction sequence ;

[0180] (2) The height direction sequence, width direction sequence, and brightness channel direction sequence are processed by a factor generation network respectively. Obtain the height orientation factor matrix Width directional factor matrix and brightness channel direction factor matrix Among them, the height orientation factor matrix Describes the structural variation features of image patches in the height direction, and the width direction factor matrix. Describes the structural variation characteristics of image patches in the width direction, and the brightness channel orientation factor matrix. Describes the correlation characteristics of image patches along the brightness feature channel direction; height orientation factor matrix Width directional factor matrix and brightness channel direction factor matrix They are respectively recorded as:

[0181]

[0182]

[0183]

[0184] The FactorNet (·) network consists of a convolutional feature mapping module, a channel attention module, and an axial self-attention mechanism; for any input sequence ,in, Where L is the sequence length and D is the feature dimension, the sequence is first processed by the convolution operator. Map the feature dimension of the input sequence from D to the corresponding decomposition rank dimension. and after nonlinear activation With channel attention module Perform feature recalibration to obtain intermediate feature representations. :

[0185]

[0186] Subsequently, the intermediate features were represented. An axial self-attention mechanism is applied to model long-range dependencies only along a single axis corresponding to the input sequence, while keeping other dimensions unchanged; query vectors, key vectors, and value vectors are generated through linear mapping.

[0187]

[0188] in, For learnable parameters, Embedded dimension for attention;

[0189] Attention weights are calculated based on the similarity between the query vector and the key vector. :

[0190]

[0191] value vector We perform a weighted summation to obtain the axial self-attention output Y:

[0192]

[0193] Axial self-attention output Y and intermediate feature representation The residuals are added together to form the final output of the factor generation network. ;

[0194]

[0195] (3) Convert the three-dimensional feature local tensor The input is fed into a kernel tensor prediction network (GEncoderCNN), where feature compression and global aggregation are used to generate a 3D kernel tensor containing key information. Its size is , is represented as:

[0196]

[0197] Kernel tensor prediction network It includes a dimensionality transformation module, a convolutional feature extraction module, a residual stacking module, a global feature aggregation module, and a kernel tensor mapping and reconstruction module;

[0198] First, the three-dimensional local feature tensor From the perspective of spatial priority representation Adjusted to channel-priority representation To obtain the initial feature representation ;

[0199] Subsequently, the initial feature representation The overall calculation process for convolutional feature extraction is as follows:

[0200]

[0201]

[0202] in, This represents a convolution head composed of convolution operators, used to extract primary features; This represents a stacked residual module consisting of multiple dense residual blocks, used to model local structure and multi-scale texture information; This represents the convolution tail composed of convolution operators; As an intermediate feature; This indicates a global average pooling operation; This is the global feature vector;

[0203] Finally, the global feature vector Generate a length of [length] using a fully connected mapping. The feature vectors are reconstructed into a form of size . 3D kernel tensor ;in, To pass through the rank mapping function The decomposition rank, determined by the adaptive luminance energy, ranges from 4 to 32.

[0204] Step 5: Apply singular value thresholding suppression to the three-dimensional kernel tensor to reduce the noise subspace components under low illumination conditions;

[0205] No. The three-dimensional kernel tensor corresponding to each image patch is: , along the three-dimensional kernel tensor Modular expansion, n∈{1,2,3}, yields the kernel tensor expansion matrix. :

[0206]

[0207] in, Modal expansion This represents the dimension of the kernel tensor along the nth modulus, corresponding to the decomposition rank determined by the rank mapping function, when n=1, 2, 3. Corresponding to ;

[0208] Expansion matrix of kernel tensor Perform singular value decomposition:

[0209]

[0210] in, and It is an orthogonal matrix. It is a singular value matrix, which is a diagonal matrix, and its diagonal elements are... It is a singular value;

[0211] Thresholding is applied to outliers; thresholding includes hard thresholding and soft thresholding.

[0212] (1) Hard threshold suppression method;

[0213] Hard threshold suppression defines the singular values ​​after threshold processing. for:

[0214]

[0215] in, For the first The threshold parameter corresponding to the modulus is used to remove the low-energy subspace fraction dominated by noise;

[0216] When singular values Greater than or equal to the threshold parameter When singular values ​​are present, they are retained; when singular values ​​are present, they are retained. Less than threshold parameter Set it to zero at that time;

[0217] (2) Soft threshold suppression method;

[0218] Soft threshold suppression defines the singular values ​​after threshold processing. for:

[0219]

[0220] (3) Three-dimensional kernel tensor reconstruction;

[0221] The singular value matrix after thresholding Substituting into the singular value decomposition and reconstructing, we obtain the noise-suppressed expansion matrix. :

[0222]

[0223] in, This indicates that while preserving the singular value matrix Without changing the structure, change its diagonal elements Replace with singular values ​​after thresholding The resulting diagonal matrix;

[0224] Subsequently, the inverse unrolling operation is used to restore the three-dimensional kernel tensor to a noise subspace-suppressed state. :

[0225]

[0226] in, This indicates the reverse unrolling operation.

[0227] Step 6: Apply high-frequency direction enhancement to the factor matrix to improve the reconstruction capability of edge and texture details;

[0228] For the height direction factor matrix respectively Width directional factor matrix and brightness channel direction factor matrix High-frequency enhancement processing is applied using a high-pass filter operator. The high-frequency components are extracted to obtain the enhanced height orientation factor matrix. Enhanced width direction factor matrix and the enhanced luminance channel orientation factor matrix ;

[0229]

[0230]

[0231]

[0232] in, , , They may be the same or different, and both can be taken from the scale factor range, with a value range of 0.1 to 0.5;

[0233] High-pass filter operator Using a one-dimensional difference form, for any directional factor matrix Its definition is:

[0234] , ;

[0235] ,

[0236] in, Representation of factor matrix The Middle line, number The element values ​​of the column, where This is a direction index used to indicate the position of the factor matrix in the corresponding space or channel direction; This is a rank-dimensional index used to represent different low-rank basis vector components; Representation of factor matrix In position The high-frequency components at a given location are obtained by calculating the first-order difference between elements at adjacent locations.

[0237] Subsequently, the kernel tensor, after noise subspace suppression, is used. With the enhanced height orientation factor matrix Enhanced width direction factor matrix and the enhanced luminance channel orientation factor matrix Tensor reconstruction is performed to obtain the enhanced local feature tensor. The reconstruction process is represented as:

[0238]

[0239] in, For the first Modular tensor-matrix multiplication.

[0240] Step 7: Based on the brightness consistency constraint, perform weighted fusion on each enhanced image block to generate an enhanced image with uniform overall brightness and clear details;

[0241] During the fusion stage, for low-light near-infrared images The same pixel in Due to the adoption Sliding window processing will result in multiple block-level enhancement results at the pixel level. The corresponding candidate pixel values ​​are given at this location. ;

[0242] Calculate the brightness difference :

[0243]

[0244] in, It is a pixel. The reference brightness value is the average of all candidate values;

[0245] Final merged pixel values It is obtained through a weighted average method, and its calculation formula is as follows:

[0246]

[0247] Among them, weight The principle that the smaller the brightness difference, the greater the weight is, is defined as:

[0248]

[0249] in, To prevent positive constants with a denominator of zero, its value is 10. -3 .

[0250] By employing the aforementioned fusion strategy, enhanced near-infrared images with uniform brightness, clear details, and natural transitions between blocks can be obtained across the entire image range.

Claims

1. A learnable tensor low-rank enhancement method for low-light near-infrared images, characterized in that, The steps are as follows: Step 1: Based on the mean local brightness and variance local noise of the low-light near-infrared image, perform non-uniform adaptive division of the low-light near-infrared image to obtain multiple image blocks. The specific implementation process of step 1 is as follows: First, analyze the low-light near-infrared image. Perform local statistical estimation, where, H The height of a near-infrared image in low illumination. W The width of the near-infrared image in low illumination; using A sliding window, by controlling the pixels within the sliding window. Calculate the local brightness mean and local noise variance to obtain the brightness mean map and noise variance map of the sliding window. The local brightness mean... and local noise variance The calculation is as follows: Local average brightness: Local noise variance: in, Represented by pixels Centered Local neighborhood; Represents local neighborhood The inner position is Pixel intensity value; Subsequently, based on the local brightness average and local noise variance Joint distribution defines a difficulty index : in, This represents the maximum image brightness. , These are the weighting coefficients; According to the difficulty index The area in question will be a low-light near-infrared image. Divide into image patches of different sizes; threshold , Set to low-light near-infrared images respectively All difficulty indicators The lower and upper quantiles of the numerical distribution, and the threshold. : like Then the determination is based on the pixel point. The central sliding window region is a high-difficulty region, and this sliding window region is divided into small image blocks for processing; like If the sliding window region is determined to be a medium-difficulty region, then the sliding window region will be divided into blocks using medium image blocks. like If the sliding window region is determined to be a low-difficulty region, then the sliding window region will be divided into large image blocks for processing. A set of image patches The spatial size of each image patch is represented as... , , The height and width of the image block; the image blocks are divided into small image blocks, medium image blocks and large image blocks according to their size. The small image blocks, medium image blocks and large image blocks correspond to the preset image block size, and their side length values ​​range from 8 to 16 pixels, 16 to 32 pixels and 32 to 64 pixels, respectively. Step 2: Use a lightweight convolutional feature extraction network to perform feature mapping on each image patch, representing it as a three-dimensional local feature tensor containing spatial structure information and brightness-related features; Step 3: Construct a rank mapping function, and adaptively determine the tensor decomposition rank of each image block based on the brightness energy of the image block using a monotonic mapping rule; Step 4: Generate factor matrices and three-dimensional kernel tensors for each direction using the factor generation network and kernel tensor prediction network in the learnable tensor decomposition module; Step 5: Apply singular value thresholding suppression to the three-dimensional kernel tensor to reduce the noise subspace components under low illumination conditions; Step 6: Apply high-frequency direction enhancement to the factor matrix to improve the reconstruction capability of edge and texture details; Step 7: Based on the brightness consistency constraint, perform weighted fusion on each enhanced image block to generate an enhanced image with uniform overall brightness and clear details.

2. The learnable tensor low-rank enhancement method for low-light near-infrared images according to claim 1, characterized in that, The specific implementation process of step 2 is as follows: No. The original pixels of each image patch are represented as For each image block Configure a lightweight convolutional feature extraction network with identical structure and shared parameters. Lightweight convolutional feature extraction network It contains 2 to 4 convolutional layers, each using a 3×3 convolutional kernel, and sequentially connects non-linear activation functions (ReLU or LeakyReLU) after the convolution operation; layer normalization is introduced between adjacent convolutional layers; each image patch The final representation is a size of 3D local feature tensor : in, This represents a lightweight convolutional feature extraction network consisting of 2 to 4 layers of 3×3 convolutional layers, nonlinear activation functions, and layer normalization operations. This represents the number of feature channels after image patch compression, and its value ranges from 8 to 32.

3. The learnable tensor low-rank enhancement method for low-light near-infrared images according to claim 2, characterized in that, The specific implementation process of step 3 is as follows: 3D local feature tensor Aggregate along the height, width, and channel directions respectively, calculate the corresponding directional energy, and obtain the height direction energy. Energy in the width direction Energy in the direction of the channel The directional energy is defined in the form of mean energy as follows: Using a monotonically increasing rank mapping function The decomposition rank in each direction is adaptively determined based on the energy in each direction: in, The rank mapping function It adopts an exponential form, and its decomposition rank monotonically does not decrease with increasing energy; it is defined as: in, Represents the rank in any direction , The minimum rank is 4. The maximum rank is 32. To control the constant of the growth rate, This indicates the floor function.

4. The learnable tensor low-rank enhancement method for low-light near-infrared images according to claim 3, characterized in that, The specific implementation process of step 4 is as follows: (1) For each three-dimensional feature local tensor First, construct three matrix sequences in different directions: height direction sequence. Width direction sequence and brightness channel direction sequence ; (2) The height direction sequence, width direction sequence, and brightness channel direction sequence are processed by a factor generation network respectively. Obtain the height orientation factor matrix Width directional factor matrix and brightness channel direction factor matrix Among them, the height orientation factor matrix Describes the structural variation features of image patches in the height direction, and the width direction factor matrix. Describes the structural variation characteristics of image patches in the width direction, and the brightness channel orientation factor matrix. Describes the correlation characteristics of image patches along the brightness feature channel direction; height orientation factor matrix Width directional factor matrix and brightness channel direction factor matrix They are respectively recorded as: The FactorNet (·) network consists of a convolutional feature mapping module, a channel attention module, and an axial self-attention mechanism; for any input sequence ,in, Where L is the sequence length and D is the feature dimension, the sequence is first processed by the convolution operator. Map the feature dimension of the input sequence from D to the corresponding decomposition rank dimension. and after nonlinear activation With channel attention module Perform feature recalibration to obtain intermediate feature representations. : Subsequently, the intermediate features were represented. An axial self-attention mechanism is applied to model long-range dependencies only along a single axis corresponding to the input sequence, while keeping other dimensions unchanged; query vectors, key vectors, and value vectors are generated through linear mapping. in, For learnable parameters, Embedded dimension for attention; Attention weights are calculated based on the similarity between the query vector and the key vector. : value vector We perform a weighted summation to obtain the axial self-attention output Y: Axial self-attention output Y and intermediate feature representation The residuals are added together to form the final output of the factor generation network. ; (3) Convert the three-dimensional feature local tensor The input is fed into a kernel tensor prediction network, where feature compression and global aggregation are used to generate a 3D kernel tensor containing key information. Its size is , is represented as: Kernel tensor prediction network It includes a dimension transformation module, a convolutional feature extraction module, a residual stacking module, a global feature aggregation module, and a kernel tensor mapping and reconstruction module. The convolutional feature extraction module includes at least a convolutional head and a convolutional tail. First, the three-dimensional local feature tensor From the perspective of spatial priority representation Adjusted to channel-priority representation To obtain the initial feature representation ; Subsequently, the initial feature representation The overall calculation process for convolutional feature extraction is as follows: in, This represents a convolution head composed of convolution operators, used to extract primary features; This represents a stacked residual module consisting of multiple dense residual blocks, used to model local structure and multi-scale texture information; This represents the convolution tail composed of convolution operators; As an intermediate feature; This indicates a global average pooling operation; This is the global feature vector; Finally, the global feature vector Generate a length of [length] using a fully connected mapping. The feature vectors are reconstructed into a form of size . 3D kernel tensor ;in, To pass through the rank mapping function The decomposition rank, determined by the adaptive luminance energy, ranges from 4 to 32.

5. The learnable tensor low-rank enhancement method for low-light near-infrared images according to claim 4, characterized in that, The specific implementation process of step 5 is as follows: No. The three-dimensional kernel tensor corresponding to each image patch is: , along the three-dimensional kernel tensor Modular expansion, n∈{1,2,3}, yields the kernel tensor expansion matrix. : in, Modal expansion This represents the dimension of the kernel tensor in the nth modulus direction, corresponding to the decomposition rank determined by the rank mapping function, when n=1, 2, 3. Corresponding to ; Expansion matrix of kernel tensor Perform singular value decomposition: in, and It is an orthogonal matrix. It is a singular value matrix, which is a diagonal matrix, and its diagonal elements are... It is a singular value; Thresholding is applied to outliers; thresholding includes hard thresholding and soft thresholding. (1) Hard threshold suppression method; Hard threshold suppression defines the singular values ​​after threshold processing. for: in, For the first The threshold parameter corresponding to the modulus is used to remove low-energy subspace fractions dominated by noise; When singular values Greater than or equal to the threshold parameter When singular values ​​are present, they are retained; when singular values ​​are present, they are retained. Less than threshold parameter Set it to zero at that time; (2) Soft threshold suppression method; Soft threshold suppression defines the singular values ​​after threshold processing. for: (3) Three-dimensional kernel tensor reconstruction; The singular value matrix after thresholding Substituting into the singular value decomposition and reconstructing, we obtain the noise-suppressed expansion matrix. : in, This indicates that while preserving the singular value matrix Without changing the structure, change its diagonal elements Replace with singular values ​​after thresholding The resulting diagonal matrix; Subsequently, the inverse unrolling operation is used to restore the three-dimensional kernel tensor to a noise subspace-suppressed state. : in, This indicates the reverse unrolling operation.

6. The learnable tensor low-rank enhancement method for low-light near-infrared images according to claim 5, characterized in that, The specific implementation process of step 6 is as follows: For the height direction factor matrix respectively Width directional factor matrix and brightness channel direction factor matrix High-frequency enhancement processing is applied using a high-pass filter operator. The high-frequency components are extracted to obtain the enhanced height orientation factor matrix. Enhanced width direction factor matrix and the enhanced luminance channel orientation factor matrix ; in, , , They may be the same or different, and both can be taken from the scale factor range, with values ​​ranging from 0.1 to 0.5; The high-pass filter operator Using a one-dimensional difference form, for any directional factor matrix Its definition is: ， ； ， in, Representation of factor matrix The Middle line, number The element values ​​of the column, where This is a direction index used to indicate the position of the factor matrix in the corresponding space or channel direction; This is a rank-dimensional index used to represent different low-rank basis vector components; Representation of factor matrix In position The high-frequency components at a given location are obtained by calculating the first-order difference between elements at adjacent locations. Subsequently, the kernel tensor, after noise subspace suppression, is used. With the enhanced height orientation factor matrix Enhanced width direction factor matrix and the enhanced luminance channel orientation factor matrix Tensor reconstruction is performed to obtain the enhanced local feature tensor. The reconstruction process is represented as: in, For the first Modular tensor-matrix multiplication.

7. The learnable tensor low-rank enhancement method for low-light near-infrared images according to claim 6, characterized in that, The specific implementation process of step 7 is as follows: During the fusion stage, for low-light near-infrared images The same pixel in Due to the adoption Sliding window processing will result in multiple block-level enhancement results at the pixel level. The corresponding candidate pixel values ​​are given at this location. ; Calculate the brightness difference : in, It is a pixel. The reference brightness value is the average of all candidate values; Final merged pixel values It is obtained through a weighted average method, and its calculation formula is as follows: Among them, weight The principle that the smaller the brightness difference, the greater the weight is, is defined as: in, To prevent positive constants with a denominator of zero, its value is 10. -3 .

Images