Low-illumination image enhancement method, system, equipment and medium

Abstract

The invention relates to the technical field of image processing, and discloses a low-illumination image enhancement method, system and device and a medium. The method comprises the following steps: acquiring a low-illumination image and corresponding event stream data; fusing the image features of the low-illumination image and the event features of the event stream data, and enhancing the fused image and the event features to obtain a preliminary enhanced image; dividing the features of the preliminarily enhanced image into three subspaces, enabling the three subspaces to activate linear features, nonlinear features and boundary features of the preliminarily enhanced image through a convex function, a non-convex function and a saturation function respectively, and obtaining low-order features of the preliminarily enhanced image after aggregation; the low-order features are modeled into a hypergraph, a plurality of hyperedges of the hypergraph describe semantic visual association among feature points in the low-order features, message transmission is carried out among nodes in each hyperedge through hypergraph convolution, and high-order features of the preliminarily enhanced image are obtained; and enhancing the low-illumination image through the low-order features and the high-order features to obtain a target enhanced image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, device and medium for low-light image enhancement. Background Technology

[0002] Low-light image enhancement aims to improve image quality in low-light environments by suppressing noise, restoring details, and correcting color distortion, making images visually clearer and more natural. Since images under low-light conditions often suffer from insufficient brightness, amplified noise, and motion blur, they not only affect human visual recognition but also reduce algorithm performance in applications such as nighttime autonomous driving, security monitoring, and medical image recognition. Therefore, the demand for high-quality, real-time low-light image enhancement technologies with scene adaptability is increasingly urgent.

[0003] Existing low-light image enhancement methods can be mainly divided into traditional image processing methods, convolutional neural network-based methods, and Transformer-based methods. Traditional image processing methods rely on manually designed enhancement rules, such as histogram equalization and Retinex-derived models. Although computationally efficient, they cannot adapt to different lighting conditions, easily leading to over-enhancement and color distortion. Convolutional neural network-based enhancement methods extract local features through deep convolutional structures, which can improve image quality to some extent. However, limited by the receptive field of convolution, they have weak modeling ability for long-distance feature dependencies and often only utilize RGB image data, making it difficult to simultaneously achieve noise reduction and detail preservation in extremely low-light environments. Transformer-based enhancement methods utilize self-attention mechanisms to achieve global feature modeling, achieving some improvement in brightness restoration and structural consistency. However, they focus more on low-order feature associations, resulting in insufficient enhancement results at the detail and structural levels.

[0004] The emergence of event cameras has provided a new data dimension for low-light image enhancement. Event cameras possess high dynamic range and high temporal resolution, enabling them to capture pixel-level brightness changes under extremely low light conditions, providing edge and motion information that is difficult to obtain with traditional cameras. In recent years, some studies have attempted to fuse event data with image data to improve enhancement results, achieving brightness restoration and detail enhancement through multi-scale fusion, signal-to-noise ratio guidance, or generative adversarial networks. However, existing methods rely on coarse-grained fusion of event and image features, failing to fully leverage the complementary advantages of event cameras.

[0005] In summary, existing low-light image enhancement methods have not achieved satisfactory enhancement results. Summary of the Invention

[0006] The purpose of this invention is to provide a method, system, device and medium for low-light image enhancement, which can solve the problem of poor low-light image enhancement effect.

[0007] To address the aforementioned technical problems, embodiments of the present invention provide a low-light image enhancement method, comprising the following steps: Acquire low-light images and corresponding event stream data obtained through the event camera; The image features of the low-light image and the event features of the event stream data are fused, and the low-light image is enhanced by the fused features to obtain a preliminary enhanced image; The features of the initial enhanced image are divided into three subspaces. The linear, nonlinear and boundary features of the initial enhanced image are activated by convex, non-convex and saturation functions, respectively, in the three subspaces. The activated features in the three subspaces are aggregated to obtain the low-order features of the initial enhanced image. The low-order features of the initially enhanced image are modeled as a hypergraph. The semantic visual association between each feature point in the low-order features is described by multiple hyperedges of the hypergraph. The hypergraph convolution enables message passing between each node in each hyperedge to obtain the interaction relationship between each feature point in the low-order features, which serves as the high-order features of the initially enhanced image. The low-light image is enhanced by initially enhancing its low-order and high-order features to obtain the target enhanced image.

[0008] Optionally, the target enhanced image is obtained through a low-light image enhancement model constructed based on a low-light image enhancement framework; wherein, the low-light image enhancement framework includes an encoder, a HyperFormer module, and a decoder; The encoder includes an event-image dual cross-attention unit and a HybridFFN module; The event-image dual-cross attention unit first enhances the interaction between image features and event features, and then first enhances the interaction between image features and image features, and then fuses the image features and event features again to obtain fused features; The HybridFFN module obtains low-order features by dividing the fused features into three subspaces for processing. The HyperFormer module obtains the interaction relationships between feature points in the low-order features, and then the HybridFFN module processes them to obtain the high-order features. The decoder fuses low-order and high-order features, uses an attention mechanism to extract features from the fused low-order and high-order features, and then uses the HybridFFN module to enhance the low-light image to output the target enhanced image.

[0009] Optionally, when acquiring low-order features, the HybridFFN module first uses point convolution to increase the channel dimension of the fused features, and then uses a preset ratio parameter to divide the channel-dimensional fused features into three subspaces. After aggregating the activated features in the three subspaces, depth convolution is used to reduce the channel dimension of the aggregated features to obtain low-order features.

[0010] Optionally, the decoder captures the global spatial features, local spatial features, and temporal features of the fused low-order and high-order features through grid attention mechanism, window attention mechanism, and residual channel attention mechanism, respectively. The global spatial features, local spatial features, and temporal features are then concatenated and enhanced in low-light images by the HybridFFN module.

[0011] Optionally, the non-convex function From the set of functions Selected from, Let f(x) be a set of functions that are zero-centered, monotonically increasing, non-convex, and bounded below, where f(x) = f(x) + ... x Represents input features; Convex functions adopt ; The saturation function is: .

[0012] Optionally, the low-light image enhancement model is trained using the L1 loss function and the LPIPS loss function; The L1 loss function is constructed based on the pixel-level absolute error between the target augmented image and the normal illumination image corresponding to the low-light image, while the LPIPS loss function is constructed based on the similarity between the perceptual features of the target augmented image and the normal illumination image extracted by the pre-trained Alex network.

[0013] Optionally, the step of enhancing the low-light image using the stitched features to obtain a preliminary enhanced image includes: The concatenated features are processed by convolution kernels with non-negative constraints to obtain a lighting map; The initial enhanced image is obtained by combining the illumination image and the low-light image through element-wise multiplication.

[0014] Embodiments of the present invention also provide a low-light image enhancement system, comprising: The image acquisition module is used to acquire low-light images and corresponding event stream data obtained through the event camera; The first image enhancement module is used to fuse the image features of the low-light image and the event features of the event stream data, and enhance the low-light image using the fused features to obtain a preliminary enhanced image; The first feature extraction module is used to divide the features of the preliminary enhanced image into three subspaces, and activate the linear features, nonlinear features, and boundary features of the preliminary enhanced image through convex functions, non-convex functions, and saturation functions, respectively. The activated features in the three subspaces are aggregated to obtain the low-order features of the preliminary enhanced image. The second feature extraction module is used to model the low-order features of the preliminary enhanced image as a hypergraph. The multiple hyperedges of the hypergraph describe the semantic visual association between each feature point in the low-order features. The hypergraph convolution enables message passing between each node in each hyperedge to obtain the interaction relationship between each feature point in the low-order features, which serves as the high-order features of the preliminary enhanced image. The second image enhancement module is used to enhance the low-light image by initially enhancing the low-order and high-order features of the image to obtain the target enhanced image.

[0015] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the low-light image enhancement method described above.

[0016] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described low-light image enhancement method.

[0017] The low-light image enhancement method provided by this invention has at least the following beneficial effects: This invention enhances the enhancement effect by fusing event data and image data. First, it performs preliminary enhancement on low-light images through the fusion of image features and event features. For the preliminary enhanced image, its features are divided into three subspaces. These subspaces are then activated by convex, non-convex, and saturated functions, respectively, to enhance the linear, non-linear, and boundary features of the preliminary enhanced image. This ensures the richness of the basic features by activating multi-level low-order features using convex, non-convex, and saturated functions. Then, the low-order features are modeled as a hypergraph, with hyperedges describing the semantic visual relationships between feature points. Hypergraph convolution is combined to achieve cross-regional message passing, thereby capturing high-order relational features. Combining these low-order and high-order features to enhance low-light images ensures that the target enhanced image retains pixel-level details such as texture clarity in low light while maintaining global structural consistency—no motion blur in dynamic scenes and no edge breaks in static scenes—effectively improving the enhancement effect of low-light images. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0019] Figure 1 A schematic flowchart of a low-light image enhancement method provided by the present invention; Figure 2 A schematic diagram of the network structure of a low-light image enhancement framework provided by the present invention; Figure 3 A schematic diagram of the structure of a HybridFFN module provided by the present invention; Figure 4 This is a schematic diagram of the structure of a HyperFormer module provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] One embodiment of the present invention relates to a low-light image enhancement method. The specific process of the low-light image enhancement method of this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Acquire low-light images and corresponding event stream data obtained through the event camera.

[0023] Step 102: The image features of the low-light image and the event features of the event stream data are fused, and the low-light image is enhanced using the fused features to obtain a preliminary enhanced image.

[0024] Step 103: Divide the features of the preliminary enhanced image into three subspaces, and activate the linear, nonlinear and boundary features of the preliminary enhanced image through convex, non-convex and saturation functions, respectively, in the three subspaces. Aggregate the activated features in the three subspaces to obtain the low-order features of the preliminary enhanced image.

[0025] Step 104: Model the low-order features of the preliminarily enhanced image as a hypergraph. Use multiple hyperedges of the hypergraph to describe the semantic visual associations between the feature points in the low-order features. Use hypergraph convolution to enable message passing between the nodes in each hyperedge to obtain the interaction relationships between the feature points in the low-order features, which are then used as the high-order features of the preliminarily enhanced image.

[0026] Step 105: Enhance the low-light image by initially enhancing the low-order and high-order features of the image to obtain the target enhanced image.

[0027] The implementation details of the low-light image enhancement method in this embodiment are described below. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0028] This embodiment specifically trains a low-light image enhancement model based on a low-light image enhancement framework (event-guided). Given a low-light image requiring enhancement, this low-light image enhancement model is used to obtain the target enhanced image. The training process of this low-light image enhancement model is described below:

[0029] First, training and testing sets are constructed. Both sets consist of low-light image-event pairing data (i.e., low-light images and corresponding event stream data obtained through the event camera) and corresponding clear images with normal lighting. Specifically, the low-light images are RGB images.

[0030] Specifically, this embodiment uses three types of publicly available benchmark datasets to construct the training and testing sets: the SDE dataset, the SDSD dataset, and the LIE dataset. The specific parameters and preprocessing methods for these three datasets are as follows: The SDE dataset contains 91 image-event pairing sequences captured by a DAVIS346 event camera, covering 43 indoor scenes and 48 outdoor scenes. Both the low-light images (i.e., RGB images) and the event stream data have a resolution of 346×260. During preprocessing, the event stream data was converted to tensor format and divided into a training set (76 pairs) and a test set (15 pairs) in an 8:2 ratio.

[0031] The SDSD dataset selects dynamic versions of low-light / normal-light paired videos with an original resolution of 1920×1080. During preprocessing, the resolution is first downsampled to 346×260 using bilinear interpolation, and then a noisy event stream is generated using a v2e event simulator. The training set (125 groups) and the test set (25 groups) are divided in a 5:1 ratio. The LIE dataset is a real-world low-light image-event pairing dataset, with most scenes having light intensities below 2 lux. It contains 2218 datasets, divided into a training set (1981 datasets) and a test set (237 datasets) in a 9:1 ratio. The test set consists of 1823 datasets of indoor scenes (1634 training datasets and 189 validation datasets) and 395 datasets of outdoor scenes (347 training datasets and 48 validation datasets). Preprocessing only involves normalizing the pixel values ​​of the images.

[0032] 0. Then, the low-light image enhancement framework is trained using the above training set. The low-light image enhancement framework can be found in [reference needed]. Figure 2 It includes a feature fusion unit, an encoder, a HyperFormer module, and a decoder. The feature fusion unit splices event and image features to generate a preliminary enhanced image. The encoder outputs low-order (abstract) features. The HyperFormer module captures high-order feature associations. The decoder outputs the final enhanced image (i.e., the target enhanced image), thereby realizing the fusion of event and image features and the capture of high-order features.

[0033] Specifically, the feature fusion unit is used to stitch together image features of the low-light image and event features of the event stream data, and then enhance the low-light image using the stitched features to obtain a preliminary enhanced image. For example... Figure 2 As shown, the feature fusion unit adopts a two-step strategy of "channel stitching + illumination map generation". The first step stitches the 64-dimensional event feature map and the 64-dimensional image feature map along the channel dimension (⊕) to obtain a 128-dimensional fused feature. The second step generates a 3-dimensional illumination map through a 1×1 convolution kernel with non-negative constraints. The illumination map is then combined with the original low-light image through element-wise multiplication (⊙) to obtain the preliminary enhanced image (I'). The calculation formula is as follows:

[0034] ; In the formula, I is the original low-light image, @ represents matrix multiplication, and |K| is a convolution kernel with non-negative constraints.

[0035] In this embodiment, when generating the initial enhanced image, a lighting map is constructed using a convolution kernel with non-negative constraints. This avoids insufficient brightness in the lighting map due to negative values ​​of the convolution kernel, ensuring that the initial brightened image meets the requirements for normal exposure.

[0036] See also Figure 2 The encoder consists of three encoder modules connected in series. Adjacent encoder modules are connected by a downsampling unit. Each encoder module contains a normalization layer, an event-image dual cross attention unit, and a HybridFFN module in sequence.

[0037] After layer normalization, the input features are further fused by the event-image dual-cross attention unit through the interaction between the initial enhanced image features and event features, and the interaction between the initial enhanced image features and image features, resulting in fused features. This dual-cross attention mechanism achieves deep fusion of event and image information. It first allows event features to interact with the main feature (i.e., the features of the initial enhanced image), and then allows image features to interact with the main feature, strengthening the deep fusion of event and image information through these two attention interactions. The interaction formulas between event features and image features and the main feature are as follows:

[0038] ; ; In the formula, Q e , K e , V e These are the query, key, and value matrices representing the event characteristics; Q i , K i , V i These represent the query, key, and value matrices for image features, respectively; @ represents matrix operations in the attention mechanism. softmax It is a normalization function used to transform the feature association weights into a probability distribution between 0 and 1; I' e It is an event-side fused feature obtained after the interaction between event features and main features. I' i It is an image-side fusion feature obtained after the interaction between image features and main features. After the fusion feature is enhanced by the HybridFFN module, it is reduced in dimensionality by the downsampling unit, and finally outputs a low-order abstract feature with 384 dimensions and a resolution of 32×32.

[0039] The HybridFFN module processes the fused features into three subspaces to obtain low-order features. Its structure is as follows: Figure 3 As shown, it sequentially includes point convolutional layers, feature partitioning units, multi-type activation function layers, feature aggregation units, and depthwise convolutional layers. During operation, the point convolutional layers first map the input N-dimensional feature map to 3×N dimensions, which is three times the original number of channels (i.e., channel upscaling). Then, a preset scaling parameter, such as... Divided into three subspaces, satisfying This involves dividing the features of the initially enhanced image into three subspaces, and then activating the linear, nonlinear, and boundary features of the initially enhanced image through convex, non-convex, and saturation functions, respectively.

[0040] The activation function is defined as follows: (1) Non-convex activation function From the set of functions Selecting functions from a set of zero-centered, monotonically increasing, non-convex functions with lower bounds, such as ReLU and GELU, is used to capture complex nonlinear patterns and enrich the model's feature representation capabilities. x Represents input features; (2) The convex activation function is That is, it is obtained by calculating the non-convex activation function with central symmetry, and is used to process linear features, alleviate gradient vanishing and accelerate convergence; (3) The saturation activation function is defined as: ; It is used to handle boundary features, introduces smooth nonlinear behavior, and improves the model's flexibility in handling boundary-specific scenarios.

[0041] The activated features are concatenated into 3×N dimensions through channel splicing, and then compressed to N dimensions through a deep convolutional layer to output enhanced features (i.e., low-order features).

[0042] The result of the HyperFormer module is as follows: Figure 4 As shown, it specifically includes a normalization layer, a hypergraph modeling unit, a spatial hypergraph convolutional layer, a residual connection module (Emb), and a HybridFFN module. The residual connection module consists of two 3×3 deep convolutional layers with GELU activation function. The HyperFormer module first models low-order features as a hypergraph, and then realizes high-order message passing through spatial hypergraph convolution. Combined with LayerNorm normalization, residual connections, and HybridFFN optimization, it finally outputs features that capture high-order correlations to the decoder.

[0043] During operation, the HyperFormer module first models the features output by the encoder (i.e., low-order features) as a hypergraph G=(V,E), where V is the vertex set and E is the hyperedge set. The hyperedges are constructed using a distance threshold and are defined as follows: ; In HyperFormer, low-order features are treated as a set of hypergraph vertices V, and for each vertex in the set V... Draw a "sphere" (denoted as ) with ... This "ball" contains all of the... exist d Other vertices in the dimensional feature space whose distance is less than a threshold The set of these vertices is the corresponding The hyperedges lay the foundation for higher-order feature interactions in hypergraph convolution.

[0044] Then, higher-order message passing is achieved through spatial hypergraph convolution, and the calculation formula is as follows: ; ; ; In the formula, I' h It is the feature map to be processed; D Let the neighborhood vertex set matrix be... M v , M e These are the angle matrices for the vertices and hyperedges, respectively; LN This is a normalization operation (Layer Normalization). HConv The symbol @ represents a hypergraph convolution operation, and @ represents a matrix operation. HybridFFN It is a hybrid feedforward network. emb ( I' h ) represents the residual connection module's connection to the feature map. I' h Process it; I' h1 , I' h2 These are the new feature maps obtained after processing by hypergraph convolution and hybrid feedforward networks, respectively.

[0045] After hypergraph convolution and HybridFFN optimization, high-order features are output to the decoder.

[0046] It should be noted that low-order features are local, basic visual features of an image, specifically including pixel-level brightness distribution, local texture, object edges or contours. These features only reflect the basic visual attributes of local areas of the image. High-order features are cross-scale, cross-semantic, and cross-temporal related features built on low-order features. Essentially, they are "complex interactive relationships between low-order features." Since low-order features are obtained through multiple downsampling in the encoder, hypergraphs are used to further perform inter-node message passing (feature interaction) on these low-order features. For similar low-order features, the connection will be further strengthened, such as those belonging to the same entity feature part, which will improve the contour boundaries and other performance of the final enhanced image.

[0047] The decoder fuses low-order and high-order features, employs an attention mechanism to extract features from the fused low-order and high-order features, and then uses a HybridFFN module to enhance the low-light image, outputting the target enhanced image. Its structure is further described in [link to structural details]. Figure 2It consists of three decoder modules connected in series. Adjacent decoder modules are connected through upsampling units. Each decoder module contains a normalization layer, a grid attention module, a window attention module, a residual channel attention module, and a HybridFFN module in sequence. It also fuses the output features of the encoder symmetric module through residual connections.

[0048] The decoder first uses an upsampling unit to gradually restore the resolution of high-order features. After fusing them with the low-order features of the encoder, the decoder extracts features through the collaborative efforts of the aforementioned multi-attention modules: the grid attention module captures global spatial information, the window attention module captures local spatial information, and the residual channel attention module captures temporal features. The output features of the three modules are concatenated and enhanced by the HybridFFN module. Finally, a 1×1 convolutional layer reduces the feature dimension from high-dimensional to 3-dimensional, outputting an enhanced image with the same resolution as the original image, thus improving the spatiotemporal consistency of the enhanced image.

[0049] Next, the loss function used during the training of the low-light image enhancement model is explained, including the L1 loss function and the LPIPS loss function. That is, the joint loss function includes pixel-level L1 loss and perceptual-level LPIPS loss, and the total loss is a weighted sum of the two loss terms, as detailed below:

[0050] The L1 loss function is constructed using the pixel-level absolute error between the enhanced target image and the normal illumination image corresponding to the low-light image, as shown in the following formula: ; In the formula, N is the total number of pixels in the image, and output i To enhance the value of the i-th pixel in the image, gt i It is the value of the i-th pixel in the real image (i.e., the image under normal lighting).

[0051] The LPIPS loss function is constructed based on the similarity between the perceptual features of the target augmented image extracted through the pre-trained Alex network and the normal lighting image, as shown in the following formula: ; In the formula, B Batch size (in this embodiment) B =8), which corresponds to the total number of images input during one training process; b Represents the first in the batch b Images (values ​​ranging from 1 to...) B ); k The number of output channels for the pre-trained model is the number of channels contained in a feature map of a certain layer of the pre-trained Alex network. i , j These are the pixel coordinates on the feature map, corresponding to the spatial location of the image; output bThe first in the batch b Zhang enhanced image, gt b This is the first in the batch b A true image (i.e., a reference image under normal lighting); LPIPS b,k,i,j This refers to the perceptual error of a single channel and a single pixel, specifically targeting the first... b Zhang Tu, No. k The first feature channel, the first i OK 、 No. j Column spatial location, calculate the perceptual difference between the enhanced image and the real image at that feature location.

[0052] The Alexa network is a deep neural network pre-trained on large-scale image data (capable of learning high-level visual features of images). It first extracts "perceptual features" from the target augmented image and the real reference image. These features are not simply pixel arrangements, but rather abstract information perceptible to human vision, reflecting image semantics, structure, texture style, etc. (e.g., the integrity of object outlines, the naturalness of colors, the subtlety of textures, etc.). Then, by calculating the differences between the two types of images in these perceptual feature dimensions (such as the distance between feature vectors), this difference is used as the loss value. Minimizing this loss value during training ensures that the augmented image output by the model not only closely approximates the real image at the pixel level but also maintains visual consistency with the real image, avoiding the problem of "small pixel-level errors but visually distorted colors and blurred details."

[0053] In the total loss function, the L1 loss weight is set to 1, and the LPIPS loss weight is set to 0.1, as shown in the following formula: ; The specific configuration of the training optimization strategy for the low-light image enhancement model is as follows: The Adam optimizer (β1=0.9, β2=0.999) is used, with an initial learning rate of 1×10⁻⁶. -4 The cosine annealing strategy is used to adaptively reduce the size to 1×10. -5 The total training cycle is 200 rounds. The performance is evaluated using a test set every 20 rounds. If the loss does not decrease for 3 consecutive rounds (the decrease is <0.001), the training is terminated early. The training is based on the PyTorch framework and is conducted on an NVIDIA RTX 3090 GPU (24GB VRAM). Mixed precision training is used to accelerate the process.

[0054] In addition, the model adopts an end-to-end training mode, and the process is as follows: load training set samples (low-light images, event streams, and real images) and perform data augmentation; input the samples into the augmentation framework, calculate the augmented image and joint loss value through forward propagation; based on the backpropagation algorithm, minimize the joint loss through the Adam optimizer, and iteratively update all trainable parameters in the framework (convolutional kernel weights, attention mechanism parameters, etc.); after each round of training, evaluate the performance with the test set, save the model with the minimum validation loss; repeat the above steps until the preset training cycle is reached or the early stopping strategy is triggered, and output the final trained model.

[0055] In the specific implementation, the input training set samples need to undergo data augmentation operations first: the input feature map is randomly cropped into 256×256 sub-images, and the cropped sub-images are randomly rotated by 90°, 180°, and 270°. The training set samples are randomly shuffled before each round of training.

[0056] Finally, the test set is input into the trained model, which outputs an enhanced low-light image to test the model.

[0057] Specifically, the test set low-light images are preprocessed by unifying resolution and normalizing pixel values, and then input into the training convergence framework. The inference process is as follows (see the framework processing flow above): Event streams and image data are processed by a convolutional neural network to extract features; event features and image features are processed by a feature fusion unit to generate a preliminary enhanced image; the preliminary brightened image is processed by an encoder to reduce dimensionality and output low-order abstract features; the low-order features are processed by the HyperFormer module to capture high-order correlations and output high-order features; the high-order features are processed by a decoder to restore resolution and fuse shallow features to output the final enhanced image.

[0058] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0059] Another embodiment of the present invention relates to a low-light image enhancement system. The implementation details of this low-light image enhancement system are described below. The following details are provided for ease of understanding and are not essential for implementing this solution. The low-light image enhancement system of this embodiment includes: The image acquisition module is used to acquire low-light images and corresponding event stream data obtained through the event camera; The first image enhancement module is used to fuse the image features of the low-light image and the event features of the event stream data, and enhance the low-light image using the fused features to obtain a preliminary enhanced image; The first feature extraction module is used to divide the features of the preliminary enhanced image into three subspaces, and activate the linear features, nonlinear features, and boundary features of the preliminary enhanced image through convex functions, non-convex functions, and saturation functions, respectively. The activated features in the three subspaces are aggregated to obtain the low-order features of the preliminary enhanced image. The second feature extraction module is used to model the low-order features of the preliminary enhanced image as a hypergraph. The multiple hyperedges of the hypergraph describe the semantic visual association between each feature point in the low-order features. The hypergraph convolution enables message passing between each node in each hyperedge to obtain the interaction relationship between each feature point in the low-order features, which serves as the high-order features of the preliminary enhanced image. The second image enhancement module is used to enhance the low-light image by initially enhancing the low-order and high-order features of the image to obtain the target enhanced image.

[0060] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0061] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0062] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the low-light image enhancement methods of the above embodiments.

[0063] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0064] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0065] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0066] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A low-light image enhancement method, characterized in that, The method includes: Acquire low-light images and corresponding event stream data obtained through the event camera; The image features of the low-light image and the event features of the event stream data are fused, and the low-light image is enhanced by the fused features to obtain a preliminary enhanced image; The features of the initial enhanced image are divided into three subspaces. The linear, nonlinear and boundary features of the initial enhanced image are activated by convex, non-convex and saturation functions, respectively, in the three subspaces. The activated features in the three subspaces are aggregated to obtain the low-order features of the initial enhanced image. The low-order features of the initially enhanced image are modeled as a hypergraph. The semantic visual association between each feature point in the low-order features is described by multiple hyperedges of the hypergraph. The hypergraph convolution enables message passing between each node in each hyperedge to obtain the interaction relationship between each feature point in the low-order features, which serves as the high-order features of the initially enhanced image. The low-light image is enhanced by initially enhancing its low-order and high-order features to obtain the target enhanced image.

2. The low-light image enhancement method according to claim 1, characterized in that, The target enhanced image is obtained through a low-light image enhancement model constructed based on a low-light image enhancement framework; wherein, the low-light image enhancement framework includes an encoder, a HyperFormer module, and a decoder; The encoder includes an event-image dual cross-attention unit and a HybridFFN module; The event-image dual-cross attention unit first enhances the interaction between image features and event features, and then first enhances the interaction between image features and image features, and then fuses the image features and event features again to obtain fused features; The HybridFFN module obtains low-order features by dividing the fused features into three subspaces for processing. The HyperFormer module obtains the interaction relationships between feature points in the low-order features, and then the HybridFFN module processes them to obtain the high-order features. The decoder fuses low-order and high-order features, uses an attention mechanism to extract features from the fused low-order and high-order features, and then uses the HybridFFN module to enhance the low-light image to output the target enhanced image.

3. The low-light image enhancement method according to claim 2, characterized in that, When acquiring low-order features, the HybridFFN module first uses point convolution to increase the dimensionality of the fused features through channels, and then uses a preset ratio parameter to divide the fused features after channel dimensionality increase into three subspaces. After aggregating the activated features in the three subspaces, it uses depth convolution to reduce the dimensionality of the aggregated feature channels to obtain low-order features.

4. The low-light image enhancement method according to claim 3, characterized in that, The decoder captures the global spatial features, local spatial features, and temporal features of the fused low-order and high-order features through grid attention mechanism, window attention mechanism, and residual channel attention mechanism, respectively. The global spatial features, local spatial features, and temporal features are then concatenated and enhanced in low-light images through the HybridFFN module.

5. The low-light image enhancement method according to claim 1, characterized in that, The nonconvex function From the set of functions Selected from, Let f(x) be a set of functions that are zero-centered, monotonically increasing, non-convex, and bounded below, where f(x) = f(x) + ... x Represents input features; Convex functions adopt ; The saturation function is: 。 6. The low-light image enhancement method according to claim 2, characterized in that, The low-light image enhancement model is trained using the L1 loss function and the LPIPS loss function. The L1 loss function is constructed based on the pixel-level absolute error between the target augmented image and the normal illumination image corresponding to the low-light image, while the LPIPS loss function is constructed based on the similarity between the perceptual features of the target augmented image and the normal illumination image extracted by the pre-trained Alex network.

7. The low-light image enhancement method according to claim 1, characterized in that, The process of enhancing the low-light image using the stitched features to obtain a preliminary enhanced image includes: The concatenated features are processed by convolution kernels with non-negative constraints to obtain a lighting map; The initial enhanced image is obtained by combining the illumination image and the low-light image through element-wise multiplication.

8. A low-light image enhancement system, characterized in that, The system includes: The image acquisition module is used to acquire low-light images and corresponding event stream data obtained through the event camera; The first image enhancement module is used to fuse the image features of the low-light image and the event features of the event stream data, and enhance the low-light image using the fused features to obtain a preliminary enhanced image; The first feature extraction module is used to divide the features of the preliminary enhanced image into three subspaces, and activate the linear features, nonlinear features, and boundary features of the preliminary enhanced image through convex functions, non-convex functions, and saturation functions, respectively. The activated features in the three subspaces are aggregated to obtain the low-order features of the preliminary enhanced image. The second feature extraction module is used to model the low-order features of the preliminary enhanced image as a hypergraph. The multiple hyperedges of the hypergraph describe the semantic visual association between each feature point in the low-order features. The hypergraph convolution enables message passing between each node in each hyperedge to obtain the interaction relationship between each feature point in the low-order features, which serves as the high-order features of the preliminary enhanced image. The second image enhancement module is used to enhance the low-light image by initially enhancing the low-order and high-order features of the image to obtain the target enhanced image.

9. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the low-light image enhancement method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the low-light image enhancement method as described in any one of claims 1 to 7.

Images