Night image halo removal processing method based on sub-space mapping

Abstract

The invention relates to the field of image processing, and discloses a sub-space mapping-based night image halo removal processing method, which comprises the following steps of: obtaining an original image containing halo; halo region extraction is carried out on the original image to generate a halo mask; performing multi-scale decomposition on the image, and separating image information into detail layers of different scales; the detail layer is converted into a subspace, and halo information is identified and separated through an adaptive subspace mapping algorithm; performing image fusion on the detail layer subjected to halo removal and the base layer to generate a halo-removed image; and carrying out brightness, contrast and color optimization processing on the halo-removed image to obtain a halo-removed image. The method has good adaptability and real-time processing capability, and is suitable for various night imaging application scenes such as night monitoring, automatic driving and the like.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a night image dehalation processing method based on sub-space mapping. Background Art

[0002] In the field of image processing, flare refers to the halo effect that appears in the surrounding area due to the presence of a strong light source, which often affects the quality and visibility of the image, especially when shooting at night. The flare phenomenon usually originates from the optical characteristics of the camera lens, the response of the imaging sensor, and the imperfection of the image processing algorithm.

[0003] To address this problem, researchers have proposed a variety of image enhancement techniques. Among them, the technology based on subspace mapping has attracted increasing attention. Subspace mapping is a mathematical tool that maps high-dimensional data to low-dimensional space. It is widely used in feature extraction, noise reduction, and image reconstruction. In the process of dehalation, subspace mapping can effectively extract and separate the halo and effective signal in the image, thereby improving the image quality.

[0004] The commonly used subspace mapping method achieves the purpose of removing unnecessary halo effects by linearly transforming the input image and representing the effective signal and halo with orthogonal basis vectors in the orthogonal subspace. These techniques can significantly improve the clarity and readability of images by processing night images.

[0005] These methods can be divided into two categories: one is focused on generating dehalation datasets for daytime and nighttime scenes, solving the problem of how to distinguish light sources from halos through a variety of strategies; the other is optimization techniques, which mainly target halos with unique shapes or patterns in daytime images.

[0006] On the one hand, some image-based optimization and analysis methods mainly rely on the specific position and shape of the halo. These solutions are suitable for halos in limited areas, such as bright spots of light. However, in images taken at night, halo pollution often appears as local blur, which is difficult to remove. At this time, it is more important to identify and extract data information in the blurred area than to focus only on the unique shape or position of the halo. There is currently no dedicated network for night halo removal, and existing methods lack prior knowledge and have difficulty handling complex halo removal tasks at night. General networks mainly include UFormer[1], Restormer[2], UNet[3], and HINet[4]. This type of neural network mainly uses its U-shaped encoder-decoder structure to achieve multi-scale feature extraction and image restoration. It adopts a general image restoration structure and does not make special processing on the image halo part. Flare7kpp[5] is a new type of dataset based on real halo. The emergence of this type of dataset makes the dataset used in the training process closer to the halo in the real dataset. Zhou[6] et al. proposed to consider the ISP process and tone mapping in the process of halo removal, and increased the number of halos to achieve the generalization of the algorithm. DCP[7] is a dehazing algorithm based on dark channel prior, which also has some effect on halo removal and is also used for comparison here.

[0007] On the other hand, in order to solve the problem of insufficient paired training datasets, Wu et al. [8] first proposed a semi-synthetic paired dataset method to address the challenge of daytime halo removal in a data-driven way. In order to enrich the dataset, Dai et al. introduced the first nighttime halo dataset Flare7K [9], and subsequently launched an improved version Flare7kpp [5], which is one of the widely used benchmark datasets. The above two types of algorithms or datasets have supplemented the halo removal task, but have not completely solved the halo removal problem. Therefore, the present invention proposes a subspace mapping-based method to solve the halo removal problem in nighttime scenes.

[0008] The prior art related to the present invention includes the following references:

[0009] [1] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR, pages 17662–17672, 2022.

[0010] [2]Syed Waqas Zamir,Aditya Arora,Salman Khan,Munawar Hayat,FahadShahbaz Khan,and Ming–Hsuan Yang.Restormer:Efficient transformer for high-resolution image restoration.In CVPR,pages 5718–5729,2022.

[0011] [3]Olaf Ronneberger,Philipp Fischer,and Thomas Brox.U-Net:Convolutional networks for biomedical image segmentation.In Medical ImageComputing and Computer-Assisted Intervention,pages 234–241.Springer,2015.

[0012] [4]Liangyu Chen,Xin Lu,Jie Zhang,Xiaojie Chu,and ChengpengChen.HINet:Half instance normalization network for image restoration.In CVPRWorkshop,pages 182–192,2021.

[0013] [5]Yu Dai,Chongyi Li,Shangchen Zhou,Ruicheng Feng,Yihang Luo,and ChenChange Loy.Flare7k++:Mixing synthetic and real datasets for nighttime flareremoval and beyond.ArXiv,abs / 2306.04236,2023.

[0014] [6] Yuyan Zhou, Dong Liang, Songcan Chen, Sheng-Jun Huang, Shuo Yang, and Chongyi Li. Improving lens flare removal with general-purpose pipeline and multiple light sources recovery. In 2023 IEEE / CVF International Conference on Computer Vision (ICCV), pages 12923–12933, 2023.

[0015] [7] Kaiming He, Jian Sun, and Xiaoou Tang. Single image haze removal using dark channel prior. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1956–1963, 2009.

[0016] [8] Yicheng Wu, Qiurui He, Tianfan Xue, Rahul Garg, Jiawen Chen, Ashok Veeraraghavan, and Jonathan T Barron. How to train neural networks for flareremoval. In ICCV, pages 2239–2247, 2021.

[0017] [9] Yuekun Dai, Chongyi Li, Shangchen Zhou, Ruicheng Feng, and Loy Chen Change. Flare7K: a phenomenological nighttime flare removal dataset. In NeurIPS, 2022. Summary of the invention

[0018] In view of the deficiencies in the prior art, the present invention provides a night image dehalation processing method based on sub-space mapping, which solves the problem of halo effect caused by light sources in night images, thereby improving the clarity and visual quality of night images.

[0019] To achieve the above objectives, the present invention is implemented by the following technical solutions: A night image dehalation processing method based on sub-space mapping, comprising the following steps:

[0020] Get the original image including the halo;

[0021] Extract the halo area of ​​the original image and generate a halo mask;

[0022] Perform multi-scale decomposition on the image to separate the image information into detail layers of different scales;

[0023] The detail layer is converted into subspace, and the halo information is identified and separated by an adaptive subspace mapping algorithm;

[0024] The dehaloated detail layer is fused with the base layer to generate a dehaloated image;

[0025] The dehalated image is post-processed to obtain a dehalated image.

[0026] Preferably, the step of extracting the halo area from the original image and generating a halo mask comprises:

[0027] Perform color channel decomposition on the original image, decomposing the image into three channels: red, green, and blue;

[0028] Set the threshold for halo detection in the green channel and extract the highlight area based on the threshold;

[0029] Mark the extracted highlight area as the halo candidate area and generate a preliminary halo mask;

[0030] Perform morphological processing on the preliminary halo mask to generate an accurate halo mask;

[0031] The precise halo mask is superimposed on the original image to record the spatial location of the halo area.

[0032] Preferably, the step of performing multi-scale decomposition on the image comprises:

[0033] The image is downsampled step by step through the U-shaped transformation unit to generate feature maps of different resolutions;

[0034] In each downsampling level, an encoding module containing convolution operations and attention mechanisms is used to extract local and global features;

[0035] In the downsampling process, the image resolution is gradually reduced and the number of channels is increased so that the low-resolution feature map contains global information and the high-resolution feature map retains edges and details;

[0036] The feature maps at each level are passed to the decoder through skip connections to maintain key information at different scales;

[0037] The decoder upsamples the feature map layer by layer, restores it to the original resolution, and fuses the details of each layer;

[0038] The weights of halo and effective signal are adjusted according to the distribution of different frequency characteristics, so that the high-resolution layer focuses on edge details and the low-resolution layer processes the halo outline, thereby achieving the separation of halo and effective signal.

[0039] Preferably, the step of converting the detail layer into a subspace and identifying and separating the halo information by an adaptive subspace mapping algorithm comprises:

[0040] The decomposed detail layer is input into the subspace mapping unit, and the detail layer features are converted into a low-dimensional subspace using the subspace decomposition operation;

[0041] The transformed detail layer is decomposed into multiple orthogonal basis vectors in the subspace, each basis vector represents a different feature dimension, and the basis vector parameters are adaptively adjusted so that the basis vectors can correspond to the halo information and background information respectively;

[0042] By using the supervised learning mechanism, the adaptive subspace mapping algorithm is optimized through the loss function, and the basis vector weights are adjusted to minimize the difference between the de-haloated image and the reference image without halo.

[0043] Preferably, the step of converting the detail layer into a subspace and identifying and separating the halo information by an adaptive subspace mapping algorithm further comprises:

[0044] The subspace mapping parameters are dynamically adjusted to adapt the basis vectors to the changes in image features, thereby achieving real-time separation of halo features in different scenes.

[0045] Preferably, the step of fusing the de-haloated detail layer with the base layer to generate a de-haloated image comprises:

[0046] The de-haloated detail layer and base layer are sent back to the neural network through skip connections;

[0047] Using the image fusion technology of neural network, the de-haloated detail layer and the base layer are gradually fused at multiple scales to maintain the naturalness and detail integrity of the output image.

[0048] In the fusion process, the light source is supplemented for the background part. First, the light source area of ​​the original image is extracted by thresholding, and the light source area is separated from the background.

[0049] The separated light source area is supplemented in nonlinear space and the processed light source information is integrated into the background without halo;

[0050] The fused image is converted into a standard color space through the inverse transformation of nonlinear processing to generate a reconstructed image that contains the background and light source but removes the halo.

[0051] Preferably, the step of post-processing the de-haloated image to obtain the de-haloated image comprises:

[0052] Adjust image brightness and contrast to enhance the visual effect after dehalation.

[0053] The present invention also provides a night image dehalation processing device based on sub-space mapping, comprising:

[0054] An image acquisition module, used for acquiring an original image including a halo;

[0055] A halo region extraction module, used for extracting the halo region from the original image and generating a halo mask;

[0056] A multi-scale decomposition module, used to perform multi-scale decomposition on the original image and separate the image information into detail layers of different scales;

[0057] A subspace mapping module, used to convert the detail layer into a subspace, and identify and separate halo information through an adaptive subspace mapping algorithm;

[0058] An image fusion module is used to fuse the detail layer after dehalation with the base layer to generate a dehalation image;

[0059] The post-processing module is used to optimize the brightness, contrast and color of the de-haloated image to obtain the de-haloated image.

[0060] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0061] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0062] The present invention provides a method for removing halo from night images based on subspace mapping.

[0063] Beneficial effects:

[0064] 1. The present invention uses an adaptive subspace mapping algorithm to accurately separate halo information in the subspace, ensuring efficient removal of halo in nighttime images. Compared with traditional methods, the present invention first uses a mask to determine the halo position information, and then combines multi-scale decomposition and subspace mapping, so that the halo area and the effective signal can be processed separately at different scales, achieving a more accurate dehalation effect.

[0065] 2. The present invention adopts a multi-scale decomposition module and a jump connection structure, so that the high-frequency details and low-frequency structures of the image can be retained in layers, avoiding the loss of details that may occur during the dehalation process. Through the weighted combination of the image fusion module, not only the halo is removed, but also the edge and texture information of the image is maintained, making the final image clear and natural.

[0066] 3. The adaptive subspace mapping module of the present invention can dynamically adjust the mapping parameters according to the changes in the halo characteristics in different scenes, and adaptively separate the halo area. This adaptability enables the present invention to be applicable to a variety of night scenes, such as streets, car lights, night scene buildings, etc., and even if the halo shapes and brightness are different, it can achieve effective halo removal.

[0067] 4. The post-processing module of the present invention optimizes the brightness, contrast and color of the dehalation image, so that the light and dark details, layering and color performance of the image are significantly improved. Especially in the night environment, the optimized image is more in line with the visual perception of the human eye, presents natural brightness and contrast, and enhances the subjective visual quality of the image.

[0068] 5. The structural design and algorithm optimization of the present invention improve the real-time performance of image processing, so that the device can achieve near real-time image dehalation processing under the condition of sufficient computing resources. This is particularly important for application scenarios that require response speed, such as nighttime monitoring and automatic driving, and ensures the stability and efficiency of the system in complex nighttime environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0070] Figure 2 It is a schematic diagram of the process framework of the method of the present invention;

[0071] Figure 3 For the present invention Figure 2 Detailed structural diagram of neutron space mapping;

[0072] Figure 4 For the present invention Figure 2 Detailed structural diagram of the transformation module;

[0073] Figure 5 For the present invention Figure 4 Detailed structural diagram of the transform sub-block;

[0074] Figure 6 It is a schematic diagram of the performance of the method of the present invention on some data sets;

[0075] Figure 7 It is a schematic diagram of the structure of the device of the present invention;

[0076] Figure 8 It is a schematic diagram of the computer device structure of the present invention.

[0077] Among them, 100, image acquisition module; 200, halo area extraction module; 300, multi-scale decomposition module; 400, subspace mapping module; 500, image fusion module; 600, post-processing module; 40, computer equipment; 41, processor; 42, memory; 43, storage medium. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] Please see attached Figure 1 -Attached Figure 5 The present invention provides a method for removing halo from night images based on subspace mapping, which can achieve accurate separation and removal of halo in night images through multiple processing steps, and effectively improve the visual quality of the image. The following is a detailed description of each step of the method.

[0080] S1. Get the original image containing the halo

[0081] In order to ensure that the dehalation processing method of the present invention can effectively separate the halo area in the night image and generate a high-quality dehalation image, it is necessary to first obtain the original image containing the halo. This step provides the data basis for the subsequent halo detection, feature decomposition and removal. Therefore, during the acquisition process, not only the image is required to be clear, but also the brightness and detail information of each area in the image must be kept as complete as possible, so as to facilitate accurate identification and separation in the subsequent processing stage.

[0082] In this embodiment, the image acquisition step specifically includes acquiring an image with a halo in a night scene. Generally, in order to ensure the accuracy and quality of image acquisition, a high-sensitivity camera device can be selected, such as a CMOS or CCD camera with low-light imaging capability, or a sensor with a light enhancement effect. During the acquisition process, by adjusting the exposure time, gain, contrast, white balance and other parameters of the camera, it is ensured that the halo area in the image can form a significant contrast with other image areas, thereby facilitating subsequent processing.

[0083] Specifically, in order to make the halo information in the original image obvious during night image acquisition, in some embodiments, the camera exposure time can be increased to enhance the brightness of the light source or reflection area, so as to generate a clear halo area image; the camera gain can also be appropriately adjusted so that the background information with lower brightness can be clearly presented, thereby providing complete background data support for subsequent halo extraction. In a possible implementation, selecting a lower white balance value can enhance the cold color characteristics of the halo area, which helps to separate the halo from the non-halo area during the halo extraction process.

[0084] In practical applications, due to the variety of light sources at night and the uneven light, multiple halo areas may appear in the image. In order to adapt to the halo types caused by different light sources, a halo feature enhancement filter can be introduced in some embodiments. As an option, Gaussian blur or other types of spatial smoothing filtering operations can be performed on the image to reduce the interference of background impurities and subtle noise on the extraction of halo areas, thereby highlighting the characteristics of the halo areas.

[0085] Generally speaking, after the above image acquisition and preprocessing, the original image data containing obvious halo areas can be obtained, thus laying the foundation for the subsequent halo mask generation.

[0086] In some implementations, to ensure that the captured image has good noise control, especially in low-light environments, the signal-to-noise ratio of the image is improved by setting the camera to a low-noise mode (such as turning off the high-gain mode or turning on the long exposure mode).

[0087] S2. Extract the halo area of ​​the original image and generate a halo mask

[0088] In order to effectively extract the halo area information in the night image and generate the halo mask, the embodiment of the present invention performs the extraction process of the halo area after acquiring the original image. The core of this step is to perform color channel analysis and threshold segmentation on the image in order to accurately locate the position of the halo area, thereby providing a basis for subsequent halo removal and detail layer separation. Generally, the halo area in the night image appears as a high brightness area due to overexposure of the light source, so the halo area can be extracted more accurately through an appropriate threshold segmentation method.

[0089] In this embodiment, the acquired original image is first decomposed into three channels: red, green, and blue. Specifically, the green channel is selected as the main analysis channel for halo extraction. The green channel is more sensitive to the high brightness of the halo area under normal night shooting conditions, and can better capture halo detail information. In some embodiments, the extraction effect of other channels can be confirmed by experiments and a suitable color channel can be selected as the main channel.

[0090] In a possible implementation, a halo detection threshold T is set, which is used to filter out high brightness areas and satisfy the pixel intensity I g >T. Among them, I g Represents the pixel brightness value of the green channel. By setting a suitable threshold T, the brightness information of the halo area can be significantly different from that of other areas, thereby accurately separating the halo area. In general, the threshold setting can be adjusted based on the brightness value range of experimental statistics to adapt to the brightness intensity of different light sources.

[0091] When the brightness threshold condition is met, a preliminary halo mask is generated to mark the halo candidate area in the image. As an option, the preliminary generated halo mask can be morphologically processed to remove small noise areas and possible edge holes in the mask. In some embodiments, dilation or erosion operations can be used to fill smaller holes or eliminate isolated noise points, thereby further improving the accuracy and continuity of the halo mask.

[0092] Specifically, the morphological processing after the halo mask generation includes the following steps:

[0093] First, dilation operation is performed to enhance the edge of the halo area and make the halo area coherent;

[0094] Then perform an erosion operation to suppress small noise points in the non-halo area;

[0095] Optionally, further opening and closing operations can be performed to smooth the mask boundaries.

[0096] In a possible implementation, in order to adapt to complex halo morphology, an adaptive threshold segmentation algorithm can be introduced to dynamically adjust the threshold according to the uneven brightness of the local halo area, thereby ensuring that the generated halo mask can fully cover the entire halo area without affecting the background.

[0097] In some embodiments, the generated halo mask can also be superimposed on the original image to mark and record the spatial position information of the halo area. Specifically, the halo area can be highlighted by fusing the mask with the original image, and the fused image is used to verify the accuracy of the halo area.

[0098] S3: Perform multi-scale decomposition on the image to separate the image information into detail layers of different scales

[0099] To ensure that the different scale feature information of the halo area in the night image can be effectively separated, the embodiment of the present invention performs multi-scale decomposition processing on the original image after generating the halo mask to separate the image information into detail layers of different scales. Generally, the halo area usually covers a large area and has fuzzy edge features, while other details of the night image have higher edge clarity and higher spatial frequency. Therefore, multi-scale decomposition can effectively distinguish the halo area from the detail information, providing a technical basis for subsequent halo removal.

[0100] In this embodiment, the image is first downsampled layer by layer using a U-shaped transformation module, and multiple feature maps of different resolutions are generated. Specifically, the U-shaped structure includes multi-layer convolution and pooling operations on the image, wherein the convolution operation is used to extract local spatial features, while the pooling operation reduces the resolution layer by layer, retaining the main contours and low-frequency information of the image. In one possible implementation, a 3×3 convolution kernel can be used to ensure the accuracy of spatial information extraction, while gradually acquiring different scale features of the image through a multi-level convolution structure. In this process, the spatial resolution of the image is reduced layer by layer, and while generating low-frequency information, the number of channels is gradually increased to more comprehensively capture the multi-scale features of the image.

[0101] During the downsampling process, an embodiment of the present invention transfers the features of the encoding stage to the corresponding level of the decoding stage through a skip connection to ensure that the image does not lose important high-frequency detail information during the subsequent decoding process. As an option, the skip connection can directly input high-resolution features of the same scale from the encoding stage to the decoding stage so as to fully retain the original details when restoring the resolution. In some embodiments, a 1×1 convolution can be selected to perform channel compression on the feature map of the skip connection to meet the input requirements of the decoding stage and reduce the amount of calculation.

[0102] Generally speaking, low-resolution feature maps are used to extract the overall outline and low-frequency information of the image, which helps to distinguish the large-scale blurred boundaries of the halo; while high-resolution feature maps are used to capture the high-frequency detail information of the image, so that in the process of removing the halo, we can focus on retaining the detail layer with clear edges in the image. As an option, an attention mechanism can be added after each level of convolution operation to further enhance the perception of different frequency features, thereby improving the accuracy of feature extraction.

[0103] In some embodiments, for each decomposition level of the encoding process, it can be expressed as a layer-by-layer iterative formula of the convolution operation:

[0104] F i =σ(W i *F i-1 +b i )

[0105] Among them, F i represents the feature map of the i-th layer, W i is the convolution kernel weight of the i-th layer, b i is the bias term, and σ is the activation function (generally ReLU). This formula describes the feature changes of the image after layer-by-layer convolution processing, and generates feature maps of different scales by extracting spatial features layer by layer.

[0106] To meet the requirements of different image resolutions, in one possible implementation, a frequency domain filter can be applied to the multi-scale feature map obtained in the encoding stage to enhance high-frequency details or suppress low-frequency background noise. Specifically, the feature map can be converted into the frequency domain space through Fourier transform, and the high-frequency and low-frequency components can be adjusted separately to improve the effectiveness of feature decomposition.

[0107] After completing the multi-scale decomposition, in this embodiment, these multi-scale feature maps are passed to the decoding stage through jump connections, so that the decoding stage can gradually reconstruct the image structure by using high-frequency detail information when restoring the resolution layer by layer, thereby ensuring that the overall information and detail information of the image can remain consistent after dehalation.

[0108] S4: Convert the detail layer into subspace, and identify and separate the halo information through the adaptive subspace mapping algorithm

[0109] In order to achieve accurate separation and removal of the halo area, after completing the multi-scale decomposition of the image, the embodiment of the present invention uses an adaptive subspace mapping algorithm to process the decomposed detail layer, converts the detail layer features into the subspace, and identifies and separates the halo information by adaptive subspace mapping. In general, the halo information and the effective signal of the image have significant differences in spatial distribution and frequency characteristics. Therefore, by converting the image features into the subspace and decomposing them into different basis vectors, the halo and background information can be effectively distinguished in the subspace, thereby removing the halo effect.

[0110] In this embodiment, the detail layer features obtained by multi-scale decomposition are first mapped to a low-dimensional subspace, and decomposed into multiple orthogonal basis vectors through a subspace mapping operation. In a possible implementation, the detail layer can be decomposed into 16 basis vectors to more fully represent the halo features and background features at different scales.

[0111] Specifically, assuming that the input detail layer feature is represented as a matrix F, it can be mapped to a low-dimensional space through the subspace mapping matrix W to obtain the mapped low-dimensional feature F':

[0112] F′=W·F

[0113] Wherein, W represents a mapping matrix, which is used to project the detail layer features in the original feature space to the subspace. Generally, the parameters of the mapping matrix W can be determined by pre-training to ensure the distinction between the halo features and the background features in the projected subspace.

[0114] In the subspace, this embodiment realizes adaptive separation of halo information by dynamically adjusting the basis vectors. As an option, the weight coefficients of the basis vectors can be adjusted according to the halo characteristics of the image so that each basis vector can correspond to the halo information and the background information respectively. Specifically, in the mapped subspace, the linear combination of the basis vectors can be expressed as:

[0115]

[0116] Among them, B i represents the i-th basis vector, α i is the corresponding weight coefficient. By adjusting α i The value of can reduce the basis vector weights related to the halo and enhance the basis vector weights of the background information to achieve the separation of the halo and the background.

[0117] In some embodiments, in order to further optimize the effect of subspace mapping, the embodiments of the present invention introduce a supervised learning mechanism to optimize the adaptive subspace mapping algorithm through a loss function. In this implementation, a loss function L is defined, and the difference between the dehaloated image and the reference image is used as the optimization target:

[0118] L=||I ref -I out || 2

[0119] Among them, I ref represents the reference image without halo, I out is the output image after dehalation. By minimizing this loss function, the mapping matrix W and the weight coefficient α can be optimized. i , making the dehaloated image closer to the reference image without halo.

[0120] As a possible implementation method, the loss function L can be iteratively optimized using the gradient descent method or other optimization algorithms, and the mapping matrix W and the weight coefficient α can be gradually updated. i In general, through repeated iterations, better mapping parameters can be obtained in the training stage, so that the image after dehalation has higher clarity and background information fidelity.

[0121] In some embodiments, for different types of nighttime image halo features, the present invention can also add an adaptive parameter adjustment mechanism in the subspace mapping according to actual needs. As an option, the suppression strength of the basis vectors related to the halo can be increased in scenes with strong halo, and the suppression strength of the basis vectors can be reduced in scenes with weak halo, so as to ensure that the processed image maintains the integrity of the background as much as possible.

[0122] Through the above-mentioned adaptive subspace mapping algorithm, this embodiment can accurately identify and separate halo information, and finally provide accurate halo separation features for subsequent image fusion and de-halation effect optimization.

[0123] S5. Fusing the detail layer after dehalation with the base layer to generate a dehalation image

[0124] In order to ensure that the image after the halo is removed is natural in visual effect and the details are intact, the embodiment of the present invention combines the de-haloed detail layer with the base layer through image fusion technology after the halo is separated, so as to generate a complete de-haloed image. Generally, after the de-halo process, the detail layer contains the fine texture information after the de-halo process, while the base layer maintains the overall contour and low-frequency information of the image. Therefore, through multi-scale image fusion, high-frequency details and low-frequency background can be effectively and reasonably combined to ensure the integrity of the image structure and the naturalness of the de-halo effect.

[0125] In this embodiment, the dehalation detail layer and the base layer are first transferred to the image fusion module of the neural network through a jump connection, so as to gradually fuse the different frequency components of the image at multiple scales. Specifically, in the image fusion process, the feature map of the detail layer and the feature map of the base layer are weighted fused at each level of scale. As an option, the fusion weight can be adaptively adjusted according to the texture density or frequency component of each layer of the feature map to enhance the expressiveness of high-frequency details while ensuring a smooth transition in the low-frequency area.

[0126] In a possible implementation, the image fusion process can be expressed by the following formula:

[0127]

[0128] Among them, I fusion is the fused image, F detail,i represents the features of the dehalation detail layer at the i-th scale, B base,i is the feature of the base layer at the same scale, α i and β i are the fusion weights of the detail layer and the base layer respectively. In the fusion process, the weight α i and β i It can be adaptively adjusted according to the content of the feature map so that the detail layer has a larger weight in the high-frequency area, while the base layer has a larger weight in the low-frequency area, thereby achieving efficient image fusion.

[0129] In some embodiments, in order to further enhance the expressiveness of the light source area, the present invention can selectively perform nonlinear enhancement processing on the light source area in the base layer. Specifically, the light source area in the original image is first extracted by threshold segmentation to identify the spatial position and brightness information of the light source; then, brightness enhancement processing is performed on these light source areas during the fusion process to ensure that the final image still maintains appropriate light source brightness after the halo is removed.

[0130] In a possible implementation, light source enhancement can be controlled by a nonlinear mapping function, for example, an exponential function is used to increase the brightness of the light source area, and the formula is as follows:

[0131] I light =I base ·exp(k·I mask )

[0132] Among them, I light Indicates the brightness of the light source area after enhancement, I base is the original brightness value of the base layer, I maskis the brightness value of the light source mask area, and k is the gain parameter of the nonlinear mapping. In general, by adjusting the k value, the enhancement intensity of the light source can be flexibly controlled to ensure that the image after dehalation has no halo and maintains the visual effect of the light source.

[0133] After the fusion is completed, the embodiment of the present invention also performs color space conversion on the fused image to ensure the color balance of the image. As an option, an inverse nonlinear transformation can be used to restore the processed image from the nonlinear space to a standard color space (such as RGB color space) to obtain a natural color transition and a balanced brightness distribution.

[0134] In some embodiments, the color space conversion process of the fused image can be further optimized by linear interpolation to make the color transition between the high-frequency details and the low-frequency background smoother. Specifically, the color information of the high-frequency layer and the low-frequency layer can be calculated by weighted interpolation to ensure that the transition between the two is visually consistent.

[0135] Through the above-mentioned image fusion steps, the embodiment of the present invention can generate an image with high-quality dehalation effect, ensure the coordination and consistency of details and light sources, and provide a solid foundation for subsequent image post-processing.

[0136] S6. Post-process the de-haloated image to obtain a de-haloated image.

[0137] To ensure that the image has a good visual effect after the dehalation process, the embodiment of the present invention further optimizes the dehalation image through a post-processing module after completing the image fusion step to adjust the brightness, contrast and color saturation, thereby generating a natural dehalation image with high visual quality. Generally, the image after dehalation and image fusion processing may change in brightness distribution and contrast, so it is necessary to refine and adjust various visual parameters of the image through the post-processing step to meet the visual perception needs of the human eye.

[0138] In this embodiment, the brightness of the dehaloated image is first adjusted to ensure that the brightness of the dehaloated image meets the nighttime visual effect. As an option, the brightness adjustment can be expressed as a linear transformation of the pixel brightness, and the specific formula is:

[0139] I'=α·I+β

[0140] Wherein, I' is the adjusted brightness, I is the brightness value of the original image, α is the brightness gain coefficient, and β is the brightness offset. Generally, the values ​​of the brightness gain coefficient α and the offset β can be determined by experiments to ensure that the brightness of the image in the bright and dark areas is balanced.

[0141] In some embodiments, the values ​​of α and β can be dynamically adjusted based on the average brightness value of the image to adapt to the brightness requirements of different scenes. For example, if the overall brightness of the image is low, the value of α can be appropriately increased to improve the overall brightness of the image.

[0142] Next, this embodiment optimizes the contrast of the image to enhance the detail expression of the image. Specifically, contrast adjustment can usually be implemented through a nonlinear mapping function to highlight the difference between bright and dark areas in the image. As a possible implementation method, contrast adjustment can use the following nonlinear transformation formula:

[0143]

[0144] Among them, I” is the adjusted contrast, I' is the image after brightness adjustment, I mean Represents the average brightness value of the image, I max and I min are the maximum and minimum brightness values ​​of the image, respectively, and γ is the contrast gain parameter. By adjusting the value of γ, the contrast of the image can be enhanced or weakened to achieve a balanced visual effect.

[0145] As an option, the contrast optimization can be refined by the brightness contrast of the local area, that is, the image is divided into multiple small areas, and the average brightness and contrast parameters of each area are calculated respectively, so as to perform adaptive contrast adjustment in each area. This can avoid the overall image distortion caused by a single contrast adjustment, and is suitable for night scenes with complex light sources or halo distribution.

[0146] After the brightness and contrast are adjusted, the present embodiment also appropriately adjusts the color saturation of the dehalation image to ensure the color authenticity and visual consistency of the image. In general, the color saturation can be achieved by stretching the color components of the image. Specifically, in a possible implementation, the color saturation is adjusted using the following formula:

[0147] S′=S·(1+δ)

[0148] Among them, S' represents the adjusted color saturation, S is the color saturation of the original image, and δ is the saturation gain coefficient. By adjusting the δ value, the color expression of the image can be enhanced, especially after removing the halo, the color of the night image can be restored to make it close to the color effect of the real scene.

[0149] In some embodiments, the adjustment of color saturation can be adaptively adjusted in combination with the main hue of the image. As an option, the saturation gain parameters of different regions can be determined according to the average hue of the image to achieve balanced color adjustment in different color regions, thereby avoiding oversaturation or color distortion.

[0150] After adjusting the brightness, contrast and color saturation, the embodiment of the present invention further optimizes the dynamic range of the image to ensure that the bright and dark details of the dehalation image are fully displayed. Specifically, the brightness distribution of the image can be finely adjusted by local histogram equalization so that the highlight and shadow areas of the image achieve visual balance. In some embodiments, a layered processing method can be used to adjust the dynamic range of different brightness areas respectively, so that the dehalation image has clear details at different brightness levels.

[0151] Through the above post-processing steps, the embodiment of the present invention can ensure that the brightness, contrast, color and dynamic range of the de-halation image achieve visual optimization effects, and provide users with clear and natural night images.

[0152] In general, the present invention effectively removes the halo effect in night images by performing halo region extraction, multi-scale decomposition, adaptive subspace mapping, image fusion and post-processing on the original image. Through multi-scale feature extraction and adaptive subspace mapping, the present invention can accurately separate halo and background information to ensure that the details and structure of the image are preserved. In the image fusion and post-processing stage, the brightness, contrast and color saturation of the image are further optimized, so that the image after dehalation is visually natural and clear, suitable for night imaging application scenarios, and significantly improves the quality and usability of night images.

[0153] Experimental analysis:

[0154] In order to verify the effectiveness and superiority of the night image dehalation processing method based on subspace mapping in practical applications, the present invention has conducted extensive experimental comparisons on multiple public standard data sets. The experimental results show that the method of the present invention is significantly superior to the current mainstream dehalation and defogging methods in multiple key evaluation indicators, especially in image detail restoration and background information preservation.

[0155] 1. Experimental Dataset

[0156] This experiment uses the Flare7k[9] dataset as the main test platform. This dataset contains a wealth of night scene images, including real halo scenes and synthesized halo images, and is highly representative and challenging. During the test, experiments were conducted on real and synthetic data subsets to ensure the versatility and adaptability of the algorithm. The diversity of the Flare7k dataset enables the experimental results to better reflect the dehalation effect of the present invention under different types of halo conditions.

[0157] 2. Experimental Setup

[0158] The method of the present invention is based on modular design and performs six steps in sequence according to the standard process: image acquisition, halo region extraction, multi-scale decomposition, subspace mapping, image fusion and post-processing. The parameters of each module are optimized according to the actual situation of the data set to ensure the robustness and generalization ability of the algorithm. The comparison method includes representative algorithms in the current field of halo removal, such as DCP[7], Uformer[1], Flare7kpp[5], Zhou[6] and other methods, to reflect the performance of the method of the present invention under different halo removal strategies.

[0159] 3. Evaluation indicators

[0160] In order to comprehensively evaluate the dehalation effect, the present invention adopts the following three evaluation indicators:

[0161] Peak Signal-to-Noise Ratio (PSNR): It is used to measure the similarity between the de-haloated image and the non-haloated reference image at the pixel level. The higher the PSNR value, the smaller the image noise and the better the de-haloating effect.

[0162] Structural Similarity (SSIM): Evaluates the structural similarity between the de-haloated image and the reference image. The higher the SSIM value, the closer the image details and structural information are to the de-haloated image.

[0163] Image Perceptual Similarity (LPIPS): A perceptual metric based on deep learning that is used to evaluate the visual similarity between a de-haloated image and a reference image. The lower the LPIPS value, the better the visual perception effect.

[0164] 4. Experimental results and analysis

[0165] The following table shows the experimental results of the proposed method on the Flare7k dataset, and quantitatively compares it with other methods. It can be seen from the table that the PSNR, SSIM and LPIPS indicators of the proposed method on both synthetic and real datasets are optimal, showing excellent dehalation effect and image quality.

[0166]

[0167] In the synthetic dataset, the proposed method achieves a PSNR of 31.00, a SSIM of 0.968, and a LPIPS of 0.015, all of which are the best performances, significantly outperforming current advanced methods such as Uformer and Restormer. In contrast, the traditional DCP and Wu methods have lower PSNRs, indicating that their processing effects on synthetic halo images are limited.

[0168] On the real data set, the PSNR of the method of the present invention is 27.35, the SSIM is 0.895, and the LPIPS is 0.043, which is also better than other comparison methods. This shows that the present invention has better detail restoration and visual effects in real scenes. Methods such as Uformer and Restormer have some improvements in dehalation processing, but the LPIPS values ​​are relatively high, and the visual perception effect is not as good as the method of the present invention.

[0169] 5. Visual comparative analysis

[0170] like Figure 6 As shown in the figure, the method of the present invention can better maintain background details and natural color transition while removing halo. Compared with the methods such as DCP, Uformer, Flare7kpp and Zhou, the details of the image processed by the method of the present invention are clearer around the light source, and the brightness and contrast of the background area are more uniform. The traditional DCP method is prone to color cast and detail loss in high-brightness areas, and although the Uformer and Flare7kpp methods improve the dehalation effect, there is still blurring at the halo edge, and the detail recovery effect is not as good as the method of the present invention.

[0171] The night image dehalation processing method based on subspace mapping of the present invention shows excellent performance in multiple key indicators. Its adaptive subspace mapping and multi-scale decomposition technology enable the present invention to accurately distinguish halo from background information, ensuring detail recovery and background fidelity. Compared with existing methods, the present invention achieves optimal performance in PSNR, SSIM and LPIPS, especially showing obvious advantages in visual perception effects, and is suitable for complex night scenes. The modular design of the method further enhances its application flexibility and adaptability, providing users with a high-quality dehalation image experience.

[0172] Please see attached Figure 7 The present invention also provides a night image halo removal processing device based on subspace mapping, which is designed as a modular structure to facilitate subsequent maintenance, upgrading and expansion. The device achieves accurate removal of halo in night images through the collaborative work of multiple functional modules, thereby improving the clarity and visual quality of the image. The device includes an image acquisition module 100, a halo region extraction module 200, a multi-scale decomposition module 300, a subspace mapping module 400, an image fusion module 500 and a post-processing module 600.

[0173] Image acquisition module 100

[0174] The image acquisition module 100 is responsible for capturing the original nighttime image containing the halo, providing a data basis for subsequent processing. Generally, a high-sensitivity camera or a low-light imaging sensor is used for nighttime image acquisition to preserve the halo area and surrounding details. The module adjusts parameters such as exposure, gain, and white balance according to scene requirements to ensure that the brightness and contrast of the image meet the requirements for halo area extraction.

[0175] The collected nighttime images are transmitted to the halo region extraction module 200 through the output interface of the image acquisition module 100 for preliminary analysis and segmentation.

[0176] Halo region extraction module 200

[0177] The halo region extraction module 200 is mainly composed of a mask threshold segmentation unit, which is used to identify the halo region based on the brightness threshold and generate a halo mask. By decomposing the color channel of the image (usually the green channel is selected), the module sets a threshold and marks the area with a brightness value higher than the threshold as a halo candidate area. Further, morphological operations such as dilation and erosion are used to remove noise and small pseudo halo areas to generate an accurate halo mask.

[0178] The generated halo mask is used to mark the spatial position of the halo and is transmitted to the multi-scale decomposition module 300 together with the original image to provide input for subsequent detail separation.

[0179] Multi-scale decomposition module 300

[0180] The multi-scale decomposition module 300 includes a neural network processing operator and uses a U-shaped structure for feature decomposition to decompose the image into detail layers and base layers of different scales. Through layer-by-layer convolution and pooling operations (such as 3×3 convolution kernels and maximum pooling), the module extracts feature maps of different resolutions. The skip connection directly passes the high-resolution features of the encoding stage to the decoding stage to ensure that more detail information is retained in the high-resolution layer.

[0181] The decomposed image detail layer is transmitted to the subspace mapping module 400, and the base layer is separated from the detail layer as background information.

[0182] Subspace mapping module 400

[0183] The subspace mapping module 400 uses an adaptive algorithm processing unit to perform subspace mapping on the detail layer to achieve separation of halo and effective signal. The module converts the detail layer to a low-dimensional subspace through a mapping matrix and decomposes it into a series of basis vectors. According to the different halo characteristics, the weight coefficients of the basis vectors are adjusted to achieve the distinction between halo and background in the subspace. The supervised learning mechanism optimizes the mapping matrix and weight coefficients through a loss function to ensure that the image after dehalation is as close to the reference image as possible.

[0184] After being processed by the subspace mapping module 400 , the dehalated detail layer is transmitted to the image fusion module 500 together with the base layer.

[0185] Image fusion module 500

[0186] The image fusion module 500 includes an image fusion output mapping unit, which is responsible for fusing the detail layer after dehalation with the base layer to generate a complete dehalation image. The module weights and combines the feature information of the detail layer and the base layer at each scale, and ensures that the high-frequency area shows more details and the low-frequency area remains smooth through adaptive weight adjustment. For the light source area in the base layer, the fusion module also performs nonlinear light source enhancement to ensure that the brightness of the image after dehalation is consistent with the original image. Figure 1 To.

[0187] The generated dehalation image is output from the image fusion module 500 to the post-processing module 600 to optimize the visual effect.

[0188] Post-processing module 600

[0189] The post-processing module 600 is composed of a brightness adjustment unit and a contrast adjustment unit, and is used to optimize the brightness, contrast and color of the image after dehalation. By adjusting the brightness parameter and the offset, the module balances the bright and dark areas of the image; the contrast optimization module uses a nonlinear mapping function to improve the detail level of the image; the color saturation adjustment unit enhances the image saturation according to the color component to ensure the naturalness of the color of the night image.

[0190] The image optimized by the post-processing module 600 is output to a final display or storage device, providing the user with clear and natural night images.

[0191] Through the synergistic effect of the above modules, this device can effectively remove the halo of night images. Specifically, the image acquisition module collects night images and transmits them to the halo area extraction module to generate a halo mask. The multi-scale decomposition module decomposes the image into a detail layer and a base layer by decomposing features of different scales. The subspace mapping module separates the halo features in the subspace and removes the halo information in the detail layer. Subsequently, the image fusion module fuses the de-haloed detail layer with the base layer to generate a de-haloed image. Finally, the post-processing module adjusts the brightness, contrast and color of the image to output a high-quality de-haloed image.

[0192] Please see attached Figure 8 The present invention further provides a computer device 40, comprising: a processor 41 and a memory 42, wherein the memory 42 stores a computer program executable by the processor, and when the computer program is executed by the processor, the above method is executed.

[0193] The present invention further provides a storage medium 43 on which a computer program is stored. When the computer program is run by the processor 41, the above method is executed.

[0194] Among them, the storage medium 43 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0195] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for removing halo from night images based on subspace mapping, characterized in that: The following steps are involved: Get the original image including the halo; Extract the halo area of ​​the original image and generate a halo mask; Perform multi-scale decomposition on the image to separate the image information into detail layers of different scales; The detail layer is converted into subspace, and the halo information is identified and separated by an adaptive subspace mapping algorithm; The dehaloated detail layer is fused with the base layer to generate a dehaloated image; The dehalated image is post-processed to obtain a dehalated image.

2. The method for removing halo from night images based on subspace mapping according to claim 1, characterized in that: The step of extracting the halo area from the original image and generating a halo mask comprises: Perform color channel decomposition on the original image, decomposing the image into three channels: red, green, and blue; Set the threshold for halo detection in the green channel and extract the highlight area based on the threshold; Mark the extracted highlight area as the halo candidate area and generate a preliminary halo mask; Perform morphological processing on the preliminary halo mask to generate an accurate halo mask; The precise halo mask is superimposed on the original image to record the spatial location of the halo area.

3. The night image dehalation processing method based on subspace mapping according to claim 1 is characterized in that: The step of performing multi-scale decomposition on the image comprises: The image is downsampled step by step through the U-shaped transformation unit to generate feature maps of different resolutions; In each downsampling level, an encoding module containing convolution operations and attention mechanisms is used to extract local and global features; In the downsampling process, the image resolution is gradually reduced and the number of channels is increased so that the low-resolution feature map contains global information and the high-resolution feature map retains edges and details; The feature maps at each level are passed to the decoder through skip connections to maintain key information at different scales; The decoder upsamples the feature map layer by layer, restores it to the original resolution, and fuses the details of each layer; The weights of halo and effective signal are adjusted according to the distribution of different frequency characteristics, so that the high-resolution layer focuses on edge details and the low-resolution layer processes the halo outline, thereby achieving the separation of halo and effective signal.

4. The night image dehalation processing method based on subspace mapping according to claim 1 is characterized in that: The step of converting the detail layer into a subspace and identifying and separating the halo information by an adaptive subspace mapping algorithm comprises: The decomposed detail layer is input into the subspace mapping unit, and the detail layer features are converted into a low-dimensional subspace using the subspace decomposition operation; The transformed detail layer is decomposed into multiple orthogonal basis vectors in the subspace, each basis vector represents a different feature dimension, and the basis vector parameters are adaptively adjusted so that the basis vectors can correspond to the halo information and background information respectively; By using the supervised learning mechanism, the adaptive subspace mapping algorithm is optimized through the loss function, and the basis vector weights are adjusted to minimize the difference between the de-haloated image and the reference image without halo.

5. The method for removing halo from night images based on subspace mapping according to claim 4, characterized in that: The step of converting the detail layer into a subspace and identifying and separating the halo information by an adaptive subspace mapping algorithm further comprises: The subspace mapping parameters are dynamically adjusted to adapt the basis vectors to the changes in image features, thereby achieving real-time separation of halo features in different scenes.

6. The method for removing halo from night images based on subspace mapping according to claim 1, characterized in that: The step of fusing the de-haloated detail layer with the base layer to generate a de-haloated image comprises: The de-haloated detail layer and base layer are sent back to the neural network through skip connections; Using the image fusion technology of neural network, the de-haloated detail layer and the base layer are gradually fused at multiple scales to maintain the naturalness and detail integrity of the output image. In the fusion process, the light source is supplemented for the background part. First, the light source area of ​​the original image is extracted by thresholding, and the light source area is separated from the background. The separated light source area is supplemented in nonlinear space and the processed light source information is integrated into the background without halo; The fused image is converted into a standard color space through the inverse transformation of nonlinear processing to generate a reconstructed image that contains the background and light source but removes the halo.

7. The night image dehalation processing method based on subspace mapping according to claim 1 is characterized in that: The step of post-processing the de-haloated image to obtain the de-haloated image comprises: Adjust image brightness and contrast to enhance the visual effect after dehalation.

8. A night image dehalation processing device based on sub-space mapping, executing the night image dehalation processing method based on sub-space mapping according to any one of claims 1 to 7, characterized in that: include: An image acquisition module, used for acquiring an original image including a halo; A halo region extraction module, used for extracting the halo region from the original image and generating a halo mask; A multi-scale decomposition module, used to perform multi-scale decomposition on the original image and separate the image information into detail layers of different scales; A subspace mapping module, used to convert the detail layer into a subspace, and identify and separate halo information through an adaptive subspace mapping algorithm; An image fusion module is used to fuse the detail layer after dehalation with the base layer to generate a dehalation image; The post-processing module is used to optimize the brightness, contrast and color of the de-haloated image to obtain the de-haloated image.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the night image dehalation processing method based on sub-space mapping is implemented as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the night image dehalation processing method based on sub-space mapping as described in any one of claims 1 to 7 is implemented.

Images