Nighttime defogging methods and devices based on nighttime physical perception and grayscale world

Abstract

This invention discloses a nighttime dehazing method and apparatus based on nighttime physical perception and the grayscale world. The method includes: firstly, a foggy image is processed by a local residual learning module and a compression-excitation module to extract important image feature information and learn weights; the important image feature information is input into a nighttime physical perception module to obtain feature information of a clear nighttime image including glow and transmission maps; the feature information of the clear nighttime image is passed to a feature processing group composed of several subsequent feature processing blocks, and the feature maps output by multiple feature processing blocks are connected in the channel direction to obtain third feature information; the third feature information is input into a global residual learning recovery module to finally obtain the recovered nighttime dehazed image result. This invention can effectively improve the dehazing performance in foggy nighttime scenes, while eliminating the degradation effect of nighttime fog on images and realistically restoring the color and texture details of the image.

Description

Technical Field

[0001] This invention belongs to the technical field of image classification, specifically relating to a nighttime defogging method and apparatus based on nighttime physical perception and grayscale world. Background Technology

[0002] Nighttime haze is a common atmospheric phenomenon that significantly reduces visibility and image quality. In nighttime haze imaging, the reduced ambient lighting and interference from artificial light sources lead to images affected by haze and various degradation types, including light scattering, non-uniform illumination, glow, and noise artifacts. These degradations not only increase the complexity of nighttime dehazing tasks but also hinder the effectiveness of subsequent visual processing tasks, including segmentation, autonomous driving, and object detection. Early nighttime dehazing methods directly transferred daytime methods to nighttime scenarios. However, due to the low brightness, complex light sources, and severe color cast in nighttime scenes, daytime dehazing methods performed poorly at night, often resulting in severe loss of detail and residual haze in the dehazed images. To further improve the dehazing effect in nighttime scenes, researchers proposed some prior hypotheses based on statistics and designed dehazing methods using these hypotheses. However, these hypotheses are not always valid in real-world scenarios, thus failing to effectively avoid artifacts and glow. With the development of deep learning, many new nighttime dehazing methods have emerged. They use deep networks to predict parameters in physical models such as transport maps or atmospheric light to recover clear images, or they directly use end-to-end methods to obtain clear images. However, these methods generally have two problems: (1) They do not use physical models of nighttime atmospheric scattering as guidance or only use physical models of nighttime atmospheric scattering in surface space. The former will lead to poor generalization of the model and poor performance in the real world, while the latter will make the accumulation of experimental errors unavoidable and affect the final dehazing effect. (2) They cannot handle the glow, uneven illumination distribution and strong color cast interference caused by multiple artificial light sources at night, which usually leads to problems such as image color distortion, loss of detail and obvious artifacts.

[0003] As can be seen from the above problems, it is urgent to solve the problem of how to utilize the physical model of nighttime atmospheric scattering in the processing of foggy nighttime images while avoiding error accumulation, and at the same time dealing with the serious degradation problems such as glow and color shift caused by artificial light sources. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a nighttime defogging method and device based on nighttime physical perception and grayscale world, which improves the defogging performance in foggy nighttime scenes, and can eliminate and reduce the impact of common nighttime degradation such as fog, glow, and color cast on images, realistically and delicately restore the color and detail texture of the image, and improve the realism and visibility of the defogging image.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a nighttime defogging method based on nighttime physical perception and the grayscale world, comprising the following steps:

[0007] The foggy image first passes through a local residual learning module and a compression-excitation module to extract important image feature information and learn weights;

[0008] The important image feature information is input into the nighttime physical perception module to obtain feature information of a clear nighttime image containing glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. The glow region information and global information are combined and activated using an activation function to obtain first feature information based on glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain second feature information. The first feature information and the second feature information are fused and deconvolution is performed to obtain feature information of the clear nighttime image.

[0009] The feature information of the clear night image is passed to a feature processing group consisting of several feature processing blocks connected together. The feature maps output by multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence.

[0010] The third feature information is input into the global residual learning recovery module to finally obtain the recovered nighttime dehazed image result.

[0011] As a preferred technical solution, the nighttime physical sensing module is obtained through the following steps:

[0012]

[0013] Where I(x) represents a foggy nighttime image, L(x) and η(x) represent the ambient lighting intensity and the color deviation of the active light source, R(x) represents the reflectivity, t(x) represents the transmission map, x represents the pixel index, and J(x) represents a clear nighttime image, as shown below:

[0014] J(x) = R(x)L(x);

[0015] Based on the above formula, we obtain the following expression:

[0016]

[0017] Assuming the kernel size for feature extraction is k, the following formula is obtained:

[0018]

[0019] in, The symbol represents the convolution operator, and ⊙ represents the element-wise multiplication operation. This introduces color distortion removal loss based on the assumption of color constancy in the gray world, making... It is approximated as a constant matrix with all elements equal to 1; then, through several algebraic operations in matrix-vector form, the following formula is obtained:

[0020]

[0021] Where, K, T and T represent k, J, I, L and respectively. The matrix-vector form, where 1 represents a matrix of appropriate size consisting entirely of 1s, derived from the diagonal vectors of the diagonal matrix T and the vectorized representation. Alignment;

[0022] Decomposing matrix KT into the product of two matrices, denoted as GK, since K, T, and G are all unknown variables, this decomposition can be formally represented as solving an underdetermined system of equations, thus yielding the following formula:

[0023]

[0024] Where K represents the feature extraction operation. The formula describes the relationship between sharp images and foggy nighttime images in the feature space. The following method is used to analyze sharp nighttime images... Modeling features:

[0025]

[0026] Based on the above modeling, the nighttime physical perception module is constructed as a dual-branch structure.

[0027] As a preferred technical solution, the local residual learning module bypasses information in foggy or low-frequency regions by connecting multiple local residuals.

[0028] As a preferred technical solution, the feature processing block consists of a local residual learning module, a compression excitation module, and a nighttime physical perception module. The input feature information is first processed by a convolutional layer and then enters the nighttime physical perception module. The constructed nighttime physical perception module processes the image features in the feature space and outputs the feature information of a clear image. Finally, it is combined with the input feature information as the output of the feature processing block.

[0029] As a preferred technical solution, the feature processing group consists of multiple feature processing blocks, which process the input feature information sequentially, and finally combine the feature information of the multiple feature processing blocks as the output of the feature processing group.

[0030] As a preferred technical solution, the grayscale world assumption is specifically as follows:

[0031] Inspired by the gray-world hypothesis algorithm, a loss function based on the gray-world hypothesis algorithm was designed. This function leverages the algorithm's powerful ability to handle color casts to assist the nighttime physical perception module in recovering clear, fog-free nighttime images with rich detail. The loss function is defined as follows:

[0032] L color This technology is designed to mitigate color cast issues in foggy nighttime images. Inspired by the grayscale world hypothesis, it works by comparing information between different color channels. color Represented as:

[0033]

[0034] Where (m,n) represents a set of channels, J m J represents the average value of the m channels of the image after processing by multiple feature processing groups and the global residual learning and recovery module. n This represents the average value of the n channels of the image after processing by multiple feature processing groups and the global residual learning and recovery module, while RGB represents the R, G, and B color channels of the image, respectively.

[0035] As a preferred technical solution, in the global residual learning recovery module, all feature maps output by several feature processing block structures are connected in the channel direction, and then multiplied by the adaptive learning weights obtained in the network to fuse features. By retaining the information of the low layer and passing it to the deep layer, the network pays more attention to the effective information of thick fog areas, high frequency textures and color fidelity.

[0036] Secondly, the present invention provides a nighttime defogging system based on nighttime physical perception and grayscale world, applied to the aforementioned nighttime defogging method based on nighttime physical perception and grayscale world, including a feature information extraction module, a first processing module, a second processing module, and an image restoration module;

[0037] The feature information extraction module is used to first extract important image feature information from the foggy image through the local residual learning module and the compression-excitation module, and then learn the weights.

[0038] The first processing module is used to input the important image feature information into the nighttime physical perception module to obtain feature information of a clear nighttime image including glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. After combining the glow region information and global information, an activation function is used to activate them to obtain first feature information based on the glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain second feature information. The first feature information and the second feature information are fused, and after performing a deconvolution operation, the feature information of the clear nighttime image is obtained.

[0039] The second processing module is used to pass the feature information of the clear night image to a feature processing group composed of several subsequent feature processing blocks. The feature maps output by the multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module, and a night physical perception module connected in sequence.

[0040] The image restoration module is used to input the third feature information into the global residual learning restoration module to finally obtain the restored nighttime dehazing image result.

[0041] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores computer program instructions that can be executed by the at least one processor, which enable the at least one processor to perform the nighttime defogging method based on nighttime physical perception and grayscale world.

[0045] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the nighttime defogging method based on nighttime physical perception and grayscale world.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] (1) The present invention sets up a color distortion removal loss function including the assumption of grayscale world color constancy and a nighttime physical perception module. The nighttime physical perception model avoids the degradation of model effect caused by the accumulation of model errors by applying the nighttime atmospheric scattering physical model in the feature domain and combining the color distortion removal loss function of the assumption of grayscale world color constancy. At the same time, it effectively solves the image degradation problem caused by nighttime fog, glow, color cast, etc., and restores a clearer and more delicate foggy image.

[0048] (2) The nighttime physical perception module provided by the present invention uses a dual-branch structure to predict two important factors at night: glow and transmission map. For glow, the present invention combines global average pooling and global max pooling, so that the network can better pay attention to and process glow areas. For transmission map, due to its non-uniform characteristics, the present invention uses multi-layer convolution combined with appropriate activation functions to fit it. Finally, the two are combined to obtain high-quality clear nighttime image features.

[0049] (3) The color distortion removal loss function of the gray world color constancy hypothesis used in this invention introduces the gray world hypothesis algorithm as the loss function, effectively utilizing the superiority of the gray world algorithm in dealing with image color shift problems, and assisting the nighttime physical perception model to focus more on solving the more important and difficult image degradation problems, thereby restoring a more realistic nighttime dehazed image, solving the serious color difference problem caused by multiple artificial light sources in nighttime foggy scenes. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a nighttime defogging method based on nighttime physical perception and a grayscale world, according to an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram of the color distortion removal loss under the grayscale world color constancy assumption and the nighttime defogging network of the nighttime physical perception module in an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the nighttime physical sensing module structure in an embodiment of the present invention;

[0054] Figure 4This is a schematic diagram illustrating the effect of the grayscale world hypothesis algorithm in an embodiment of the present invention. The first row of images shows hazy, color-shifted images under various complex scenes, and the second row of images shows the color-shifted images obtained after processing by the grayscale hypothesis algorithm. Furthermore, Figure 4 Parts (a) and (b) in the image are selected from self-synthesized nighttime images. Figure 4 Part (c) in the image is taken from real-world nighttime images;

[0055] Figure 5 This is a comparison chart of the defogging effects of this invention and several advanced defogging algorithms;

[0056] Figure 6 This is a schematic diagram of the nighttime defogging system of the color distortion removal loss and the nighttime physical perception module based on the grayscale world color constancy assumption in an embodiment of the present invention.

[0057] Figure 7 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0059] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0060] Please see Figure 1 This embodiment proposes a nighttime defogging method based on nighttime physical perception and the grayscale world, which includes the following steps:

[0061] S1. The foggy image first goes through the local residual learning module and the compression-excitation module to extract important image feature information and learn weights.

[0062] Furthermore, the local residual learning module allows less important information such as fog or low-frequency regions to be bypassed through multiple local residual connections, while the main network focuses on effective information. Experimental results show that the local residual learning module can further improve network performance and training stability.

[0063] Furthermore, the compression-excitation module first compresses the spatial information of each channel of the image feature map U into a single value, that is, it obtains a 1*1*C vector from the H*W*C size U. Then, it applies a set of fully connected layers to the vector for weight adjustment to obtain a 1*1*C channel attention vector. Finally, it weights the channel attention vector onto U to form a weighted image feature map.

[0064] S2. Input the important image feature information into the nighttime physical perception module to obtain feature information of a clear nighttime image containing glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. After combining the glow region information and global information, an activation function is used to activate them to obtain the first feature information based on the glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain the second feature information.

[0065] It is understood that the nighttime physical perception module in this embodiment consists of two branches: a glow prediction branch and a transmission map prediction branch. The two branches are designed according to the characteristics of glow and transmission maps in the nighttime atmosphere, which can effectively process glow and transmission maps and provide support for the application of nighttime atmospheric scattering models in the feature space.

[0066] In one specific embodiment Figure 2 The paper demonstrates the color distortion removal loss under the assumption of color constancy in the world and the nighttime dehazing network of the nighttime physical perception module. It also shows the dehazing effect of the network on a foggy nighttime image. Figure 2 G1, G2, and G3 are three consecutive feature processing groups. Each feature processing group consists of three feature processing blocks: B1, B2, and B3. SE is the compression-excitation module, NPAM is the nighttime physical perception module, ReLU is the activation layer using ReLU as the activation function, and Concatenate is the feature map connection operation in the channel direction.

[0067] In one specific embodiment Figure 3The nighttime physical perception module consists of two branches: the top branch is for glow prediction, and the middle branch is for transmission map prediction. In the glow prediction branch, image feature information undergoes global max pooling and global average pooling respectively. The two processed results are then concatenated and input into a 1×1 convolution. The result is activated using the sigmoid function. The transmission map prediction branch mainly consists of multiple convolutional layers and activation functions to fit the transmission map. The obtained information is processed according to the following formula, where… F represents sharp image features, while F represents foggy image features. For the transmission graph predicted by the transmission graph branches, The glow feature information obtained through the glow prediction branch is then passed through a final deconvolution layer to obtain the features of the nighttime dehazed image.

[0068]

[0069] Furthermore, Figure 3 This means subtracting each element from 1. This indicates that the elements are multiplied pixel by pixel. This indicates that the elements are incremented pixel by pixel.

[0070] Understandably, the nighttime physical perception module can enhance and acquire features of clear nighttime images. This module includes a glow prediction branch and a transmission map prediction branch. It utilizes a nighttime atmospheric scattering model in the feature space to process the fogging process, improving the interpretability of the feature space and thus better removing nighttime haze. The referenced nighttime atmospheric scattering model is as follows:

[0071]

[0072] Where I(x) represents a foggy nighttime image, L(x) and η(x) represent the ambient lighting intensity and color deviation of the active light source, R(x) represents reflectivity, t(x) represents the transmission map, x represents the pixel index, and J(x) represents a clear nighttime image.

[0073] J(x) = R(x)L(x);

[0074] It was observed that recovering a fog-free image J from a foggy observation image I is an ill-posed problem that depends on accurate estimation of latent variables: atmospheric light intensity L, ambient lighting color η, and transmission rate t.

[0075] Based on the above formula, the present invention can obtain the following formula:

[0076]

[0077] Then, assuming the kernel size for feature extraction is k, we can obtain the following formula:

[0078]

[0079] in, The symbol represents the convolution operator, and ⊙ represents the element-wise multiplication operation. Inspired by prior methods, this invention finds that the gray-world algorithm can effectively solve the color distortion problem in foggy images at night. Therefore, to simplify the feature decoupling process, this invention introduces a color distortion removal loss based on the gray-world color constancy assumption, making... It is approximated as a constant matrix with all elements equal to 1. Then, through several algebraic operations in matrix-vector form, the present invention yields the following formula:

[0080]

[0081] Here, K, T and T represent k, J, I, K and respectively. The matrix-vector form of .1 represents a matrix of appropriate size consisting entirely of 1s. The diagonal vectors from the diagonal matrix T and their vectorized representations... Alignment.

[0082] Next, this invention decomposes matrix KT into the product of two matrices, denoted as GK. Since K, T, and G are all unknown variables, achieving this decomposition can be formally characterized as solving an underdetermined system of equations. In an underdetermined system, the number of equations is less than the number of unknown variables, which means there are infinitely many solutions that satisfy the given equations. Therefore, the existence of G is guaranteed because the system always has at least one solution. Thus, we can obtain:

[0083]

[0084] Where K can be considered as a feature extraction operation. This formula describes the relationship between a clear image and a foggy nighttime image in the feature space. Due to the powerful representational capabilities of traditional neural networks (CNNs), this invention proposes a nighttime physical perception module and combines it with a deep CNN to approximate G and... Corresponding features.T Therefore, useful discriminative features can be better estimated for sharp image restoration. To this end, the proposed NPAM comprises two parts. The first part is feature learning. and The key components G and illuminance features associated with the transmission map and illumination characteristics of foggy nighttime images are primarily approximated. When features When the number of channels in the image matches the number of channels in the module input feature F, the present invention can process clear nighttime images in the following way. Modeling features:

[0085]

[0086] Based on the above construction, NPAM is constructed as a two-branch structure. For example... Figure 2 As shown, the topmost branch, the illuminance map construction branch, is specifically used to construct L. Since nighttime illumination mainly comes from multiple artificial light sources, the number of L in each scene is uncertain and is often accompanied by glow and color distortion. Therefore, this invention proposes to use Global Max Pooling (GMP) to focus information on the light sources and Global Average Pooling (GAP) to eliminate unnecessary information in the feature space. The middle branch, the nighttime haze transmittance estimation branch, is specifically used to predict the corresponding transmittance map. Due to the non-uniformity of the transmission map, this invention chooses multi-layer convolution instead of global pooling operators to approximate the pixel-perceived transmission map of nighttime hazy scenes. Using the proposed NPAM, interpretable features f can be derived from the input features F to recover hazy nighttime images. Unlike using only GAP(·) to predict latent features, NPAM thoughtfully integrates the corresponding physical features of irregular nighttime lighting conditions and significant color distortion to achieve stable dehazing at night. This method facilitates the estimation of more relevant features within a dual-interaction framework in nighttime conditions.

[0087] Specifically, the method includes the following steps: First, in the glow prediction branch, this invention uses global max pooling (GMP) and global average pooling to obtain glow region information and global information, respectively. Then, a concat operation is used to combine the two, and the combined features are then subjected to a 1×1 convolution followed by activation using the sigimoid function to finally obtain feature information based on glow region information. Second, in the transmission map prediction branch, since the information of the transmission map is not uniform in the image, this invention uses several convolutional layers and ReLU and sigimoid activation functions to fit the transmission map. Finally, the information from the two branches is fused together, and a deconvolution operation is performed to obtain the feature information of the clear nighttime image.

[0088] L color To mitigate color cast in foggy nighttime images, this algorithm is inspired by the gray-world hypothesis and works by comparing information across different color channels. In gray-world correlation algorithms, the color in each sensor channel is averaged as gray across the entire image, which has been proven to effectively reduce color cast. color It can be represented as:

[0089]

[0090] S3. The feature information of the clear night image is passed to a feature processing group consisting of several feature processing blocks connected together. The feature maps output by the multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence.

[0091] Furthermore, the feature processing block consists of a local residual learning module, a compression-excitation module, and a nighttime physical perception module. The input feature information is first processed by a convolutional layer and then enters the nighttime physical perception module. In the feature space, the nighttime physical perception module, which is constructed under the guidance of the above-mentioned nighttime atmospheric scattering physical model, processes the image features and outputs the feature information of a clear image. Finally, it is combined with the input feature information as the output of the feature processing block.

[0092] The feature processing group consists of multiple feature processing blocks. The multiple feature processing blocks process the input feature information in sequence, and finally combine the feature information of the multiple feature processing blocks as the output of the feature processing group.

[0093] Understandably, sequential feature processing groups increase the depth and expressiveness of the network, while jump connections help the network avoid training difficulties.

[0094] S4. Input the third feature information into the global residual learning recovery module to finally obtain the recovered nighttime dehazing image result.

[0095] Furthermore, in the global residual learning recovery module, all feature maps output by several feature processing block structures are connected in the channel direction, and then multiplied by the adaptive learning weights obtained in the network to fuse features. Thus, the present invention can retain low-level information and pass it to the deep layers. Due to the existence of the weight mechanism, the network pays more attention to effective information such as thick fog areas, high-frequency textures and color fidelity.

[0096] In one embodiment, training is performed on an RTX3090 with a batch size of 2. The Adam optimizer is used during training, and a cosine annealing strategy is employed. Each image is cropped to a size of 256×256, and the resulting images are used as input to the network for training.

[0097] This embodiment can better address various degradation factors in foggy nighttime scenes, such as uneven lighting, glow, low brightness, severe loss of detail, and color cast, resulting in dehazed images with rich detail and texture, correct color cast processing, and no artifacts or fog residue. Figure 5As shown, NIDM, GS, MRP, OSFD, and RIDCP produced severe artifact retention when processing bowling ball images. Furthermore, they failed to properly handle color casts and remove fog effectively when processing images of vehicles traveling on roads at night, resulting in significant differences from the reference image. Faced with the various challenges of foggy nighttime scenes, none of these methods successfully addressed the various factors affecting image degradation. However, the method provided in this embodiment not only completely removed the fog from the scene but also eliminated artifact retention, achieving a balanced distribution of bright and dark areas, rich and delicate details, and making the overall image most similar to a standard clear image. The comparison demonstrates that, under the same foggy nighttime scene, the color distortion removal loss based on the grayscale world color constancy assumption provided in this embodiment, along with the nighttime defogging method of the nighttime physical perception module, can better solve the interference problems such as color casts, uneven illumination, and glow in nighttime scene images.

[0098] Table 1

[0099]

[0100] To more concretely demonstrate the technical effectiveness of this embodiment, as shown in Table 1, the objective evaluation results of the method are presented. Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Patch Similarity (LPIPS) are used as evaluation metrics. Red and blue represent the first and second ranked metrics, respectively. For the first two metrics, higher values ​​indicate a closer resemblance between the generated image and a standard clear image. For LPIPS, lower values ​​indicate better image quality. In the table, this embodiment, trained on one training set (NHR), shows significantly higher PSNR, SSIM, and LPIPS values ​​on the three test sets (NHR, NHM, and Unreal-NH) compared to other image processing algorithms, demonstrating its outstanding performance and robustness in image processing.

[0101] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0102] Based on the same ideas as the nighttime defogging method based on nighttime physical perception and grayscale world in the above embodiments, the present invention also provides a nighttime defogging system based on nighttime physical perception and grayscale world, which can be used to execute the above-described nighttime defogging method based on nighttime physical perception and grayscale world. For ease of explanation, the structural schematic diagram of the nighttime defogging system embodiment based on nighttime physical perception and grayscale world only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0103] Please see Figure 6 In another embodiment of this application, a night defogging system 100 based on nighttime physical perception and grayscale world is provided. The system includes a feature information extraction module 101, a first processing module 102, a second processing module 103, and an image restoration module 104.

[0104] The feature information extraction module 101 is used to first extract important image feature information from the foggy image through the local residual learning module and the compression-excitation module, and then learn the weights.

[0105] The first processing module 102 is used to input the important image feature information into the nighttime physical perception module to obtain feature information of a clear nighttime image including glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. After combining the glow region information and global information, an activation function is used to activate them to obtain first feature information based on the glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain second feature information. The first feature information and the second feature information are fused, and after performing a deconvolution operation, the feature information of the clear nighttime image is obtained.

[0106] The second processing module 103 is used to transmit the feature information of the clear night image to a feature processing group composed of several subsequent feature processing blocks. The feature maps output by the multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module, and a night physical perception module connected in sequence.

[0107] The image restoration module 104 is used to input the third feature information into the global residual learning restoration module to finally obtain the restored nighttime dehazing image result.

[0108] It should be noted that the nighttime defogging system based on nighttime physical perception and grayscale world of the present invention corresponds one-to-one with the nighttime defogging method based on nighttime physical perception and grayscale world of the present invention. The technical features and beneficial effects described in the above embodiments of the nighttime defogging method based on nighttime physical perception and grayscale world are applicable to the embodiments of nighttime defogging based on nighttime physical perception and grayscale world. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0109] Furthermore, in the above embodiments of the night defogging system based on nighttime physical perception and grayscale world, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the night defogging system based on nighttime physical perception and grayscale world is divided into different program modules to complete all or part of the functions described above.

[0110] Please see Figure 7 In one embodiment, an electronic device is provided for implementing a nighttime defogging method based on nighttime physical perception and a grayscale world. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a nighttime defogging program 203 based on nighttime physical perception and a grayscale world.

[0111] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal and external storage units of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of a nighttime defogging program 203 based on nighttime physical perception and grayscale world, but also to temporarily store data that has been output or will be output.

[0112] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0113] Figure 7 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 7 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0114] The night defogging program 203, stored in the first memory 202 of the electronic device 200, based on nighttime physical perception and the grayscale world, is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0115] The foggy image first passes through a local residual learning module and a compression-excitation module to extract important image feature information and learn weights;

[0116] The important image feature information is input into the nighttime physical perception module to obtain feature information of a clear nighttime image containing glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. The glow region information and global information are combined and activated using an activation function to obtain first feature information based on glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain second feature information. The first feature information and the second feature information are fused and deconvolution is performed to obtain feature information of the clear nighttime image.

[0117] The feature information of the clear night image is passed to a feature processing group consisting of several feature processing blocks connected together. The feature maps output by multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence.

[0118] The third feature information is input into the global residual learning recovery module to finally obtain the recovered nighttime dehazed image result.

[0119] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0120] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A nighttime defogging method based on nighttime physical perception and the grayscale world, characterized in that, Includes the following steps: The foggy image first passes through a local residual learning module and a compression-excitation module to extract important image feature information and learn weights; The important image feature information is input into the nighttime physical perception module to obtain feature information of a clear nighttime image containing glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. After combining the glow region information and global information, an activation function is used to activate them to obtain the first feature information based on the glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain the second feature information; the first and second feature information are fused together, and after deconvolution, the feature information of the clear night image is obtained. The feature information of the clear night image is passed to a feature processing group consisting of several feature processing blocks connected together. The feature maps output by multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence. The third feature information is input into the global residual learning recovery module to finally obtain the recovered nighttime dehazed image result.

2. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1, characterized in that, The nighttime physical sensing module is obtained through the following steps: Where I(x) represents a foggy nighttime image, L(x) and η(x) represent the ambient lighting intensity and the color deviation of the active light source, R(x) represents the reflectivity, t(x) represents the transmission map, x represents the pixel index, and J(x) represents a clear nighttime image, as shown below: J(x) = R(x)L(x); Based on the above formula, we obtain the following expression: Assuming the kernel size for feature extraction is k, the following formula is obtained: in, The symbol represents the convolution operator, and ⊙ represents the element-wise multiplication operation. This introduces color distortion removal loss based on the assumption of color constancy in the gray world, making... It is approximated as a constant matrix with all elements equal to 1; then, through several algebraic operations in matrix-vector form, the following formula is obtained: Where, K, T and T represent k, J, I, L and respectively. The matrix-vector form, where 1 represents a matrix of appropriate size consisting entirely of 1s, derived from the diagonal vectors of the diagonal matrix T and the vectorized representation. Alignment; Decomposing matrix KT into the product of two matrices, denoted as GK, since K, T, and G are all unknown variables, this decomposition can be formally represented as solving an underdetermined system of equations, thus yielding the following formula: Where K represents the feature extraction operation. The formula describes the relationship between sharp images and foggy nighttime images in the feature space. The following method is used to analyze sharp nighttime images... Modeling features: in, For the transmission graph predicted by the transmission graph branches, The glow feature information obtained after glow prediction branch; Based on the above modeling, the nighttime physical perception module is constructed as a dual-branch structure.

3. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1, characterized in that, The local residual learning module bypasses information in foggy or low-frequency regions by connecting multiple local residuals.

4. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1, characterized in that, The feature processing block consists of a local residual learning module, a compression excitation module, and a nighttime physical perception module. The input feature information is first processed by a convolutional layer and then enters the nighttime physical perception module. The constructed nighttime physical perception module processes the image features in the feature space and outputs the feature information of the clear image. Finally, it is combined with the input feature information as the output of the feature processing block.

5. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1, characterized in that, The feature processing group consists of multiple feature processing blocks. The multiple feature processing blocks process the input feature information in sequence, and finally combine the feature information of the multiple feature processing blocks as the output of the feature processing group.

6. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1, characterized in that, Inspired by the gray-world hypothesis algorithm, a loss function based on the gray-world hypothesis algorithm was designed. This function leverages the algorithm's powerful ability to handle color casts to assist the nighttime physical perception module in recovering clear, fog-free nighttime images with rich detail. The loss function is defined as follows: L color This technology is designed to mitigate color cast issues in foggy nighttime images. Inspired by the grayscale world hypothesis, it works by comparing information between different color channels. color Represented as: Where (m,n) represents a set of channels, J m J represents the average value of the m channels of the image after processing by multiple feature processing groups and the global residual learning and recovery module. n This represents the average value of the n channels of the image after processing by multiple feature processing groups and the global residual learning and recovery module, while RGB represents the R, G, and B color channels of the image, respectively.

7. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1, characterized in that, In the global residual learning recovery module, all feature maps output by several feature processing block structures are connected in the channel direction, and then multiplied by the adaptive learning weights obtained in the network to fuse features. By retaining the information of the low layer and passing it to the deep layer, the network pays more attention to the effective information of thick fog areas, high frequency textures and color fidelity.

8. A nighttime defogging system based on nighttime physical perception and the grayscale world, characterized in that, The nighttime defogging method based on nighttime physical perception and grayscale world, applied to any one of claims 1-7, includes a feature information extraction module, a first processing module, a second processing module, and an image restoration module; The feature information extraction module is used to first extract important image feature information from the foggy image through the local residual learning module and the compression-excitation module, and then learn the weights. The first processing module is used to input the important image feature information into the nighttime physical perception module to obtain feature information of a clear nighttime image containing glow and transmission map. Specifically, the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. In the glow prediction branch, glow region information and global information are obtained based on global max pooling and global average pooling, respectively. After combining the glow region information and global information, an activation function is used to activate them to obtain the first feature information based on the glow region information. In the transmission map prediction branch, multiple convolutional layers, ReLU, and activation functions are used to fit the transmission map to obtain the second feature information; the first and second feature information are fused together, and after deconvolution, the feature information of the clear night image is obtained. The second processing module is used to pass the feature information of the clear night image to a feature processing group composed of several subsequent feature processing blocks. The feature maps output by the multiple feature processing blocks are connected in the channel direction to obtain the third feature information. The feature processing block includes a local residual learning module, a compression-excitation module, and a night physical perception module connected in sequence. The image restoration module is used to input the third feature information into the global residual learning restoration module to finally obtain the restored nighttime dehazing image result.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the nighttime defogging method based on nighttime physical perception and grayscale world as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the nighttime defogging method based on nighttime physical perception and grayscale world as described in any one of claims 1-7.

Images