Night defogging method and device based on night physical perception and gray world
Through the defog removal method based on night physical perception and grayscale world, combined with local residual learning and grayscale world color constancy assumption, the problem of image details loss and artifacts in night fog removal is solved, and the night atmospheric scattering and artificiality are realized Effective processing of light sources restores high-quality clear night images.
Patent Information
- Application Number
- CN202410939864.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-15
AI Technical Summary
The existing night-time defogging method is poor in dealing with interference from the physical model of atmospheric scattering at night and artificial light sources, resulting in loss of image details and artifacts, and cannot effectively deal with glow, light unevenness and color shift problems caused by multiple artificial light sources.
Using the defogging method based on night physical perception and grayscale world, image features are extracted through the local residual learning module and the compression-excitation module, combined with glow prediction branches and transmission graph prediction branches, using global maximum pooling and global average pooling Process the glow region information, combine multi-layer convolution and activation functions to fit the transmission diagram, and finally remove the loss function through the color distortion assumption of color constancy in the grayscale world to achieve clear image recovery.
It effectively avoids the accumulation of model errors, improves the night fog removal effect, eliminates the influence of fog, glow and color offset, restores delicate image details and real colors, and improves the visibility and authenticity of the image.
Smart Images

Figure CN119048373A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image classification, and in particular relates to a nighttime defogging method and device based on nighttime physical perception and a grayscale world. Background Art
[0002] Nighttime haze is a common atmospheric phenomenon that significantly reduces visibility and degrades image quality. In nighttime haze imaging, the images are affected by haze and various types of degradation, including light scattering, non-uniform illumination, glow, and noise artifacts, due to the reduction of ambient lighting and interference from artificial light sources. These degradations not only increase the complexity of nighttime defogging tasks, but also hinder the effectiveness of subsequent visual processing tasks, including segmentation, autonomous driving, and object detection. Early nighttime defogging methods directly migrated daytime methods to nighttime. However, due to factors such as low brightness, complex light sources, and severe color shift in nighttime scenes, daytime defogging methods performed poorly at night, resulting in severe detail loss and residual fog in defogging images. In order to further improve the defogging effect of nighttime scenes, some prior assumptions for nighttime scenes have been proposed based on statistics, and some defogging methods have been designed using these assumptions. However, such assumptions are not always valid in real-world scenarios, and therefore artifacts and glow cannot be avoided well. With the development of deep learning, many new nighttime defogging methods have also emerged. They use deep networks to predict parameters in physical models such as transmission maps or atmospheric light to restore clear images, or directly use end-to-end methods to directly obtain clear images. However, these methods generally have two problems: (1) They do not use the physical model of nighttime atmospheric scattering as a guide or only use the physical model of nighttime atmospheric scattering in surface space. The former will lead to weak generalization of the model and poor performance in the real world, and the latter will make the accumulation of experimental errors inevitable, affecting the final dehazing effect. (2) They are unable to handle the glow caused by multiple artificial light sources at night, uneven illumination distribution, and multiple and strong color cast interference, which usually leads to problems such as image color distortion, detail loss, and obvious artifacts.
[0003] In view of the above problems, it can be seen that how to utilize the nighttime atmospheric scattering physical model while avoiding error accumulation in the process of nighttime foggy image processing, and how to deal with serious degradation problems such as glow and color cast caused by artificial light sources, is an urgent problem to be solved. Summary of the invention
[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and to provide a nighttime defogging method and device based on nighttime physical perception and the grayscale world, thereby improving the defogging performance in foggy scenes at night, while eliminating and reducing the impact of common nighttime degradations such as fog, glow, and color cast on images, truly and delicately restoring the color and detailed texture of the image, and improving the authenticity and visibility of the defogged image.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] In a first aspect, the present invention provides a nighttime defogging method based on nighttime physical perception and grayscale world, comprising the following steps:
[0007] The foggy image first passes through the local residual learning module and the compression-excitation module to extract important image feature information in the image and learn the weights;
[0008] The important image feature information is input into a nighttime physical perception module to obtain feature information of a nighttime clear image including glow and a transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, glow region information and global information are obtained based on global maximum pooling and global average pooling, respectively, and the glow region information and the global information are combined and activated using an activation function to obtain first feature information based on the glow region information; in the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain second feature information; the first feature information and the second feature information are fused, and a deconvolution operation is performed to obtain feature information of a nighttime clear image;
[0009] The feature information of the night clear image is transmitted to a feature processing group formed by connecting a plurality of subsequent feature processing blocks, and the feature graphs output by the plurality of feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block comprises a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence;
[0010] The third feature information is input into a global residual learning recovery module, and finally a restored nighttime defogging image result is obtained.
[0011] As a preferred technical solution, the nighttime physical perception module is obtained by the following steps:
[0012]
[0013] Among them, I(x) represents the night foggy image, L(x) and η(x) represent the ambient lighting intensity and the color deviation of the activated light source, R(x) represents the reflectivity, t(x) represents the transmission map, x represents the pixel index, and J(x) represents the night clear image, which is expressed as follows:
[0014] J(x) = R(x)L(x);
[0015] Based on the above formula, the following formula is obtained:
[0016]
[0017] Assuming that the kernel size of the extracted features is k, the following formula is obtained:
[0018]
[0019] in, represents the convolution operator, ⊙ represents the element-wise product operation, and introduces the color distortion removal loss of the grayscale world color constancy assumption, so that is approximated as a constant matrix with all elements equal to 1; then by using a few algebraic operations in matrix-vector form, we get the following formula:
[0020]
[0021] Among them, K, and T represent k, J, I, L and The matrix-vector form of , 1 represents a matrix of all 1s of the appropriate size, and the diagonal vectors from the diagonal matrix T are the same as the vectorized representation of Alignment;
[0022] Decompose the matrix KT into the product of two matrices, expressed as GK. Since K, T and G are all unknown variables, the realization of this decomposition can be formally represented as solving an underdetermined system of equations, so the following formula is obtained:
[0023]
[0024] Where K is the feature extraction operation, The formula expresses the relationship between the clear image and the night foggy image in the feature space. The night clear image is analyzed in the following way Modeling of features:
[0025]
[0026] According to the above modeling, the nighttime physical perception module is constructed as a dual-branch structure.
[0027] As a preferred technical solution, the local residual learning module bypasses the information of the haze or low-frequency area through multiple local residual connections.
[0028] As a preferred technical solution, the feature processing block is composed of a local residual learning module, a compression excitation module and a night physical perception module. The input feature information will first be processed by the convolution layer and then enter the night physical perception module. The constructed night physical perception module will be used to process the image features in the feature space and output the feature information of the clear image. Finally, it will be combined with the input feature information as the output of the feature processing block.
[0029] As a preferred technical solution, the feature processing group is composed of multiple feature processing blocks, which process the input feature information in sequence, and finally combine the feature information of the multiple feature processing blocks as the output of the feature processing group.
[0030] As a preferred technical solution, the gray world hypothesis is specifically:
[0031] Inspired by the grayscale world hypothesis algorithm, a loss function based on the grayscale world hypothesis algorithm is designed. Its powerful ability to handle color cast is used to assist the nighttime physical perception module in restoring clear nighttime fog-free images with fine details. The loss function is defined as follows:
[0032] L color It is used to alleviate the color cast problem in foggy images at night. It is inspired by the grayscale world hypothesis and is achieved by comparing the information between different color channels. color It is expressed as:
[0033]
[0034] Among them, (m,n) represents a stack of channels, J m represents the average value of the m channels of the image after being processed by multiple feature processing groups and the global residual learning recovery module, J n It represents the average value of n channels of the image after being processed by multiple feature processing groups and the global residual learning recovery module, and RGB represents the R, G, and B channels of the image respectively.
[0035] As a preferred technical solution, in the global residual learning recovery module, all feature maps output by several feature processing block structures are connected in the channel direction, and then multiplied by the adaptive learning weights obtained in the network to fuse the features. By retaining the low-level information and passing it to the deep layer, the network pays more attention to the effective information of thick fog areas, high-frequency textures and color fidelity.
[0036] In a second aspect, the present invention provides a nighttime defogging system based on nighttime physical perception and grayscale world, which is applied to the nighttime defogging method based on nighttime physical perception and grayscale world, and includes a feature information extraction module, a first processing module, a second processing module and an image restoration module;
[0037] The feature information extraction module is used for foggy images. It first extracts important image feature information from the image through a local residual learning module and a compression-excitation module and learns weights.
[0038] The first processing module is used to input the important image feature information into the nighttime physical perception module to obtain feature information of a nighttime clear image including glow and transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, glow region information and global information are obtained based on global maximum pooling and global average pooling respectively, and the glow region information and the global information are combined and activated using an activation function to obtain first feature information based on the glow region information; in the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain second feature information; the first feature information and the second feature information are fused, and a deconvolution operation is performed to obtain feature information of a nighttime clear image;
[0039] The second processing module is used to transmit the feature information of the night clear image to a feature processing group formed by connecting a plurality of subsequent feature processing blocks, and the feature maps output by the plurality of feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence;
[0040] The image restoration module is used to input the third feature information into a global residual learning restoration module, and finally obtain a restored nighttime defogging image result.
[0041] In a third aspect, the present invention provides an electronic device, the electronic device comprising:
[0042] at least one processor; and,
[0043] a memory communicatively connected to the at least one processor; wherein,
[0044] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the nighttime defogging method based on nighttime physical perception and grayscale world.
[0045] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the nighttime defogging method based on nighttime physical perception and a grayscale world.
[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0047] (1) The present invention sets up a color distortion removal loss function including a grayscale world color constancy assumption and a nighttime physical perception module. The nighttime physical perception model avoids the degradation of the model effect caused by the accumulation of model errors by applying the guidance of the nighttime atmospheric scattering physical model in the feature domain and combining it with the color distortion removal loss function of the grayscale world color constancy assumption. At the same time, it effectively solves the image degradation problems caused by nighttime fog, glow, color cast, etc., and restores a clearer and more delicate foggy image.
[0048] (2) The nighttime physical perception module provided by the present invention adopts a dual-branch structure to predict two important factors, namely, the glow and the transmission map at night. For the glow, the present invention combines global average pooling and global maximum pooling, so that the network can better pay attention to and process the glow area. For the transmission map, due to its uneven characteristics, the present invention adopts multi-layer convolution combined with appropriate activation functions to fit it. Finally, the two are combined to obtain high-quality nighttime clear image features.
[0049] (3) The color distortion removal loss function of the grayscale world color constancy hypothesis used in the present invention introduces the grayscale world hypothesis algorithm as the loss function, effectively utilizing the superiority of the grayscale world algorithm in dealing with image color deviation problems, and assisting the nighttime physical perception model to more focus on solving more important and difficult image degradation problems, thereby restoring a nighttime defogging image with more realistic colors, and solving the serious color difference problem caused by multiple artificial light sources in foggy scenes at night. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0051] Figure 1 is a flow chart of a nighttime defogging method based on nighttime physical perception and grayscale world according to an embodiment of the present invention;
[0052] Figure 2 It is a schematic diagram of a nighttime defogging network of a color distortion removal loss and a nighttime physical perception module of a grayscale world color constancy hypothesis according to an embodiment of the present invention;
[0053] Figure 3 is a schematic diagram of the structure of a nighttime physical perception module in an embodiment of the present invention;
[0054] Figure 4Schematic diagram of the effect of the grayscale world hypothesis algorithm in the embodiment of the present invention. The first row of images are foggy color-shifted images in various complex scenes, and the second row of images are images with color-shift removed after being processed by the grayscale hypothesis algorithm. Figure 4 Parts (a) and (b) are selected from synthetic night images. Figure 4 Part (c) is selected from night images collected from the real world;
[0055] Figure 5 This is a comparison chart of the defogging effects of the present invention and several advanced defogging algorithms;
[0056] Figure 6 A schematic diagram of the structure of a nighttime defogging system for removing color distortion loss and a nighttime physical perception module based on a grayscale world color constancy hypothesis according to an embodiment of the present invention;
[0057] Figure 7 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0058] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.
[0059] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.
[0060] See also Figure 1 , this embodiment proposes a nighttime defogging method based on nighttime physical perception and grayscale world, including the following steps:
[0061] S1. The foggy image first passes through the local residual learning module and the compression-excitation module to extract important image feature information in the image and learn the weights.
[0062] Furthermore, the local residual learning module allows bypassing less important information such as haze or low-frequency areas through multiple local residual connections, while the main network focuses on valid information. Experimental results show that the local residual learning module can further improve network performance and stability during training.
[0063] Furthermore, the compression-excitation module first compresses the spatial information of each channel of the image feature map U into a single value, that is, a vector of 1*1*C size is obtained from U of size H*W*C, and then a set of fully connected layers are applied to the vector for weight adjustment to obtain a channel attention vector of 1*1*C. Finally, the channel attention vector is weighted to U to form a weighted image feature map.
[0064] S2. Input the important image feature information into the nighttime physical perception module to obtain feature information of a clear nighttime image including glow and transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, the glow area information and the global information are obtained respectively based on the global maximum pooling and the global average pooling, and the glow area information and the global information are combined and activated using an activation function to obtain the first feature information based on the glow area information; in the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain the second feature information.
[0065] It can be understood that the nighttime physical perception module in this embodiment is composed of two branches, namely the glow prediction branch and the transmission map prediction branch. The two branches are designed according to the characteristics of the glow and transmission map in the nighttime atmosphere, respectively, and can effectively process the glow and transmission map, providing support for the application of the nighttime atmospheric scattering model in the feature space.
[0066] In a specific embodiment, Figure 2 The color distortion removal loss of the world color constancy hypothesis and the nighttime physical perception module of the nighttime defogging network are demonstrated, and the defogging effect of the nighttime foggy image after passing through the network is also demonstrated. Figure 2 Among them, G1, G2, and G3 are three consecutive feature processing groups, each feature processing group is composed of three feature processing blocks, B1, B2, and B3, SE is a compression-excitation module, NPAM is the nighttime physical perception module, ReLU is an activation layer using ReLU as the activation function, and Concatenate is a feature map concatenation operation in the channel direction.
[0067] In a specific embodiment, Figure 3The nighttime physical perception module has the glow prediction branch at the top and the transmission map prediction branch in the middle. In the glow prediction branch, the image feature information is processed by global maximum pooling and global average pooling respectively, and then the two processed information are connected and input into a 1×1 convolution. The result is activated by the sigmoid function. In the transmission map prediction branch, there are mainly multiple convolutional layers and activation functions used to fit the transmission map. The information obtained is processed according to the following formula, where represents the clear image feature, and F represents the foggy image feature. is the transmission graph after transmission graph branch prediction, The glow feature information is obtained through the glow prediction branch, and finally a layer of deconvolution is performed to obtain the features of the night dehazed image.
[0068]
[0069] Furthermore, Figure 3 It means subtracting 1 from each element. Indicates pixel-by-pixel multiplication of elements. Indicates that the elements are accumulated pixel by pixel.
[0070] It is understandable that the nighttime physical perception module can enhance and obtain the features of clear nighttime images. The nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch. The module uses the nighttime atmospheric scattering model in the feature space to process the atomization process, improve the interpretability of the feature space, and better remove nighttime haze. The guidance of the reference nighttime atmospheric scattering model is as follows:
[0071]
[0072] Where I(x) represents the night foggy image L(x) and η(x) represent the ambient lighting intensity and the color deviation of the activated light source, R(x) represents the reflectivity, t(x) represents the transmission map, x represents the pixel index, and J(x) represents the night clear image:
[0073] J(x) = R(x)L(x);
[0074] It is observed that recovering a haze-free image J from a haze-observed image I is an ill-posed problem that relies on accurate estimation of the latent variables: the atmospheric light intensity L, the color of the ambient lighting η, and the transmission rate t.
[0075] Based on the above formula, the present invention can obtain the following formula:
[0076]
[0077] Then, assuming that the kernel size of the extracted features is k, we can obtain the following formula:
[0078]
[0079] in, represents the convolution operator, and ⊙ represents the element-wise product operation. Inspired by the prior method, the present invention finds that the grayscale world algorithm can effectively solve the color distortion problem in night foggy images. Therefore, in order to simplify the feature decoupling process, the present invention introduces the color distortion removal loss of the grayscale world color constancy assumption, so that It is approximated as a constant matrix with all elements being 1. Then, by using several algebraic operations in matrix-vector form, the present invention can obtain the following formula:
[0080]
[0081] Here, K, and T represent k, J, I, K and The matrix-vector form of .1 represents a matrix of all 1s of the appropriate size. The diagonal vectors from the diagonal matrix T are the same as the vectorized representation of Alignment.
[0082] Then the present invention decomposes the matrix KT into the product of two matrices, expressed as GK. Since K, T and G are all unknown variables, the realization of this decomposition can be formally characterized as solving an underdetermined system of equations. In an underdetermined system, the number of equations is less than the number of unknown variables, which means that there are infinite solutions that satisfy the given equations. Therefore, the existence of G is guaranteed because the system always has at least one solution. Therefore, it can be obtained:
[0083]
[0084] where K can be regarded as a feature extraction operation. This formula describes the relationship between clear images and night foggy images in the feature space. Since traditional neural networks (CNNs) have powerful representation capabilities, this paper proposes a nighttime physical perception module and combines it with deep CNN to approximate the same as G and Corresponding features. Therefore, the useful features with discriminativeness can be better estimated for clear image restoration. To this end, the proposed NPAM consists of two parts. The first part is to learn the features and It mainly approximates the key components G and When the characteristics When the number of channels in matches the number of channels of the module input feature F, the present invention can generate a clear night image in the following manner Modeling of features:
[0085]
[0086] According to the above construction, NPAM was constructed as a double-branched structure. Figure 2 As shown, the top branch, i.e., the illumination map construction branch, is dedicated to constructing L. Since nighttime lighting mainly comes from multiple artificial light sources, the amount of L in each scene is uncertain and is usually accompanied by glow and color distortion. Therefore, the present invention proposes to use global maximum pooling (GMP) to focus information on the light source and use global average pooling (GAP) to eliminate unnecessary information in the feature space. The middle branch, i.e., the nighttime haze transmission estimation branch, is dedicated to predicting the corresponding transmission map. Due to the non-uniformity of the transmission map, we choose multi-layer convolution instead of global pooling operators to approximate the pixel-perceived transmission map of the nighttime haze scene. Using the proposed NPAM, interpretable features f can be derived from the input features F to restore the nighttime foggy image. Different from using only GAP(·) to predict latent features, NPAM carefully integrates the corresponding physical features of irregular nighttime lighting conditions and obvious color distortion to achieve stable defogging effect at night. This approach promotes the estimation of more relevant features within the dual interaction framework in nighttime situations.
[0087] Specifically, it includes the following steps. First, in the glow prediction branch, the present invention uses global maximum pooling (GMP) and global average pooling to obtain glow area information and global information respectively, and then uses the concat operation to combine the two. The combined features are activated by the sigimod function after 1×1 convolution, and finally the feature information based on the glow area information is obtained. Secondly, in the transmission map prediction branch, since the information of the transmission map is not uniform in the image, the present invention uses several convolutional layers and ReLu and Sigimoid activation functions to fit the transmission map. Finally, the information of the two branches is fused together, and a deconvolution operation is performed to obtain the feature information of the clear image at night.
[0088] L color Used to alleviate the color cast problem in foggy night images. It is inspired by the gray world hypothesis and is achieved by comparing the information between different color channels. In the gray world correlation algorithm, the color in each sensor channel is averaged gray in the entire image, which has been proven to effectively alleviate the color cast in the image. color It can be expressed as:
[0089]
[0090] S3. The feature information of the night clear image is passed to a feature processing group formed by connecting a number of subsequent feature processing blocks, and the feature maps output by multiple feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence.
[0091] Furthermore, the feature processing block is composed of a local residual learning module, a compression-excitation module and a nighttime physical perception module. The input feature information will first be processed by a convolution layer and then enter the nighttime physical perception module. The nighttime physical perception module constructed under the guidance of the above-mentioned nighttime atmospheric scattering physical model will process the image features in the feature space and output the feature information of the clear image, which will be finally combined with the input feature information as the output of the feature processing block.
[0092] The feature processing group is composed of a plurality of feature processing blocks, which process the input feature information in sequence, and finally combine the feature information of the plurality of feature processing blocks as the output of the feature processing group.
[0093] It is understandable that the continuous feature processing groups increase the depth and expressiveness of the network, and the jump connections enable the network to avoid the problem of difficult training.
[0094] S4. Input the third feature information into a global residual learning recovery module to finally obtain a restored nighttime defogging image result.
[0095] Furthermore, in the global residual learning recovery module, all feature maps output by several feature processing block structures are connected in the channel direction and then multiplied by the adaptive learning weights obtained in the network to fuse the features. Thus, the present invention can retain the low-level information and pass it to the deep layer. Due to the existence of the weight mechanism, the network pays more attention to effective information such as thick fog areas, high-frequency textures and color fidelity.
[0096] In one embodiment, training is performed on RTX3090 with batchsize=2, during which the Adam optimizer is used and the cosine annealing strategy is used for training. The present invention crops each image into a size of 256×256 and uses the obtained image as the input network for training.
[0097] This embodiment can better deal with various degradation factors of foggy scenes at night, such as uneven lighting, glow, low brightness, serious detail loss, color cast and other problems, and obtain a nighttime defogging image with rich detail texture, correct color cast processing, no artifacts and no fog residue. Figure 5As shown. NIDM, GS, MRP, OSFD, and RIDCP produced very serious residual artifacts when processing the bowling image. At the same time, they did not correctly handle the color cast when processing the image of driving on the road at night, and did not cleanly remove the fog in the scene, which was far from the reference image. When facing various challenges of foggy scenes at night, the above methods did not successfully deal with various factors that affect image degradation. After being processed by the method provided in this embodiment, not only the fog in the scene is completely removed, but also no artifacts remain, the bright and dark parts are balanced, the details are rich and delicate, and the overall image is closest to the standard clear image. By comparison, it can be seen that in the same foggy scene at night, the color distortion removal loss of the grayscale world color constancy hypothesis provided in this embodiment and the nighttime defogging method of the nighttime physical perception module can better solve the interference problems such as color cast, uneven lighting, and glow in the nighttime scene images.
[0098] Table 1
[0099]
[0100] In order to more specifically reflect the technical effect of this embodiment, as shown in the objective evaluation results of the method in Table 1, peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learning-perceptual image block similarity (LPIPS) are used as evaluation indicators, where red and blue represent the first and second ranked indicators respectively. For the first two indicators, the higher the value, the closer the generated image is to the standard clear image, and the lower the LPIPS value, the better the image quality. In the table, under the training of one training set of NHR, the peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learning-perceptual image block similarity (LPIPS) of this embodiment on three test sets of NHR, NHM and Unreal-NH are significantly higher than other image processing algorithms, proving that this embodiment has outstanding performance and robustness in processing images.
[0101] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.
[0102] Based on the same idea as the nighttime defogging method based on nighttime physical perception and grayscale world in the above-mentioned embodiment, the present invention also provides a nighttime defogging system based on nighttime physical perception and grayscale world, which can be used to execute the above-mentioned nighttime defogging method based on nighttime physical perception and grayscale world. For ease of explanation, the structural schematic diagram of the embodiment of the nighttime defogging system based on nighttime physical perception and grayscale world only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.
[0103] See also Figure 6 In another embodiment of the present application, a nighttime defogging system 100 based on nighttime physical perception and grayscale world is provided, the system comprising a feature information extraction module 101, a first processing module 102, a second processing module 103 and an image restoration module 104;
[0104] The feature information extraction module 101 is used for the foggy image. It first passes through the local residual learning module and the compression-excitation module to extract important image feature information in the image and learn weights;
[0105] The first processing module 102 is used to input the important image feature information into the nighttime physical perception module to obtain feature information of a nighttime clear image including glow and transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, glow region information and global information are obtained based on global maximum pooling and global average pooling respectively, and the glow region information and the global information are combined and activated using an activation function to obtain first feature information based on the glow region information; in the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain second feature information; the first feature information and the second feature information are fused, and a deconvolution operation is performed to obtain feature information of a nighttime clear image;
[0106] The second processing module 103 is used to transmit the feature information of the night clear image to a feature processing group formed by connecting a plurality of subsequent feature processing blocks, and the feature maps output by the plurality of feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence;
[0107] The image restoration module 104 is used to input the third feature information into a global residual learning restoration module, and finally obtain a restored nighttime defogging image result.
[0108] It should be noted that the nighttime defogging system based on nighttime physical perception and the grayscale world of the present invention corresponds one-to-one to the nighttime defogging method based on nighttime physical perception and the grayscale world of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the nighttime defogging method based on nighttime physical perception and the grayscale world are applicable to the embodiment of the nighttime defogging based on nighttime physical perception and the grayscale world. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.
[0109] In addition, in the implementation of the nighttime defogging system based on nighttime physical perception and the gray world in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the nighttime defogging system based on nighttime physical perception and the gray world is divided into different program modules to complete all or part of the functions described above.
[0110] See also Figure 7 In one embodiment, an electronic device for implementing a nighttime defogging method based on nighttime physical perception and a gray world is provided. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a nighttime defogging program 203 based on nighttime physical perception and a gray world.
[0111] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Further, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed in the electronic device 200, such as the code of the nighttime defogging program 203 based on nighttime physical perception and grayscale world, but also can be used to temporarily store data that has been output or is to be output.
[0112] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 200 by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202.
[0113] Figure 7 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device 200 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0114] The nighttime defogging program 203 based on nighttime physical perception and grayscale world stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions, and when executed in the first processor 201, the following can be achieved:
[0115] The foggy image first passes through the local residual learning module and the compression-excitation module to extract important image feature information in the image and learn the weights;
[0116] The important image feature information is input into a nighttime physical perception module to obtain feature information of a nighttime clear image including glow and a transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, glow region information and global information are obtained based on global maximum pooling and global average pooling, respectively, and the glow region information and the global information are combined and activated using an activation function to obtain first feature information based on the glow region information; in the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain second feature information; the first feature information and the second feature information are fused, and a deconvolution operation is performed to obtain feature information of a nighttime clear image;
[0117] The feature information of the night clear image is transmitted to a feature processing group formed by connecting a plurality of subsequent feature processing blocks, and the feature graphs output by the plurality of feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block comprises a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence;
[0118] The third feature information is input into a global residual learning recovery module, and finally a restored nighttime defogging image result is obtained.
[0119] Furthermore, if the module / unit integrated in the electronic device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0121] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0122] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A nighttime defogging method based on nighttime physical perception and grayscale world, characterized in that: The steps include: The foggy image first passes through the local residual learning module and the compression-excitation module to extract important image feature information in the image and learn the weights; The important image feature information is input into a nighttime physical perception module to obtain feature information of a nighttime clear image including glow and a transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, glow region information and global information are obtained based on global maximum pooling and global average pooling, respectively, and the glow region information and the global information are combined and activated using an activation function to obtain first feature information based on the glow region information; In the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain second feature information; the first feature information and the second feature information are fused and deconvolution operation is performed to obtain feature information of a night clear image; The feature information of the night clear image is transmitted to a feature processing group formed by connecting a plurality of subsequent feature processing blocks, and the feature graphs output by the plurality of feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block comprises a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence; The third feature information is input into a global residual learning recovery module, and finally a restored nighttime defogging image result is obtained.
2. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1 is characterized in that: The nighttime physical perception module is obtained by the following steps: Among them, I(x) represents the foggy image at night, L(x) and η(x) represent the ambient lighting intensity and the color deviation of the activated light source, R(x) represents the reflectivity, t(x) represents the transmission map, x represents the pixel index, and J(x) represents the clear image at night, which is expressed as follows: J(x) = R(x)L(x); Based on the above formula, the following formula is obtained: Assuming that the kernel size of the extracted features is k, the following formula is obtained: in, represents the convolution operator, ⊙ represents the element-wise product operation, and introduces the color distortion removal loss of the grayscale world color constancy assumption, so that is approximated as a constant matrix with all elements equal to 1; then by using a few algebraic operations in matrix-vector form, we get the following formula: in, and T represent k, J, I, L and The matrix-vector form of , 1 represents a matrix of all 1s of the appropriate size, and the diagonal vectors from the diagonal matrix T are the same as the vectorized representation of Alignment; Decompose the matrix KT into the product of two matrices, expressed as GK. Since K, T and G are all unknown variables, the realization of this decomposition can be formally represented as solving an underdetermined system of equations, so the following formula is obtained: Where K is the feature extraction operation, The formula expresses the relationship between the clear image and the night foggy image in the feature space. The night clear image is analyzed in the following way Modeling of features: According to the above modeling, the nighttime physical perception module is constructed as a dual-branch structure.
3. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1 is characterized in that: The local residual learning module bypasses the information of hazy or low-frequency areas through multiple local residual connections.
4. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1 is characterized in that: The feature processing block is composed of a local residual learning module, a compression excitation module and a nighttime physical perception module. The input feature information will first be processed by a convolution layer and then enter the nighttime physical perception module. The constructed nighttime physical perception module will be used to process the image features in the feature space and output the feature information of the clear image. Finally, it will be combined with the input feature information as the output of the feature processing block.
5. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1 is characterized in that: The feature processing group is composed of a plurality of feature processing blocks, which process the input feature information in sequence, and finally combine the feature information of the plurality of feature processing blocks as the output of the feature processing group.
6. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1 is characterized in that: The gray world hypothesis is specifically: Inspired by the grayscale world hypothesis algorithm, a loss function based on the grayscale world hypothesis algorithm is designed. Its powerful ability to handle color cast is used to assist the nighttime physical perception module in restoring clear nighttime fog-free images with fine details. The loss function is defined as follows: L color It is used to alleviate the color cast problem in foggy images at night. It is inspired by the grayscale world hypothesis and is achieved by comparing the information between different color channels. color It is expressed as: Among them, (m,n) represents a stack of channels, J m represents the average value of the m channels of the image after being processed by multiple feature processing groups and the global residual learning recovery module, J n It represents the average value of n channels of the image after being processed by multiple feature processing groups and the global residual learning recovery module, and RGB represents the R, G, and B channels of the image respectively.
7. The nighttime defogging method based on nighttime physical perception and grayscale world according to claim 1 is characterized in that: In the global residual learning recovery module, all feature maps output by several feature processing block structures are connected in the channel direction and then multiplied by the adaptive learning weights obtained in the network to fuse the features. By retaining the low-level information and passing it to the deep layer, the network pays more attention to the effective information of thick fog areas, high-frequency textures and color fidelity.
8. The nighttime defogging system based on nighttime physical perception and grayscale world is characterized by: The nighttime defogging method based on nighttime physical perception and grayscale world applied to any one of claims 1 to 7 comprises a feature information extraction module, a first processing module, a second processing module and an image restoration module; The feature information extraction module is used for foggy images. It first extracts important image feature information from the image through a local residual learning module and a compression-excitation module and learns weights. The first processing module is used to input the important image feature information into the nighttime physical perception module to obtain feature information of a nighttime clear image including glow and transmission map, specifically: the nighttime physical perception module includes a glow prediction branch and a transmission map prediction branch; in the glow prediction branch, glow region information and global information are obtained based on global maximum pooling and global average pooling respectively, and the glow region information and the global information are combined and activated using an activation function to obtain first feature information based on the glow region information; In the transmission map prediction branch, multiple convolutional layers, ReLu and activation functions are used to fit the transmission map to obtain second feature information; the first feature information and the second feature information are fused and deconvolution operation is performed to obtain feature information of a night clear image; The second processing module is used to transmit the feature information of the night clear image to a feature processing group formed by connecting a plurality of subsequent feature processing blocks, and the feature maps output by the plurality of feature processing blocks are connected in the channel direction to obtain third feature information; the feature processing block includes a local residual learning module, a compression-excitation module and a night physical perception module connected in sequence; The image restoration module is used to input the third feature information into a global residual learning restoration module, and finally obtain a restored nighttime defogging image result.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the nighttime defogging method based on nighttime physical perception and grayscale world as described in any one of claims 1-7.
10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the nighttime defogging method based on nighttime physical perception and grayscale world described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Single image defogging method based on physical prior and deep learning
CN115719319A
End-to-end unmanned aerial vehicle visual target tracking method and device in foggy day low-illumination scene
CN115861380A
Single image defogging method and device based on image prior and global block aggregation
CN116402697A
Night defogging method and device based on double-domain feature learning and cross-dimensional feature optimization
CN117541663A
