A Nighttime Defogging Method and Apparatus Based on Dual-Domain Feature Learning and Cross-Dimensional Feature Optimization

Abstract

This invention discloses a nighttime dehazing method and apparatus based on dual-domain feature learning and cross-dimensional feature optimization. The method includes: enhancing and extracting features of a foggy nighttime image using a preset dual-domain feature encoder; extracting frequency domain features of the foggy image through an adaptive frequency selection module in the frequency domain learning branch of the dual-domain feature encoder; extracting spatial domain features of the foggy nighttime image using residual dense blocks in the spatial domain learning branch of the dual-domain feature encoder; fusing the frequency and spatial domain features; inputting the fused features into a cross-dimensional attention feature optimization module; obtaining optimized features through a cross-dimensional interactive attention module; and inputting the optimized features into a preset dual-domain feature decoder to obtain a clear nighttime image. This invention improves dehazing performance in foggy nighttime scenes, eliminates the degradation effect of fog on images, realistically restores the color and texture details of the image, and enhances the realism of dehazed nighttime images.

Description

Technical Field

[0001] This invention belongs to the technical field of image classification, specifically relating to a nighttime dehazing method and apparatus based on dual-domain feature learning and cross-dimensional feature optimization. Background Technology

[0002] Fog is a typical atmospheric phenomenon. Affected by suspended particles such as smoke and dust in the air, fog can cause color distortion, blurring, and low contrast in captured images, posing a significant challenge to visual applications such as autonomous navigation, remote sensing, and surveillance. Nighttime image dehazing is a hot topic and a difficult task in image dehazing. The uneven distribution of overall illumination and the color interference of artificial light sources make nighttime dehazing tasks more challenging and complex. Early nighttime dehazing methods directly transferred daytime dehazing methods to foggy nighttime scenes. However, these methods are not suitable for foggy nighttime scenes where illumination is uniform and constant, often resulting in fog residue and loss of detail in the dehazed images. Later, researchers proposed some specific prior assumptions for nighttime scenes based on mathematical statistics and designed some dehazing methods using these assumptions. However, these methods based on prior assumptions do not always hold true in real-world scenes. Therefore, these methods often produce image halos and artifacts. With the development of deep learning, many methods use convolutional neural networks and attention mechanisms to predict atmospheric model parameters (such as transmission maps and atmospheric light) or sharp images. These methods can overcome the limitations of prior assumptions, thus achieving better dehazing results. However, they have two problems: (1) They only focus on spatial feature learning and ignore frequency feature learning, resulting in fog residue and loss of details. Experimental studies have shown that there is a significant frequency difference between foggy and clear images, and the degradation caused by fog can be estimated through spectral parameters. (2) They cannot handle uneven illumination distribution and color interference caused by multiple artificial light sources at night, which usually leads to color distortion problems.

[0003] As can be seen from the above problems, it is urgent to solve the problems of adding frequency feature learning and optimizing image illumination and color distribution in the process of processing foggy images. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a nighttime dehazing method and apparatus based on dual-domain feature learning and cross-dimensional feature optimization. This method improves dehazing performance in foggy nighttime scenes, eliminates the degradation effect of fog on images, and realistically restores the color and texture details of images, thereby enhancing the realism of nighttime dehazed images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization, comprising the following steps:

[0007] The pre-defined dual-domain feature encoder is used to enhance and extract features from foggy nighttime images. The dual-domain feature encoder includes a frequency domain learning branch and a spatial domain learning branch. The extraction of features from foggy nighttime images includes the following steps: First, in the frequency domain learning branch, the features of foggy nighttime images are transformed into the frequency domain using an adaptive frequency selection module. Then, frequency features are adaptively selected according to a set importance weight threshold to obtain the frequency features of the foggy nighttime images. Second, in the spatial domain learning branch, spatial domain features of the foggy nighttime images are extracted using residual dense blocks. Finally, the frequency features and spatial domain features are fused.

[0008] The fused features are input into the cross-dimensional attention feature optimization module. The contextual relationship information of the image is obtained through the multi-scale dilated convolutional structure. Then, the cross-dimensional interactive attention module is set to calculate the average importance weight of the fused features in the channel, height and width dimensions, so as to obtain optimized features of color and brightness that are equally important in multiple dimensions.

[0009] The optimized features are input into a preset dual-domain feature decoder to obtain clear nighttime images.

[0010] As a preferred technical solution, the frequency domain learning branch includes an adaptive frequency selection module, characterized in that it further includes the following steps:

[0011] Two-dimensional discrete cosine transform is used to convert the features of foggy nighttime images from the spatial domain to the frequency domain; a learnable importance weight threshold is defined, and the features of foggy nighttime images are dynamically zeroed out; then, an adaptive frequency selection module compares the average importance weight of the features of foggy nighttime images with the importance weight threshold. If the average importance weight is higher than the importance weight threshold, the feature is retained; otherwise, it is discarded.

[0012] As a preferred technical solution, the spatial domain learning branch includes residual dense blocks. In the spatial domain learning branch, the spatial domain features of the foggy nighttime image are extracted using the residual dense blocks, specifically:

[0013] First, spatial features are extracted from foggy nighttime images using six 3×3 convolutional layers and then concatenated together. Next, a 1×1 convolutional layer is added to reduce the network width. Finally, residual learning is performed between the output features and the original input features to obtain the spatial domain features of the image.

[0014] As a preferred technical solution, obtaining optimized features of color and brightness that are equally important in multiple dimensions further includes the following steps:

[0015] Two 1×1 convolutional layers and a multi-scale dilated convolutional structure are used to obtain multi-scale image features with contextual information; the multi-scale image features are concatenated and fused, and then input into a cross-dimensional interactive attention module; the multi-scale dilated convolutional structure consists of four 3×3 convolutional branches with dilation rates of 1, 2, 3, and 5, respectively.

[0016] The cross-dimensional interactive attention module includes channel-channel branches, height-channel branches, and width-channel branches. In the channel-channel branches, global average pooling and global max pooling are combined to sample the input features horizontally and vertically. The sampled features are added and concatenated by directional grouping, and channel information is exchanged using channel shuffling. Then, convolution is performed to reduce the feature dimension, and batch normalization is processed using the non-linear activation function ELU. Subsequent convolution is used to recover the number of feature channels in different directions, so that features in the same direction share similar weights, resulting in interactive features.

[0017] In the height-channel branch, shallow features are learned using the input features through 1×1 and 3×3 convolutional layers. Then, the features are rotated 90 degrees counterclockwise along the H-axis to obtain the rotated shallow feature X2. A self-attention mechanism is used to calculate feature weights and obtain the correlation between each element in the sequence feature X2, as shown in the following equation:

[0018]

[0019] Among them, Q w Indicates querying the data stream, K w V represents a key-value data stream. w d represents a numerical data stream. k The dimension representing the key value;

[0020] In the width-channel branch, change the rotation axis from the H axis to the W axis; the rest of the structure is the same as the height-channel branch.

[0021] As a preferred technical solution, the channel-to-channel branch also includes a step of supplementing and improving the fused feature information, specifically:

[0022] A simplified nonlocal attention CRLC is used to obtain effective global information, as follows:

[0023]

[0024] Among them, X C For input features, For the output features of channel-to-channel branches, SC(·) represents the SC structure, which is used to reduce the channel dimension of the input features;

[0025] The simplified nonlocal attention CRLC includes 1×1 convolution, ReLU function and LayerNorm, and the SC structure includes 1×1 convolution and Softmax function.

[0026] As a preferred technical solution, the step of optimizing the feature input to the preset dual-domain feature decoder further includes the following steps:

[0027] Using mean square error loss L mse With perceived loss L p Network optimization includes:

[0028] L = L mse +λL p

[0029]

[0030]

[0031] Where L represents the total loss, x i Indicates the input image, y i Represents the target image. Let F represent the predicted value of the target image features, λ represent the weights of the loss function, and F... i (x) and F i (y) represents their feature representation in the i-th layer of the pre-trained neural network, where N is the number of feature layers.

[0032] As a preferred technical solution, the dual-domain feature encoder uses the following method for feature extraction.

[0033] As a preferred technical solution, the dual-domain feature decoder uses upsampling for decoding, and the rest of the structure is consistent with the dual-domain feature encoder structure.

[0034] Secondly, the present invention also provides a night defogging system based on dual-domain feature learning and cross-dimensional feature optimization, which is applied to the night defogging method based on dual-domain feature learning and cross-dimensional feature optimization, including a dual-domain feature encoding module, a feature optimization module and a dual-domain feature decoding module.

[0035] The dual-domain feature encoding module is used to enhance and extract features of foggy nighttime images using a preset dual-domain feature encoder. The dual-domain feature encoder includes a frequency domain learning branch and a spatial domain learning branch. The extraction of features of foggy nighttime images includes the following steps: First, in the frequency domain learning branch, the features of foggy nighttime images are converted to the frequency domain using an adaptive frequency selection module, and then the frequency features are adaptively selected according to the set importance weight threshold to obtain the frequency features of the foggy nighttime images; Second, in the spatial domain learning branch, the spatial domain features of the foggy nighttime images are extracted using residual dense blocks; Finally, the frequency features and spatial domain features are fused.

[0036] The feature optimization module inputs the fused features into the cross-dimensional attention feature optimization module. It then uses a multi-scale dilated convolutional structure to obtain the image's contextual information. Finally, the cross-dimensional interactive attention module calculates the average importance weights of the fused features across the channel, height, and width dimensions, resulting in optimized features for color and brightness that are equally important across multiple dimensions.

[0037] The dual-domain feature decoding module is used to input optimized features into a preset dual-domain feature decoder to obtain clear nighttime images.

[0038] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0039] At least one processor; and,

[0040] A memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores computer program instructions that can be executed by the at least one processor, which enable the at least one processor to execute the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] (1) The present invention sets up a dual-domain feature encoder including a frequency domain learning branch and a spatial domain learning branch. By learning and predicting the changes of frequency features caused by fog, most of the fog in the features of foggy night images can be removed, thus eliminating the degradation effect of fog on the image.

[0044] (2) The cross-dimensional attention feature optimization module provided by the present invention can learn and optimize the lighting and color features in areas that are too dark or too bright by setting a cross-dimensional interactive attention mechanism, thereby helping to restore nighttime dehazed images with real colors and solving the problems of uneven lighting distribution and light source color interference in nighttime foggy images.

[0045] (3) The present invention uses perceptual loss and mean square error loss to optimize the proposed network, so that the network can fully learn the difference between foggy and fog-free images at night, thereby making the model more robust. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a nighttime image dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of a nighttime image dehazing network based on dual-domain feature learning and cross-dimensional feature optimization according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the learnable frequency threshold structure in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the cross-dimensional interactive attention module in an embodiment of the present invention;

[0051] Figure 5 This is a comparison chart of the defogging effects of this invention and several advanced defogging algorithms;

[0052] Figure 6 This is a schematic diagram of the structure of the nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization according to an embodiment of the present invention;

[0053] Figure 7 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0055] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0056] Please see Figure 1 This embodiment proposes a nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization, which includes the following steps:

[0057] S1. Enhance and extract features of foggy nighttime images using a pre-defined dual-domain feature encoder. The dual-domain feature encoder includes a frequency domain learning branch and a spatial domain learning branch. The extraction of features of foggy nighttime images includes the following steps: First, in the frequency domain learning branch, the features of foggy nighttime images are converted to the frequency domain using an adaptive frequency selection module. Then, frequency features are adaptively selected based on the set importance weight threshold and the importance of the foggy nighttime images to obtain the frequency features of the foggy nighttime images. Second, in the spatial domain learning branch, spatial domain features of the foggy nighttime images are extracted using residual dense blocks. Finally, the frequency features and spatial domain features are fused.

[0058] The dual-domain feature encoder in this embodiment consists of three modules: a frequency domain learning branch, a spatial domain learning branch, and a module for feature fusion. In the process of extracting frequency domain and spatial domain features of a foggy image, the dual-domain feature encoder adopts a downsampling method.

[0059] To address the issues of fog retention and detail loss due to missing frequency feature learning, this embodiment provides the following preferred solution:

[0060] Preferably, the frequency domain learning branch in step S1 includes an adaptive frequency selection module, and further includes the following steps:

[0061] Two-dimensional discrete cosine transform is used to convert the features of foggy nighttime images from the spatial domain to the frequency domain; a learnable importance weight threshold is defined, and the features of foggy nighttime images are dynamically zeroed out; then, an adaptive frequency selection module compares the average importance weight of the features of foggy nighttime images with the importance weight threshold. If the average importance weight is higher than the importance weight threshold, the feature is retained; otherwise, it is discarded.

[0062] Among them, see Figure 2-3 When the average importance weight is higher than the importance weight threshold, the frequency domain features of the foggy image are enhanced through a structure similar to channel attention.

[0063] Figure 2 In the dual-domain feature encoding module, the leftmost square represents a convolution with a kernel of 1 and a stride of 1; in the other three groups of squares of the same size, the left square represents the dual-domain feature learning module, and the right square represents downsampling.

[0064] Figure 2 In the dual-domain feature decoding module, the last square represents a convolution with a kernel of 1 and a stride of 1; in the other three groups of squares of the same size, the square on the left represents upsampling, and the square on the right represents the dual-domain feature learning module.

[0065] Figure 2 In the diagram, each dual-domain feature learning module includes the part pointed to by the arrow at the bottom of the "dual-domain feature learning module", which includes residual dense blocks and adaptive frequency selection modules.

[0066] Figure 2 In the diagram, each cross-dimensional attention feature optimization module group includes the part pointed to by the arrow at the bottom of the “Cross-dimensional attention feature optimization module group”, which includes three cross-dimensional attention feature optimization modules; each cross-dimensional attention feature optimization module includes a multi-scale dilated convolutional structure and a cross-dimensional interactive attention module.

[0067] Figure 2 middle This indicates that the elements are incremented pixel by pixel. This indicates that the elements are multiplied pixel by pixel.

[0068] Figure 3 This illustrates the principle behind obtaining the learnable threshold in the adaptive frequency domain selection module of this embodiment.

[0069] Figure 3 The left side represents the image feature block, and the right side represents the weight calculation process for each pixel within the feature block.

[0070] It should be further explained that the Two-Dimensional Discrete Cosine Transform (DCT) is a mathematical transformation widely used in image and signal processing. It is applied in various fields such as image compression, image analysis, and audio compression. The two-dimensional DCT transform is built upon the one-dimensional DCT transform. For a two-dimensional image patch, the result of the two-dimensional DCT transform can be obtained by first performing a one-dimensional DCT transform on the rows, and then performing a one-dimensional DCT transform on the columns.

[0071] For the spatial domain learning branch, the preferred solution in this embodiment is as follows:

[0072] Preferably, the spatial domain learning branch in step S1 includes residual dense blocks. In the spatial domain learning branch, the spatial domain features of the foggy nighttime image are extracted using the residual dense blocks, specifically:

[0073] First, spatial features are extracted from foggy nighttime images using six 3×3 convolutional layers and then concatenated together. Next, a 1×1 convolutional layer is added to reduce the network width. Finally, residual learning is performed between the output features and the original input features to obtain the spatial domain features of the image.

[0074] In addition, see again Figure 2 The residual dense block is also used to enhance the local features of foggy nighttime images before inputting to the adaptive frequency selection module, thereby improving the generalization performance of the model.

[0075] In this embodiment, feature extraction can also be performed in parallel using three dual-domain feature encoders of different scales. The sizes of the three encoders of different scales are 64×64, 128×128, and 256×256, respectively.

[0076] S2. Input the fused features into the cross-dimensional attention feature optimization module. Use a multi-scale dilated convolutional structure to obtain the image's contextual information. Then, set the cross-dimensional interactive attention module to calculate the average importance weights of the fused features in the channel, height, and width dimensions, obtaining optimized features for color and brightness that are equally important in multiple dimensions. Please refer to [link to relevant documentation]. Figure 2 The feature optimization module group;

[0077] In this embodiment, a cross-dimensional attention feature optimization module is set up based on cross-dimensional interactive attention. This module includes channel-channel, height-channel, and width-channel branches. It constructs the interdependencies between the attention features of the three interactive branches (channel-channel, height-channel, and width-channel) in the channel and spatial dimensions. (See reference...) Figure 4 It can obtain accurate color and lighting features that are equally important in multiple dimensions by analyzing the spatial variations of channel features or the variations of spatial features in channels, thereby reducing color distortion problems caused by insufficient representation of a single feature dimension.

[0078] It needs to be explained that, Figure 4 middle This indicates that the elements are incremented pixel by pixel; "R" indicates that the elements are multiplied pixel by pixel; "R" indicates that the feature dimension is changed.

[0079] For channel-channel, height-channel, and width-channel branches, the following are preferred solutions:

[0080] Preferably, step S2, which describes obtaining optimized features for color and brightness that are equally important across multiple dimensions, further includes the following steps:

[0081] Two 1×1 convolutional layers and a multi-scale dilated convolutional structure are used to obtain multi-scale image features with contextual information; the multi-scale image features are concatenated and fused, and then input into a cross-dimensional interactive attention module; the multi-scale dilated convolutional structure consists of four 3×3 convolutional branches with dilation rates of 1, 2, 3, and 5, respectively.

[0082] The cross-dimensional interactive attention module includes channel-channel branches, height-channel branches, and width-channel branches. In the channel-channel branches, global average pooling and global max pooling are combined to sample the input features horizontally and vertically. The sampled features are added and concatenated by directional grouping, and channel information is exchanged using channel shuffling. Then, convolution is performed to reduce the feature dimension, and batch normalization is processed using the non-linear activation function ELU. Subsequent convolution is used to recover the number of feature channels in different directions, so that features in the same direction share similar weights, resulting in interactive features.

[0083] In the height-channel branch, shallow features are learned using the input features through 1×1 and 3×3 convolutional layers. Then, the features are rotated 90 degrees counterclockwise along the H-axis to obtain the rotated shallow feature X2. A self-attention mechanism is used to calculate feature weights, obtaining the correlation between each element in the sequence feature X2, thus achieving feature interaction in the height and channel dimensions, as shown in the following equation:

[0084]

[0085] Among them, Q w Indicates querying the data stream, K w V represents a key-value data stream. w d represents a numerical data stream. k The dimension representing the key value; self-attention through This allows the query value to be multiplied by the key value, thereby constructing the association between each element in the sequence feature X2 and realizing feature interaction in the height and channel dimensions.

[0086] In the width-channel branch, change the rotation axis from the H axis to the W axis; the rest of the structure is the same as the height-channel branch.

[0087] After image feature enhancement and optimization, the output is input into a dual-domain feature decoder based on residual dense blocks, adaptive frequency selection module, and upsampling to obtain a clear nighttime image after dehazing. The input of the dual-domain feature decoder comes not only from the upper-layer decoder but also from the image features extracted by the dual-domain feature encoder at the same scale level, as shown in step S3.

[0088] S3. Input the optimized features into the preset dual-domain feature decoder to obtain clear nighttime images.

[0089] In this embodiment, the dual-domain feature decoder uses mean squared error loss L mse With perceived loss L p The dehazing network proposed in steps S1 to S3 is optimized to enable the network to fully learn the difference between foggy and fog-free images at night, thereby making the model more robust. Specifically, the following preferred solutions are proposed:

[0090] Preferably, the step of optimizing the feature input to the preset dual-domain feature decoder further includes the following steps:

[0091] Using mean square error loss L mse With perceived loss L p Network optimization includes:

[0092] L = L mse +λL p

[0093]

[0094]

[0095] Where L represents the total loss, x i Indicates the input image, y i Represents the target image. F represents the predicted value of the target image features, λ represents the weight value of the loss function, and the default value is set to 0.01. i (x) and F i (y) represents their feature representation in the i-th layer of the pre-trained neural network, where N is the number of feature layers.

[0096] In one embodiment, the mean squared error loss L mse With a weight of 1.0, the perceptual loss L p The weight is 0.01; training is performed on an RTX3090 for 100 epochs, with an initial step size of 0.0001. This invention uses a 256×256 window to randomly cut the image, and then randomly rotates the cut image, using the resulting image as the input for training.

[0097] It is worth noting that the mean square error loss L mse It is calculated based on the mean squared error loss, which is a loss function used to calculate the error between the model's predicted value and the true value. It measures the accuracy of the model by calculating the average of the squared differences between the predicted value and the true value.

[0098] Furthermore, perceptual loss is calculated by using a pre-trained neural network to compute the difference between two images. The typical method for calculating perceptual loss involves passing the input and target images separately through a pre-trained neural network to obtain their feature representations within the network. These feature representations are then used as input to a loss function to calculate the Euclidean or Manhattan distance between them. The goal of perceptual loss is to minimize the distance between the input and target images in the feature space.

[0099] This embodiment can better handle foggy nighttime scenes with uneven lighting and light source color interference, obtaining dehazed nighttime images with rich texture details, no fog residue, and true colors. For example... Figure 5 As shown, while the GS algorithm removes the fog and reveals background text, lights, and other elements, it fails to adequately represent brightness, contrast, and the characteristics of both bright and dark areas. Furthermore, while the MRP and RIDCP algorithms produce similar results on the same image, compared to a standard clear image, the dark areas are noticeably brighter and exhibit significant noise, especially the image processed by the MRP algorithm, where both the dark and bright areas are significantly overexposed. However, the method provided in this embodiment not only removes the fog from the foggy nighttime image but also highlights bright features, balances contrast, and closely resembles a standard clear image. The comparison demonstrates that, based on the same foggy nighttime image, the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization provided in this embodiment better addresses the problems of uneven brightness distribution and light source color interference in nighttime scene images.

[0100] Table 1

[0101]

[0102] To more concretely demonstrate the technical effectiveness of this embodiment, as shown in Table 1, which presents the objective evaluation results of the method, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used as evaluation metrics. Higher values ​​indicate that the generated image is closer to a standard clear image. In the table, under training on both NHR and NHM training sets, this embodiment exhibits significantly higher PSNR and SSIM values ​​than other image processing algorithms, demonstrating its outstanding performance and robustness in image processing.

[0103] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0104] Based on the same idea as the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization in the above embodiments, this invention also provides a nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization. This system can be used to execute the above-described nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization. For ease of explanation, the structural diagram of the embodiment of the nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization only shows the parts related to the embodiments of this invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0105] Please see Figure 6 In another embodiment of this application, a nighttime defogging system 10 based on dual-domain feature learning and cross-dimensional feature optimization is provided. The system includes a dual-domain feature encoding module 11, a feature optimization module 12, and a dual-domain feature decoding module 13.

[0106] The dual-domain feature encoding module 11 is used to enhance and extract features of foggy nighttime images using a preset dual-domain feature encoder. The dual-domain feature encoder includes a frequency domain learning branch and a spatial domain learning branch. The extraction of features of foggy nighttime images includes the following steps: First, in the frequency domain learning branch, the features of foggy nighttime images are converted to the frequency domain using an adaptive frequency selection module, and then the frequency features are adaptively selected according to the set importance weight threshold to obtain the frequency features of the foggy nighttime images; Second, in the spatial domain learning branch, the spatial domain features of the foggy nighttime images are extracted using residual dense blocks; Finally, the frequency features and spatial domain features are fused.

[0107] The feature optimization module is used to input the fused features into the cross-dimensional attention feature optimization module, obtain the contextual relationship information of the image through the multi-scale dilated convolutional structure, and then set the cross-dimensional interactive attention module to calculate the average importance weight of the fused features in the channel, height and width dimensions, so as to obtain optimized features of color and brightness that are equally important in multiple dimensions.

[0108] The dual-domain feature decoding module 13 is used to input optimized features into a preset dual-domain feature decoder to obtain clear nighttime images.

[0109] It should be noted that the nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization of the present invention corresponds one-to-one with the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization of the present invention. The technical features and beneficial effects described in the embodiments of the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization described above are all applicable to the embodiments of the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0110] Furthermore, in the above embodiments of the nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization is divided into different program modules to complete all or part of the functions described above.

[0111] Please see Figure 7 In one embodiment, an electronic device is provided that implements a nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization. The electronic device 20 may include a first processor 21, a first memory 22 and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a nighttime defogging program 23 based on dual-domain feature learning and cross-dimensional feature optimization.

[0112] The first memory 22 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as the portable hard drive of the electronic device 20. In other embodiments, the first memory 22 can be an external storage device of the electronic device 20, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 20. Furthermore, the first memory 22 can include both internal storage units and external storage devices of the electronic device 20. The first memory 22 can be used not only to store application software and various types of data installed on the electronic device 20, such as the code of the night defogging program 23 based on dual-domain feature learning and cross-dimensional feature optimization, but also to temporarily store data that has been output or will be output.

[0113] In some embodiments, the first processor 21 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 21 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 22 and calls data stored in the first memory 22 to perform various functions of the electronic device 20 and process data.

[0114] Figure 7 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 7 The structure shown does not constitute a limitation on the electronic device 20, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0115] The night defogging program 23 based on dual-domain feature learning and cross-dimensional feature optimization stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When run in the first processor 21, it can achieve the following:

[0116] The pre-defined dual-domain feature encoder is used to enhance and extract features from foggy nighttime images. The dual-domain feature encoder includes a frequency domain learning branch and a spatial domain learning branch. The extraction of features from foggy nighttime images includes the following steps: First, in the frequency domain learning branch, the features of foggy nighttime images are transformed into the frequency domain using an adaptive frequency selection module. Then, frequency features are adaptively selected according to a set importance weight threshold to obtain the frequency features of the foggy nighttime images. Second, in the spatial domain learning branch, spatial domain features of the foggy nighttime images are extracted using residual dense blocks. Finally, the frequency features and spatial domain features are fused.

[0117] The fused features are input into the cross-dimensional attention feature optimization module. The contextual relationship information of the image is obtained through the multi-scale dilated convolutional structure. Then, the cross-dimensional interactive attention module is set to calculate the average importance weight of the fused features in the channel, height and width dimensions, so as to obtain optimized features of color and brightness that are equally important in multiple dimensions.

[0118] The optimized features are input into a preset dual-domain feature decoder to obtain a clear nighttime image. Furthermore, if the modules / units integrated into the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization, characterized in that, Includes the following steps: The features of foggy nighttime images are enhanced and extracted using a pre-defined dual-domain feature encoder, which includes a frequency domain learning branch and a spatial domain learning branch. Extracting features from foggy nighttime images involves the following steps: First, in the frequency domain learning branch, the features of the foggy nighttime image are transformed into the frequency domain using an adaptive frequency selection module. Then, frequency features are adaptively selected based on the set importance weight threshold to obtain the frequency features of the foggy nighttime image. Second, in the spatial domain learning branch, spatial domain features of the foggy nighttime image are extracted using residual dense blocks. Finally, the frequency features and spatial domain features are fused. The fused features are input into the cross-dimensional attention feature optimization module. A multi-scale dilated convolutional structure is used to obtain the contextual information of the image. Then, a cross-dimensional interactive attention module is set to calculate the average importance weights of the fused features in the channel, height, and width dimensions, obtaining optimized features for color and brightness that are equally important in multiple dimensions. The cross-dimensional interactive attention module includes channel-channel branches, height-channel branches, and width-channel branches. In the channel-channel branches, global average pooling and global max pooling are combined to perform horizontal and vertical sampling on the input features. In the height-channel branches, shallow features are learned using the input features through 1×1 and 3×3 convolutional layers, and then rotated 90 degrees counterclockwise along the H-axis to obtain the rotated shallow feature X2. In the width-channel branches, the rotation axis is changed from the H-axis to the W-axis; the remaining structure is the same as the height-channel branches. The optimized features are input into a preset dual-domain feature decoder to obtain clear nighttime images; In the channel-channel branch, the sampled features are added and connected by directional grouping, channel information is exchanged using channel shuffling, and then convolution is performed to reduce the feature dimension. A batch normalization layer is then processed using the non-linear activation function ELU. Subsequent convolution is used to recover the number of feature channels in different directions, so that features in the same direction share similar weights, resulting in interactive features.

2. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 1, wherein the frequency domain learning branch includes an adaptive frequency selection module, characterized in that, It also includes the following steps: Two-dimensional discrete cosine transform is used to convert the features of foggy nighttime images from the spatial domain to the frequency domain; a learnable importance weight threshold is defined, and the features of foggy nighttime images are dynamically zeroed out; then, an adaptive frequency selection module compares the average importance weight of the features of foggy nighttime images with the importance weight threshold. If the average importance weight is higher than the importance weight threshold, the feature is retained; otherwise, it is discarded.

3. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 1, characterized in that, The spatial domain learning branch includes residual dense blocks. Within this branch, spatial domain features of foggy nighttime images are extracted using these residual dense blocks. Specifically: First, spatial features are extracted from foggy nighttime images using six 3×3 convolutional layers and then concatenated together. Next, a 1×1 convolutional layer is added to reduce the network width. Finally, residual learning is performed between the output features and the original input features to obtain the spatial domain features of the image.

4. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 1, characterized in that, Obtaining optimized features for color and brightness that are equally important across multiple dimensions also includes the following steps: Two 1×1 convolutional layers and a multi-scale dilated convolutional structure are used to obtain multi-scale image features with contextual information; the multi-scale image features are concatenated and fused, and then input into a cross-dimensional interactive attention module; the multi-scale dilated convolutional structure consists of four 3×3 convolutional branches with dilation rates of 1, 2, 3, and 5, respectively. In the height-channel branch, a self-attention mechanism is used to calculate feature weights and obtain the correlation between each element in the shallow feature X2, as shown in the following formula: Among them, Q w Indicates querying the data stream, K w V represents a key-value data stream. w d represents a numerical data stream. k The dimension representing the key value.

5. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 4, characterized in that, The channel-to-channel branch also includes steps to supplement and improve the fused feature information, specifically: A simplified nonlocal attention CRLC is used to obtain effective global information, as follows: Among them, X C For input features, For the output features of channel-to-channel branches, SC(·) represents the SC structure, which is used to reduce the channel dimension of the input features; The simplified nonlocal attention CRLC includes 1×1 convolution, ReLU function and LayerNorm, and the SC structure includes 1×1 convolution and Softmax function.

6. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 1, characterized in that, The step of inputting optimized features into a preset dual-domain feature decoder further includes the following steps: Using mean square error loss L mse With perceived loss L p Network optimization includes: L=L mse +λL p Where L represents the total loss, x i Indicates the input image, y i Represents the target image. Let F represent the predicted value of the target image features, λ represent the weights of the loss function, and F... i (x) and F i (y) represents their feature representation in the i-th layer of the pre-trained neural network, where N is the number of feature layers.

7. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 1, characterized in that, The dual-domain feature encoder uses downsampling for feature extraction.

8. The nighttime dehazing method based on dual-domain feature learning and cross-dimensional feature optimization according to claim 1, characterized in that, The dual-domain feature decoder uses upsampling for decoding, and the rest of the structure is consistent with the dual-domain feature encoder.

9. A nighttime defogging system based on dual-domain feature learning and cross-dimensional feature optimization, characterized in that, The nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization, applied to any one of claims 1-8, includes a dual-domain feature encoding module, a feature optimization module, and a dual-domain feature decoding module; The dual-domain feature encoding module is used to enhance and extract features of foggy nighttime images using a preset dual-domain feature encoder, which includes a frequency domain learning branch and a spatial domain learning branch. Extracting features from foggy nighttime images involves the following steps: First, in the frequency domain learning branch, the features of the foggy nighttime image are transformed into the frequency domain using an adaptive frequency selection module. Then, frequency features are adaptively selected based on the set importance weight threshold to obtain the frequency features of the foggy nighttime image. Second, in the spatial domain learning branch, spatial domain features of the foggy nighttime image are extracted using residual dense blocks. Finally, the frequency features and spatial domain features are fused. The feature optimization module inputs the fused features into the cross-dimensional attention feature optimization module. It then uses a multi-scale dilated convolutional structure to obtain the image's contextual information. Finally, the cross-dimensional interactive attention module calculates the average importance weights of the fused features across the channel, height, and width dimensions, resulting in optimized features for color and brightness that are equally important across multiple dimensions. The dual-domain feature decoding module is used to input optimized features into a preset dual-domain feature decoder to obtain clear nighttime images.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the nighttime defogging method based on dual-domain feature learning and cross-dimensional feature optimization as described in any one of claims 1-8.

Images