A method and electronic device for dehazing nighttime foggy images based on image synthesis
By synthesizing nighttime foggy images and combining them with dynamic prompts and spatial guidance modules, the problem of insufficient adaptability in existing nighttime foggy image dehazing methods is solved, generating images with natural brightness, realistic colors, and clear structural details.
Patent Information
- Application Number
- CN202611259726.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing dehazing methods for nighttime foggy images struggle to adapt to different nighttime fog, low illumination, artificial halos, and local reflection degradation while maintaining textual semantic constraints. This results in unnatural brightness, inaccurate colors, and unclear structural details in the restored images.
By combining clear daytime images and original depth maps, nighttime foggy images are synthesized. Conditional latent variables and fixed text conditions are generated by freezing the VAE encoder and freezing the text encoder. Residual features are generated using IRControlNet. Dynamic cue words and spatial guidance features are generated through dynamic cue and spatial guidance modules. Multi-time step back diffusion and latent variable decoding are performed to generate a dehazed image with natural brightness, true color and clear structural details.
It achieves adaptive dehazing of nighttime foggy images, generating restored images with natural brightness, true colors, and clear structural details, thus improving image quality.
Smart Images

Figure CN123118403A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of image dehazing and deep learning technology, specifically to a method and electronic device for dehazing nighttime foggy images based on image synthesis. Background Technology
[0002] Compared to the uniform illumination during the day, images of foggy nights exhibit complex degradation characteristics: (1) Non-uniform illumination and multiple light sources: Foggy night scenes rely on artificial light sources such as streetlights and car lights, resulting in a large difference in brightness between bright areas (near the light source) and dark areas (backlit areas) in the image, and the overall histogram distribution of foggy night images is uneven. (2) Low contrast and loss of detail: Fog absorbs and scatters reflected light, and coupled with insufficient ambient light, the contrast of the edges of distant objects is extremely low, and the texture of the target is blurred or missing. (3) Halo and glare effects: When light passes through fog droplets, Mie scattering occurs, forming a large and diffuse halo around the light source, which obscures the outline and texture of objects behind the light source. (4) Sensor noise: To compensate for insufficient illumination, the ISO (sensitivity) of the camera is usually increased, which introduces a large amount of Gaussian-Poisson mixed noise in the dark areas of the captured foggy night images.
[0003] Existing methods for dehazing nighttime fog images include those based on physical models and human priors, those based on nighttime dehazing datasets, deep learning-based nighttime dehazing methods, and cue-based and diffusion-based image restoration methods. Methods based on physical models and human priors use nighttime imaging models to describe the impact of non-uniform lighting on foggy nighttime images. Based on this, MRP and FMRP methods use maximum reflectance priors to estimate nighttime fog degradation. Another approach is to add an additive glow term to the atmospheric scattering model and use glow decomposition to suppress halos around artificial light sources. OSFD uses optimal scale fusion to estimate nighttime dehazing parameters. These methods offer some physical interpretability, but rely on pre-defined imaging assumptions and human priors, limiting their adaptability to degradation in complex and non-uniform real-world nighttime scenes. Methods based on nighttime dehazing datasets construct paired datasets by applying atmospheric scattering models to clear nighttime images. The 3R synthesis framework integrates scene geometry, artificial lighting, object reflection, and fog effects to form NHC, NHM, and NHR datasets. GTA5 uses its game engine to generate foggy nighttime scenes, while UNREAL-NH utilizes Unreal Engine to control lighting, fog density, and scene variations. These datasets facilitate supervised learning, but some data do not simultaneously simulate the semantic location of the light source, illumination direction, depth-related propagation, and reflection from surrounding surfaces, resulting in discrepancies between the synthetic and real domains.
[0004] Deep learning-based nighttime dehazing methods, such as HDP-Net, use density prediction networks to recover clear nighttime images. GAPSF jointly suppresses glow and enhances dark areas through light source-aware learning and gradient adaptive feature extraction. NightHazeFormer combines physical priors with a Transformer structure, SFSNiD utilizes spatial-frequency feature interactions and pseudo-label training to handle complex nighttime degradation, and NightHaze learns real-world scene self-priorities through data augmentation and simulating lighting effects and noise. Furthermore, studies such as CGID, CoA, and PDDA attempt to unify daytime and nighttime dehazing or perform real-domain adaptation. These discriminative models typically learn the mapping from degraded images to target images directly, which may lead to insufficient brightness, color, and detail recovery when faced with complex nighttime lighting outside the training set. Hint and diffusion image restoration methods, such as PromptIR, use hint mechanisms to handle various image restoration tasks. DiffUIR and Diff-Plugin use diffusion-generated priors for general and detail restoration. DiffBIR uses a frozen latent space diffusion model and proposes the IRControlNet conditional control branch for blind image restoration. However, fixed text cues are insufficient to express the unique fog density, local glow, low illumination, and reflection characteristics of each foggy nighttime image. A single shared cue cannot meet the different requirements of U-Net's encoding, intermediate, and decoding stages for global structure, semantic regions, and local textures.
[0005] In summary, there is an urgent need for a method that, while maintaining textual semantic constraints, implements input-adaptive, temporal-adaptive, and spatial-adaptive generative prior modulation for different nighttime haze, low illumination, artificial halos, and local reflection degradation, and obtains a restored image with natural brightness, true color, and clear structural details after multi-time-step backdiffusion and latent variable decoding. Summary of the Invention
[0006] To overcome the problems existing in related technologies, the purpose of this invention is to provide a method and electronic device for dehazing nighttime foggy images based on image synthesis. While maintaining textual semantic constraints, the method implements input-adaptive, time-adaptive, and space-adaptive generation prior modulation for different nighttime fog, low illumination, artificial halos, and local reflection degradation. After multi-time-step back diffusion and latent variable decoding, a restored image with natural brightness, true color, and clear structural details is obtained.
[0007] A method for dehazing nighttime foggy images based on image synthesis, comprising: Combine clear daytime images with the original depth map to create a foggy nighttime image; The nighttime foggy image is mapped and input into a frozen VAE encoder for image encoding to obtain VAE conditional latent variables; The preset text is input into the frozen text encoder for text encoding to obtain fixed text conditions; Input the VAE conditional latent variables, fixed text conditions, noisy latent variables, and time steps into IRControlNet to generate M residual features; divide the M residual features into shallow feature groups, medium feature groups, and deep feature groups; Dynamic prompts and spatial guidance are generated based on shallow feature groups, mid-level feature groups and deep feature groups to produce dynamic prompt words and spatial guidance features. The dynamic and fixed prompts are concatenated to obtain all prompts; a complete attention map is generated based on all prompts. Extract the spatial dynamic attention subgraph corresponding to the second half of the dynamic prompt words from the complete attention map, and use the spatial dynamic attention subgraph to generate spatially perceptual dynamic prompt words from the spatial guidance features; Concatenate the fixed text value vector corresponding to the fixed prompt word, the non-spatial value vector corresponding to the first half of the dynamic prompt words, and the spatial value vector corresponding to the spatially aware dynamic prompt words along the token dimension to obtain the complete value vector. Multiply the complete value vector by the complete attention map and project it to obtain the cross-attention output; The cross-attention output is input into the frozen U-Net and backdiffusion is performed to obtain the dehazed daytime image.
[0008] The beneficial effects of this invention are as follows: The present invention provides a nighttime foggy image dehazing method based on image synthesis, which involves combining a clear daytime image and the original depth map to synthesize a nighttime foggy image. The synthesized nighttime foggy image is used to train the overall method architecture of the present invention. The nighttime foggy image is mapped and input into a frozen VAE encoder for image encoding to obtain VAE conditional latent variables. A preset text is input into a frozen text encoder for text encoding to obtain fixed text conditions. The VAE conditional latent variables, fixed text conditions, noisy latent variables, and time steps are input into IRControlNet to generate M residual features. These M residual features are divided into shallow, medium, and deep feature groups. Dynamic prompts and spatial guidance are performed based on the shallow, medium, and deep feature groups to generate dynamic prompt words and spatial guidance features. The dynamic prompt and spatial guidance module DPSG consists of a dynamic prompt generation module DPGM, a spatial guidance feature generation module SGFM, and a spatially aware cross-attention module. The dynamic prompt generation module mainly describes what degradation exists in the current input image, i.e., it generates dynamic prompt words that match the current input image. The spatial guidance feature generation module primarily describes the location of these degradations in the image, i.e., preserving the two-dimensional spatial distribution of degradation features. Dynamic and fixed cue words are concatenated to obtain all cue words; a complete attention map is generated based on all cue words. From the complete attention map, a spatial dynamic attention sub-map corresponding to the second half of the dynamic cue words is extracted, and spatially aware dynamic cue words are generated from the spatial guidance features using this sub-map. The fixed text value vectors corresponding to the fixed cue words, the non-spatial value vectors corresponding to the first half of the dynamic cue words, and the spatial value vectors corresponding to the spatially aware dynamic cue words are concatenated along the token dimension to obtain the complete value vector. The first half of the dynamic cue words serves as the regular dynamic cue, and the non-spatial value vectors are used to provide global degradation conditions for the entire image. The second half of the dynamic cue words serves as spatial readout slots, and the spatial dynamic attention sub-map is used to determine the image region that each spatial cue token focuses on. Information from the corresponding region is weighted and read from the two-dimensional spatial guidance features of the spatial guidance feature generation module using the spatial dynamic attention sub-map to generate a spatial value vector. The complete value vector is multiplied by the complete attention map and projected to obtain the cross-attention output. The cross-attention output is frozen into a U-Net and backdiffusion is performed to obtain a dehazed daytime image. This invention, while maintaining textual semantic constraints, implements input-adaptive, temporal-adaptive, and spatial-adaptive generative prior modulation for different nighttime fog, low illumination, artificial halos, and local reflection degradation. After multi-time-step backdiffusion and latent variable decoding, a dehazed daytime image with natural brightness, realistic color, and clear structural details is obtained. Attached Figure Description
[0009] Figure 1 This is the overall architecture of the nighttime foggy image dehazing method based on image synthesis of the present invention; Figure 2 This is a structural diagram of the dynamic prompting and spatial guidance module of the present invention; Figure 3 This is a structural diagram of the dynamic prompt generation module of the present invention; Figure 4 This is a structural diagram of the spatial guidance feature generation module of the present invention; Figure 5 This is a diagram showing the composition of the spatial perception cross-attention module of the present invention; Figure 6 This is a flowchart illustrating the generation process of the RINH nighttime haze dataset of this invention. Figure 7 This is a comparison of the image dehazing method of the present invention with other image dehazing methods; Figure 8 These are clear daytime images and foggy nighttime images from the RINH nighttime haze dataset of this invention. Detailed Implementation
[0010] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0011] Example 1 This embodiment provides a method for dehazing nighttime foggy images based on image synthesis, including: S1: Combine clear daytime images with the original depth map to synthesize foggy nighttime images; S2: Map the night foggy image and input it into the frozen VAE encoder for image encoding to obtain the VAE conditional latent variables; S3: Input the preset text into the frozen text encoder for text encoding to obtain fixed text conditions; S4: Input the VAE conditional latent variables, fixed text conditions, noisy latent variables and time steps into IRControlNet to generate M residual features; divide the M residual features into shallow feature groups, medium feature groups and deep feature groups; S5: Based on shallow feature groups, medium feature groups and deep feature groups, dynamic prompts and spatial guidance are generated to produce dynamic prompt words and spatial guidance features; S6: Concatenate the dynamic prompts and fixed prompts to obtain all prompts; generate a complete attention map based on all prompts; S7: Extract the spatial dynamic attention sub-graph corresponding to the second half of the dynamic prompt words from the complete attention graph, and use the spatial dynamic attention sub-graph to generate spatially perceptive dynamic prompt words from the spatial guidance features; S8: Concatenate the fixed text value vector corresponding to the fixed prompt word, the non-spatial value vector corresponding to the first half of the dynamic prompt words, and the spatial value vector corresponding to the spatially aware dynamic prompt words along the token dimension to obtain the complete value vector; S9: Multiply the complete value vector with the complete attention map and project it to obtain the cross-attention output; S10: Input the cross-attention output into the frozen U-Net and perform backdiffusion to obtain the dehazed daytime image.
[0012] In this invention, clear daytime images and synthesized foggy nighttime images are paired together. The foggy nighttime images are used to train and test the overall dehazing architecture DP-Diff. The architecture diagram of DP-Diff is shown below. Figure 1 As shown.
[0013] The process of mapping the nighttime foggy image and inputting it into a frozen VAE encoder for image encoding yields VAE conditional latent variables, including: S21: The nighttime foggy image is preprocessed to obtain a preprocessed image. The preprocessing includes scaling and size padding. The pixel values of the pixels in the preprocessed image are greater than or equal to 0 and less than or equal to 1. S22: Multiply the preprocessed image by 2 and then subtract 1 to obtain the mapped image; S23: Input the mapped image into the frozen VAE encoder for image encoding to obtain VAE conditional latent variables.
[0014] Reading images of foggy nighttime conditions I Original size ( H 0 , W 0 ), H 0 Original height W 0 The original width is used. When any side of the nighttime foggy image is smaller than the preset minimum size of 512, the shorter side of the nighttime foggy image is scaled to that size while maintaining the aspect ratio. Then, the scaled width and height are padded to integer multiples of the VAE (Variational Autoencoder) downsampling factor. Preferably, the product of the scaled width and height is set to an integer multiple of 64 to obtain the preprocessed image. , .
[0015] Using formula (1), the preprocessed image is first transferred from... Mapped to Then expand the scope to Mapped image within Input freeze VAE encoder: (1) in, This indicates that the VAE encoder is frozen. This represents the conditional latent variables of the VAE. Freezing the VAE encoder means keeping all network parameters of the VAE encoder unchanged during training, so that the network parameters do not participate in gradient updates. The purpose of freezing the VAE encoder is to preserve and utilize prior knowledge and save computational resources. The VAE encoder has a downsampling factor of 8 and 4 output channels, therefore... .
[0016] The preset text is encoded using formula (2): (2) in, This indicates the preset text. This indicates that the text encoder is frozen. This indicates a fixed text condition. Freezing the text encoder means fixing all network parameters of the text encoder during training, so that the network parameters do not participate in gradient updates. The effect of freezing the text encoder is similar to that of freezing the VAE encoder, and will not be elaborated further here. In this embodiment, the text sequence length... Text context dimension Therefore, fixed text conditions .
[0017] During the training phase, forward diffusion is performed, first encoding the sharp target image as a sharp latent variable: (3) in, Representing clear latent variables, This indicates that the VAE encoder is frozen. Represents a clear target image. .
[0018] Using formulas (4)-(5) for random sampling time steps t and image noise : (4) (5) Indicates from 0 to T Uniform distribution on -1, This represents a Gaussian distribution with a mean of 0 and a variance of 1.
[0019] This embodiment takes T=1000 as an example, and constructs a time step according to a preset noise scheduling. t The noisy latent variables, with preset noise scheduling following the settings of the pre-trained Stable Diffusion v2-1, are not considered improvements to this invention. Clear latent variables... Noisy latent variables and image noise The dimensions are consistent. .
[0020] During forward diffusion, image noise is added to the sharp latent variable, and a signal attenuation coefficient is used to adjust the ratio between the sharp latent variable and the image noise. Formula (6) is used to calculate any time step. t Noisy latent variables: (6) in, Indicates time step t The signal attenuation coefficient is such that a larger signal attenuation coefficient indicates a larger proportion of clear latent variables are retained and less image noise is added. The function of formula (6) is to directly calculate the image noise at any time step during the forward diffusion process in the training phase without gradually adding image noise to the clear latent variables. t The corresponding noisy latent variables.
[0021] This embodiment provides a nighttime foggy image dehazing method based on image synthesis, which combines a clear daytime image and the original depth map to synthesize a nighttime foggy image. The synthesized nighttime foggy image is used to train the overall method architecture of this invention. The nighttime foggy image is mapped and input into a frozen VAE encoder for image encoding to obtain VAE conditional latent variables. Preset text is input into a frozen text encoder for text encoding to obtain fixed text conditions. The VAE conditional latent variables, fixed text conditions, noisy latent variables, and time steps are input into IRControlNet to generate M residual features. These M residual features are divided into shallow, medium, and deep feature groups. Dynamic prompts and spatial guidance are performed based on the shallow, medium, and deep feature groups to generate dynamic prompt words and spatial guidance features. The dynamic prompt and spatial guidance module DPSG consists of a dynamic prompt generation module DPGM, a spatial guidance feature generation module SGFM, and a spatially aware cross-attention module. The dynamic prompt generation module mainly describes what degradation exists in the current input image, i.e., it generates dynamic prompt words that match the current input image. The spatial guidance feature generation module primarily describes the location of these degradations in the image, i.e., preserving the two-dimensional spatial distribution of degradation features. Dynamic and fixed cue words are concatenated to obtain all cue words; a complete attention map is generated based on all cue words. From the complete attention map, a spatial dynamic attention sub-map corresponding to the second half of the dynamic cue words is extracted, and spatially aware dynamic cue words are generated from the spatial guidance features using this sub-map. The fixed text value vectors corresponding to the fixed cue words, the non-spatial value vectors corresponding to the first half of the dynamic cue words, and the spatial value vectors corresponding to the spatially aware dynamic cue words are concatenated along the token dimension to obtain the complete value vector. The first half of the dynamic cue words serves as the regular dynamic cue, and the non-spatial value vectors are used to provide global degradation conditions for the entire image. The second half of the dynamic cue words serves as spatial readout slots, and the spatial dynamic attention sub-map is used to determine the image region that each spatial cue token focuses on. Information from the corresponding region is weighted and read from the two-dimensional spatial guidance features of the spatial guidance feature generation module using the spatial dynamic attention sub-map to generate a spatial value vector. The complete value vector is multiplied by the complete attention map and projected to obtain the cross-attention output. The cross-attention output is frozen into a U-Net and backdiffusion is performed to obtain a dehazed daytime image. This invention, while maintaining textual semantic constraints, implements input-adaptive, temporal-adaptive, and spatial-adaptive generative prior modulation for different nighttime fog, low illumination, artificial halos, and local reflection degradation. After multi-time-step backdiffusion and latent variable decoding, a dehazed daytime image with natural brightness, realistic color, and clear structural details is obtained.
[0022] Example 2 This embodiment further describes the scheme of Embodiment 1 based on Embodiment 1. This invention uses the IRControlNet proposed by Diff-BIR as the conditional control branch. The improvement lies in utilizing the 13-layer degradation residual features output by this branch to further construct the Dynamic Cueing and Spatial Guidance Module (DPSG). Compared to existing image restoration methods that use fixed text cueing, single global conditions, or the same cueing for all U-Net layers, the main advantage of DPSG is that it simultaneously achieves conditional guidance with input adaptation, diffusion time step adaptation, U-Net stage adaptation, and spatial location adaptation. Figure 2 As shown, DPSG consists of a dynamic cue generation module DPGM, a spatially guided feature generation module SGFM, and a spatially perceptive cross-attention process. DPGM mainly describes what degradation exists in the current input image, while SGFM mainly describes where these degradations occur.
[0023] In Example 1 , , and t Enter IRControlNet: (7) Among them, IRControlNet is the second-stage core module in the Diff-BIR image restoration framework. It mainly utilizes the generative capabilities of the latent diffusion model to generate realistic and delicate details for the initially restored image. This represents the trainable IRControlNet parameters. Indicates time step t The corresponding noisy latent variable, Represents the conditional latent variable in VAE. Indicates a fixed text condition. This represents the set of residual features consisting of 13 residual features. Indicates the first k Each residual feature.
[0024] IRControlNet uses a pre-trained U-Net encoder and intermediate blocks from Stable Diffusion to construct its conditional network. During implementation, it will... and By concatenating along the channel dimension, the network can simultaneously perceive the current generation state and the input degraded image, while fixing text conditions. Entering the cross-attention layer of IRControlNet, time step After time embedding, it enters the residual block.
[0025] The 12 encoding / downsampling positions and one intermediate position of IRControlNet generate residual features through zero-initialized 1×1 convolutions. The Dynamic Cueing and Spatial Guidance Module (DPSG) further utilizes 13 independent 1×1 convolutions. Align different channel counts to a unified cue hidden space dimension : (8) in, Indicates the first k Aligned residual features Indicates the first k There are 1 x 1 convolutions, and 13 independent 1 x 1 convolutions. k The size of each residual feature is After channel alignment is completed through independent 1×1 convolutions, In this embodiment It equals 1024. Independent projection can preserve the representational differences of different network depths and avoid channel mismatch when directly adding them.
[0026] The method of generating dynamic prompts and spatial guidance based on shallow, medium, and deep feature groups, including: S51: Obtain the set of learnable weights, and normalize the set of learnable weights to obtain the normalized weight set. S52: Perform global average pooling on each aligned residual feature in the shallow feature group, the middle feature group and the deep feature group to obtain M spatial dimensional pooling vectors. S53: Using the normalized weight set, within the same feature group, the pooled vectors of each spatial dimension in the shallow feature group, the middle feature group, and the deep feature group are weighted and summed to obtain the group-level global representation; the group-level global representation includes the shallow global representation, the middle global representation, and the deep global representation. S54: Perform bilinear interpolation on each aligned residual feature in the shallow feature group, middle feature group and deep feature group to obtain the interpolated residual feature; S55: The interpolated residual features are weighted and summed using the normalized weight set to obtain a group-level spatial representation; the group-level spatial representation includes shallow spatial representation, medium spatial representation and deep spatial representation. S56: Input the group-level global representation into the dynamic prompt generation module to generate dynamic prompt words; S57: Input the group-level spatial representation into the spatial guided feature generation module to generate spatial guided features.
[0027] The 13 aligned residual features output by IRControlNet are input into the Dynamic Cueing and Spatial Guidance Module (DPSG), which divides the 13 aligned residual features into shallow feature groups. G sh Middle-layer feature group G mid and deep feature groups G de . G sh Depend on F 1 , F 2 , F 3 , F 4 , F 5 and F 6 composition, G mid Depend on F 7 , F 8 and F 9 composition, G de Depend on F 10 , F 11 , F 12 and F 13 Composition. The shallow feature set mainly includes edges, local textures, and fine-grained degradation; the medium-level feature set mainly includes medium-scale structures; and the deep feature set mainly includes high-level semantics and large receptive field degradation.
[0028] No. l Group 1 k The learnable weights corresponding to the aligned residual features are: , All learnable weights are grouped into a learnable weight set, and within the set, the learnable weights are adaptively aggregated, i.e., normalized, using the Softmax function. (9) in, Indicates the first l Group 1 k The normalized learnable weights corresponding to the aligned residual features Indicates the first l Group 1 j The learnable weights corresponding to the aligned residual features, where exp represents an exponential function with the natural constant as the base. Representing feature groups l As an example, This represents the normalized learnable weight of the 8th aligned residual feature in the middle-layer feature group. This represents the normalized learnable weight of the second aligned residual feature in the shallow feature group. The normalized learnable weights corresponding to all aligned residual features are used as the set of normalized weights.
[0029] right Perform global average pooling in the spatial dimension: (10) in, c Indicates the first k Channel number of the aligned residual feature. The dimension of the unified hint latent space is the first... k The total number of channels of the aligned residual feature. x Indicates the first k The x-coordinate of the aligned residual feature. y Indicates the first k The ordinate of the aligned residual features. Indicates the first k The height of the aligned residual feature Indicates the first k The width of the aligned residual feature. This indicates a global average pooling operation. Indicates the first k A spatial-dimensional pooling vector of aligned residual features.
[0030] Calculated using formula (9) The first one calculated by formula (10) k Weighted summation of spatial-dimensional pooled vectors of aligned residual features: (11) in, This represents a group-level global representation. Including shallow global representation Mid-level global representation and deep global representation , , and .
[0031] The present invention The value is 1024. Global average pooling is performed on the feature maps in each channel to obtain a single value. These 1024 values are then aggregated to form a 1024-dimensional vector. Then, by using a weighted summation method within groups, three different levels of global representation are obtained, including a shallow global representation. Mid-level global representation and deep global representation Each is a 1024-dimensional vector. As an example, The group has 7-9 three aligned residual features, for , and Global average pooling is performed separately to obtain three 1024-dimensional vectors. , and We obtain the mid-level global representation by performing a weighted summation of three 1024-dimensional vectors. .
[0032] Feature group The highest spatial resolution in the interior is , each Bilinear interpolation is: (12) This represents bilinear interpolation. express The highest resolution in the interior height, express The highest resolution of the inner width. Indicates the first l Group 1 k The residual characteristics after interpolation.
[0033] The weights can be learned using the same normalization as the group-level global representation branch, and the group-level spatial representation is computed within each feature group: (13) in, Indicates the first l Group-level spatial representation, Including shallow spatial representation Mid-level spatial representation and deep space representation Group-level spatial representations and group-level global representations share the same weighted fusion weights, ensuring that layers within the same group that contribute significantly to global hints participate in spatial guidance construction in the same proportion.
[0034] Example 3 This embodiment, based on Embodiments 1 and 2, further describes the generation method of dynamic prompt words and spatial guidance features. The step of inputting group-level global representations into the dynamic prompt generation module to generate dynamic prompt words includes: S561: Perform sinusoidal position encoding, linear transformation and SiLU activation on the time step to obtain the time step embedding; S562: The shallow global representation, the middle global representation, and the time step embedding are concatenated along the feature dimension, and linear projection, encoding stage embedding, and layer normalization are performed to obtain the encoding stage features. S563: The intermediate global representation, deep global representation and time step embedding are concatenated along the feature dimension, and then linearly projected, intermediate stage embedding and layer normalization are added to obtain the intermediate stage features. S564: The shallow global representation, the middle global representation, the deep global representation and the time step embedding are concatenated along the feature dimension, and linear projection, decoding stage embedding and layer normalization are added to obtain the decoding stage features. S565: Perform linear projection on the features of the encoding stage, the intermediate stage, and the decoding stage respectively, and add the corresponding stage bias to obtain the dynamic prompt words of the encoding stage, the dynamic prompt words of the intermediate stage, and the dynamic prompt words of the decoding stage.
[0035] Calculate time steps t Sine position coding with the same number of basic channels as IRControlNet: (14) in, Indicates time step t Sine time step embedding, Represents the sine position code. In this embodiment, the encoding dimension is represented. It equals 320.
[0036] The sinusoidal time-step embedding is sequentially input into the first fully connected network, the SiLU activation function, and the second fully connected network to generate the time-step embedding. : (15) in, This represents the first layer of a fully connected network, and SiLU represents the Sigmoid linear unit. This indicates a Layer 2 fully connected network. Both the first and second fully connected layers are used for linear transformations.
[0037] The encoding stage fuses shallow details and mid-level structure, the intermediate stage fuses mid-level structure and deep semantics, and the decoding stage fuses shallow details, mid-level structure, and deep semantics. Each stage first concatenates its required group-level global features with the temporal step embeddings along the feature dimension, then performs linear projection through the corresponding fully connected layer, followed by the addition of the corresponding stage embedding and layer normalization. (16) (17) (18) in, Indicates characteristics of the encoding stage. Indicates characteristics of intermediate stages. This indicates the characteristics of the decoding stage. This indicates layer normalization during the encoding phase. This indicates layer normalization in the intermediate stage. This indicates layer normalization during the decoding phase. This indicates the fusion fully connected layer corresponding to the encoding stage. This indicates the fusion fully connected layer corresponding to the intermediate stage. This represents the fusion fully connected layer corresponding to the decoding stage. The fusion fully connected layer in the three stages is used to perform unified projection on the spliced multi-source features and temporal features. This indicates embedding during the encoding stage. This indicates embedding in the intermediate stage. This indicates embedding during the decoding stage. This indicates the concatenation of feature dimensions. This represents a shallow global representation. This represents the mid-level global representation. This represents a deep global representation, with a three-stage fusion fully connected layer used for linear projection.
[0038] like Figure 3 As shown, the Dynamic Prompt Generation (DPGM) module uses a sequence of concatenating multi-source features with temporal features, stage linear projection, adding stage embedding, and layer normalization. Feature dimension concatenation is used to preserve degradation information at different levels, stage linear projection is used to learn the joint relationship between various sources, and stage embedding is used to distinguish between the encoding stage, intermediate stage, and decoding stage.
[0039] Let the total number of dynamic prompts generated in each stage be . The channel dimension of a single dynamic prompt word is In the Dynamic Cue Generation (DPGM) module, regular cue headers and spatial cue headers are not pre-set. Instead, all dynamic cue words for a given stage are generated at once by a stage-specific cue word linear projection layer. (19) in, For the stage r The corresponding fully connected projection layer for dynamic prompt words outputs as follows: indivual Dynamic prompt words. For the stage rThe corresponding learnable cue word slot offset is used to enable different cue word slots at the same stage to have different functions. and The same size , For the cross-attention context dimension, this embodiment It equals 1024. After Gaussian initialization and model learning, different stages and dynamic prompts are generated. Different. For example, there are 3 stages, 16 dynamic prompt words, and 3×16 learnable prompt word slot biases with a learnable dimension of 1024.
[0040] h e After passing through the fully connected projection layer of the dynamic cue words in the encoding stage, and adding the corresponding learnable cue word slot bias in the encoding stage, the dynamic cue words in the encoding stage are obtained. . h d After passing through the fully connected projection layer of the dynamic cue words in the decoding stage, and adding the slot offset of the learnable cue words corresponding to the decoding stage, the dynamic cue words of the decoding stage are obtained. . h m After passing through the fully connected projection layer of the dynamic cue words in the intermediate stage, and adding the corresponding learnable cue word slot bias for the intermediate stage, we obtain the dynamic cue words for the intermediate stage. .
[0041] Preferred , , , This represents the total number of dynamic prompt words. This represents the number of non-spatial-aware dynamic prompts, corresponding to the first half of the dynamic prompts. The number of spatially aware dynamic prompts corresponds to the second half of the dynamic prompts. Each of the three stages generates its own 16 tokens, instead of sharing a fixed set of tokens across all U-Net layers. For example, the encoding stage generates 16 dynamic prompts. The model also includes 16 different learnable cue word slot biases, each a vector. The first dynamic cue word is added to the first learnable cue word slot bias, the second dynamic cue word to the second learnable cue word slot bias, and so on, generating the final dynamic cue words for each stage. This assigns each cue word slot an independent identity, preventing them from learning completely identical or highly similar content. After training, different cue words may have their own preferences, focusing more on information such as haze distribution, sky brightness, light source position, or scene structure. These functions are not manually specified in advance but are automatically formed after model training.
[0042] The Dynamic Cue Generation (DPGM) module does not distinguish between cue types when generating dynamic cue words. After entering spatial awareness cross-attention, it then assigns the cue words to preset slots. The dynamic prompt words are used as non-spatial-aware dynamic prompt words, and the following... Each dynamic prompt word serves as a spatial read slot, among which This classification only specifies the use of dynamic cues in subsequent attention.
[0043] The Dynamic Cue Generation (DPGM) module excels at generating dynamic cues that match the current input image. DPGM extracts shallow, intermediate, and deep global representations from multi-layer residual features derived from IRControlNet and combines these with the current diffusion time step and U-Net stage identifiers to generate dynamic cues for the encoding, intermediate, and decoding stages, respectively. Compared to fixed cues, these dynamic cues can adjust to actual degradations in each input image, such as haze concentration, low illumination, color shift, halo, and reflection, reducing the discrepancy between fixed text descriptions and real degradations.
[0044] The encoding, intermediate, and decoding stages use different feature sources, stage embeddings, and cue projection layers. The encoding stage focuses more on scene structure and degradation distribution, the intermediate stage focuses on high-level semantics, and the decoding stage focuses on local texture and detail reconstruction. Therefore, DPGM can provide conditions that match the functions of cross-attention at different levels, rather than reusing the same set of cues throughout the entire U-Net.
[0045] The module that inputs group-level spatial representations into spatial guiding feature generation to generate spatial guiding features includes: The following formulas are used to calculate the spatial guidance features of the encoding stage, the intermediate stage, and the decoding stage: (20) ;(twenty one) ;(twenty two) in, This represents the spatial guidance features of the encoding stage. These represent the shallow fusion parameters during the encoding phase. This represents a shallow spatial representation. This represents the mid-layer fusion parameters during the encoding phase. This represents the mid-level spatial characterization. This indicates the target resolution during the encoding stage. This indicates that the resolution of the mid-level spatial representation is scaled down to the target resolution during the coding stage; This indicates the spatial guidance characteristics of the intermediate stage. This represents the mid-layer fusion parameters in the intermediate stage. This represents the deep fusion parameters in the intermediate stage. Represents deep space characterization. This indicates the target resolution for the intermediate stage. This indicates scaling the resolution of the deep spatial representation to the target resolution of the intermediate stage; This represents the spatial guidance features of the decoding stage. These represent the shallow fusion parameters during the decoding phase. This represents the mid-layer fusion parameters during the decoding stage. Indicates the target resolution during the decoding stage. This indicates that the resolution of the mid-level spatial representation is scaled to the target resolution of the decoding stage. This represents the deep fusion parameters during the decoding stage. This indicates that the resolution of the deep spatial representation is scaled down to the target resolution of the decoding stage.
[0046] The network structure of the Spatial Guided Feature Generation Module (SGFM) is as follows: Figure 4 As shown, the group-level space is represented S sh , S mid and S de Input space guided feature generation module SGFM, set stage Group-level spatial representation The unnormalized learnable fusion parameters are During this phase, the normalized exponential function is executed within the source groups that are allowed to be used: ;(twenty three) in, for , or , , , . r For the coding stagee hour, l Include S sh and S mid These two features correspond to two learnable weights. and And after Softmax normalization among the source features allowed to be used in this stage, the result is obtained. and ,Will and Multiply by the corresponding features respectively. u and β The relationship is: u These are the original, learnable weights. β Each stage r The feature source used internally corresponds to u The weights after normalization.
[0047] The spatial guidance features of the encoding stage, intermediate stage, and decoding stage are calculated using formulas (20)-(22). The target resolution of the encoding and decoding stages is taken from the resolution of the shallow spatial representation, while the target resolution of the intermediate stage is taken from the resolution of the mid-level spatial representation. Upon entering the specific cross-attention layer, the spatial guidance features are then... Bilinear scaling is applied to the actual height and width of the query feature at that layer.
[0048] Example 4 This embodiment, based on embodiments 1-3, further describes spatially perceived cross-attention. The module diagram of spatially perceived cross-attention is as follows: Figure 5 As shown, the input to the spatial awareness cross-attention module is the output of the dynamic cues and spatial guidance module DPSG.
[0049] The step of generating a complete attention map based on all the prompt words includes: S62: Perform linear projection on all the prompt words to obtain the complete key vector; S63: Introduce grouped temporal gating into the attention mechanism, and use the following formulas to calculate the attention scores for fixed text key vectors, non-spatial key vectors, and spatial key vectors respectively: ;(twenty four) (25) (26) in, The attention score represents the fixed text key vector. The attention score represents the non-spatial key vector. The attention score represents the spatial key vector; Represents the query vector. K t Represents a fixed text key vector. This represents the matrix transpose operation. This represents the channel dimension of the query vector; log represents the logarithm function with base 1000. Representation phase r Time-based gating of fixed prompt words t Indicates a time step; K reg Represents a non-spatial key vector. Representation phase r Temporal gating of non-spatial-aware dynamic prompts; K spa Represents a spatial key vector. Representation phase r Temporal gating of spatially-aware dynamic prompts; S64: Concatenate the attention scores of the fixed text key vector, the non-spatial key vector, and the spatial key vector to obtain the concatenated attention score. S65: Perform non-linear activation on the concatenated attention score along the token dimension to obtain the complete attention map.
[0050] Before step S62, step S61 is included: concatenating the dynamic prompt words and fixed prompt words along the token dimension to obtain all prompt words. The token in this invention represents the prompt words. For example, 77 fixed prompt words are 77 1024-dimensional vectors, and 16 dynamic prompt words are 16 1024-dimensional vectors. Concatenating the two types of vectors along the token dimension yields 93 1024-dimensional vectors. All prompt words are then projected uniformly to generate a complete key vector. K r The complete key vector is divided into fixed text key vectors. K t Non-spatial key vectors K reg and spatial key vector K spa .exist , K r , K t , K reg and K spa Based on this, corresponding time gating is added to calculate the attention scores of the three key vectors.
[0051] (27) in, Indicates all prompt words, This indicates concatenation along the token dimension. Indicates a fixed text condition. Indicates a dynamic prompt word.
[0052] Dynamic prompts are used to generate all prompts. Only when grouping time gating is applied, the key vectors corresponding to the dynamic prompts are divided into the first half of the non-spatial key vectors and the second half of the spatial key vectors according to the slot order, and corresponding gating values are added to different types of prompts.
[0053] (28) (29) in, Indicates a dynamic prompt word. Represents a non-spatial key vector. Represents a spatial key vector; Indicates non-spatial-aware dynamic prompt words. Indicates spatial awareness dynamic prompts.
[0054] All dynamic prompts are responsible for generating the complete key vector and determining attention location allocation. Before calculating the attention map, they are first determined based on the current diffusion time step. t Calculate the reverse diffusion progress: (30) (31) in, Indicates time step t The intermediate quantity, T This represents the total time steps, and clip represents the numerical truncation operation. Indicates the progress of back diffusion. The function of clip is to... t Restricted to [0,1], then subtract 1 / 2. Therefore The final value range of q is [-0.5, 0.5]. Under normal circumstances, t∈{0, ..., T-1}, and thus q is calculated. t Since it is already within [0,1], clip is used to prevent out-of-bounds errors or numerical errors.
[0055] stage r and prompt phrases Unified definition of time gating: (32) in, The stage is indicated as r And the prompt phrase isu Time gating, This represents the first learnable parameter for time-gated control. This represents the second learnable parameter for time gating; the initial values of the two parameters are different. Indicates the gating amplitude, preferred The value is 0.25. The time-gated value is close to 1, applied to the first half of the non-spatial key vector and the second half of the spatial key vector, respectively. For the fixed cue word "fix", no gating correction is applied, and the time-gated value is 1. "reg" corresponds to non-spatial-aware dynamic cue words, and "spa" corresponds to spatial-aware dynamic cue words.
[0056] For calculating the attention scores of different key vectors, temporal gating is performed additively. The transpose of the key vector of each cue word token is used to calculate the dot product with the query vector Q at H×W spatial locations, resulting in a spatial response map containing H×W logits. The gating is a scalar at the cue word group level, and all tokens within the same group share the same gating. Therefore, the logarithm of the regular dynamic gating is added to all H×W logits corresponding to all non-spatial-aware dynamic cue words, that is, added to all elements corresponding to non-spatial-aware dynamic cue words in the dot product matrix of the query vector and the key vector. The spatial dynamic gating method is the same and will not be elaborated here. The logit refers to the raw spatial classification score before it is converted into a probability. The logit is used to characterize the probability score of the model predicting the class appearing in each local region of the image.
[0057] Then, the three sets of logits corresponding to the fixed cue words, non-spatial-aware dynamic cue words, and spatial-aware dynamic cue words are concatenated along the token dimension, and Softmax is performed along the token dimension at each spatial location to obtain the complete attention graph: (33) in, Represents the complete attention map. This represents the normalized exponential function along the token dimension. This indicates that the three key vectors are concatenated in the order of fixed text key vector, non-space key vector, and space key vector. Each token corresponds to a complete attention map. A spatial attention map of size H×W is generated. Gating affects the attention map of the entire token, thereby adjusting the proportion of each cue word in the attention layer, rather than the passage of information.
[0058] Extract the attention subgraphs corresponding to all dynamic cue words from the complete attention graph: (34) in, This represents the attention subgraph corresponding to all dynamic prompts. This indicates the attention subgraph cropping operation. This represents a dynamic attention subgraph extracted from the complete attention graph, containing all dynamic cue words. The size is .
[0059] All three time-gated values are positive, so the logarithm of the time gating can be used as the attention score bias. In implementation, a very small positive number is used to avoid numerical anomalies. Only after the complete attention map calculation is completed is the dynamic attention subgraph divided into a first half (non-spatial dynamic attention subgraph) and a second half (spatial dynamic attention subgraph) according to the cue word slots. (35) in, Represents a dynamic attention subgraph. This represents a non-spatial dynamic attention subgraph. Represents the spatial dynamic attention subgraph. This indicates that the first half of the non-spatial dynamic attention subgraph and the second half of the spatial dynamic attention subgraph are concatenated.
[0060] Complete attention map satisfies A complete dynamic attention subgraph satisfies The key vector branch always uses all dynamic cue words, and spatial guidance only enters the value vector branch after the complete attention map is formed.
[0061] Extracting the latter half of the space from the dynamic attention subgraph and reading the attention responses corresponding to the slots yields the spatial dynamic attention subgraph. .Will Transpose the data so that each spatial read slot corresponds to one spatial response. .
[0062] Spatial normalization is performed on each spatial read slot along all query spatial positions: (36) in, n Represents a space token. , j Indicates the first j A spatial location, , Represents the total number of spatial locations. Indicates the first A spatial location. The first element in the normalized spatial dynamic attention subgraph represents the... n The first token j An element in a spatial location, It is a positive number used to prevent the denominator from being zero. In forming Stop calculating gradients when the attention response is detached, so that the spatial value generation branch does not reverse the attention allocation used for semantic matching.
[0063] spatial guidance features S r Zoom to Flatten and normalize the layers: (37) in, Representation phase r The spatial guidance features after layer normalization. LN represents layer normalization. This indicates that the size is The feature is flattened to a length of For a one-dimensional vector, Resize represents a scaling operation. Indicates the height of the spatially guided target. This indicates the width of the spatially guided feature target.
[0064] Weighted reads of the layer-normalized spatial guided features are performed using a normalized spatial dynamic attention subgraph: (38) in, This represents the normalized spatial dynamic attention subgraph. express r Spatial slot aggregation characteristics of the stage. .
[0065] By linearly projecting the spatial slot aggregation features onto the context dimension, we obtain spatially-aware dynamic cue words: (39) in, express r Stage-based spatial perception dynamic prompts, Used only for value vector branches. Indicates to r The spatial slot aggregation features of the stage are projected.
[0066] Projecting fixed prompts, non-spatial-aware dynamic prompts, and spatial-aware dynamic prompts respectively generates three types of value vectors: (40) in, This represents a fixed text value vector obtained by projecting fixed prompt words. This represents the non-spatial value vector obtained by projecting non-spatial-aware dynamic prompt words. This represents the spatial value vector obtained by projecting the spatially perceived dynamic prompt words. Indicates non-spatial-aware dynamic prompt words. This indicates that the projection is a value vector operation. and All include A dynamic prompt word.
[0067] The three projected value vectors are concatenated along the token dimension to obtain the complete value vector: (41) in, Represents a complete value vector. This indicates concatenation along the token dimension. Finally, the product of the complete attention map and the complete value vector is calculated, and the cross-attention output is obtained through output projection. (42) in, This indicates the output of cross-attention. This indicates a linear projection of the output layer.
[0068] In this invention, all dynamic prompts in the spatially aware cross-attention module uniformly determine the attention position, and the query vector-key vector attention remains unchanged due to spatial prompt generation. The value vector branch retains the first half of the dynamic prompts and generates spatially aware dynamic prompts from the spatial guidance features using the spatial dynamic attention subgraph corresponding to the second half of the dynamic prompts. The fixed text value vector, non-spatial value vector, and spatial value vector are concatenated along the token dimension to obtain the complete value vector, ensuring that the cross-attention output simultaneously includes global degradation information and spatial details related to the input.
[0069] In addition to dynamic prompts, the 13 residual features output by IRControlNet are also injected into U-Net as control features, with a scaling factor of 1. Then the following formula holds true: (43) in, Indicates the first k One control residual, in this embodiment =1. In the middle block of U-Net, the last control residual, i.e. the 13th control residual, is added to the intermediate feature. In the decoding stage, the remaining control residuals are added to the corresponding skip connections in reverse order, and then concatenated with the current decoding feature, thus forming two complementary paths: (1) Residual control path, which directly provides the input structure and degradation conditions to the spatial features of U-Net. (2) Cue modulation path: Change the condition reading method of the pre-trained diffusion prior in cross attention. It plays the following role: injecting the spatial structure and degradation information in the input night fog image into the frozen U-Net. The control residuals in the middle stage mainly constrain the global layout and overall structure (because the resolution is the lowest and the information is the most global); the multi-scale control residuals in the decoding stage supplement the contour, edge, texture and color information step by step through the corresponding skip connections, so that the defogging result is consistent with the input scene and reduces the generation of irrelevant content or structural shift by the diffusion model. λ k The residual control path is used to adjust the strength of the k-th control residual. In this embodiment, it is set to 1, indicating that the control information is used completely. Compared with the cue modulation path, the residual control path provides direct spatial structure constraints, while the cue modulation path mainly adjusts the condition reading method in cross-attention. Together, they achieve structure preservation and degradation removal.
[0070] IRControlNet generates 13 residual features. The last residual feature is generated by an intermediate layer, so it is directly added to the intermediate features. The remaining 12 layers are generated by the encoder and correspond one-to-one with the decoder. For example, the first layer of the encoder corresponds to the last layer of the decoder, and the last layer of the encoder corresponds to the first layer of the decoder. Therefore, during the decoding stage, the 12 residual features generated by IRControlNet, excluding the last residual feature, are injected into the corresponding modules of the decoding stage in reverse order.
[0071] Non-spatial-aware dynamic cues characterize the degradation effects on the entire image, while spatial-aware dynamic cues characterize the regions requiring restoration and specific methods. Combining these two approaches allows for the simultaneous handling of global haze, overall low illumination, color shifts, as well as localized halos, localized reflections, non-uniform fog layers, and detail loss. A cross-attention layer weights and aggregates the three types of value vectors, using the result to modulate the features of the U-Net's encoder, intermediate layers, and decoder. This ensures that the restored image does not deviate from the clear image semantics of the pre-trained diffusion model while adapting to the specific degradation type and spatial distribution of the current nighttime foggy image.
[0072] The purpose of grouped temporal gating is to adjust the proportion of non-spatial-aware dynamic cues and spatial-aware dynamic cues in cross-attention based on the current diffusion time step. Grouped temporal gating does not directly fuse spatial and temporal features, nor does it accelerate inference. Its function is to dynamically change the influence strength of the two types of cue conditions at different denoising stages, enabling the model to coordinate global degradation recovery and local detail recovery according to the current denoising progress, thereby improving the accuracy and stability of condition guidance.
[0073] Example 5 This embodiment, based on embodiments 1-4, further describes the training process of the overall dehazing architecture. Nighttime foggy images with the same name are read from the paired dataset. I Clear images during the day J The two images are stitched together along the channel dimension and then randomly cropped at the same position to ensure pixel registration. If the shorter side of the original image is smaller than the cropping size, it is first enlarged proportionally. In this embodiment, the cropping size is 512×512.
[0074] Images of foggy nighttime conditions are processed with a preset probability. I JPEG compression degradation is applied, with the compression quality randomly selected between 30 and 95. The preset probability in this embodiment is 0.5, resulting in a clear image during the day. J Without applying this degradation, I Normalization to ,Will J Normalization to .
[0075] During the training of the overall dehazing architecture, the following modules are frozen: VAE encoder. VAE decoder Text encoder and Stable Diffusion U-Net noise denoiser Train the following modules: IRControlNet and DPSG, including channel projection, intra-group weights, temporal MLP, stage-specific fusion layer, cue header, slot embedding, temporal gating, spatial fusion weights, spatial normalization layer, and spatial value projection layer, to update the parameters of IRControlNet. and DPSG parameters .
[0076] The training process employs a two-stage optimization approach. The first stage involves warming up IRControlNet by disabling DPSG and prohibiting gradient calculation for its parameters. The conditions include only... and This stage enables IRControlNet to first learn to stably extract degenerate residuals associated with clear targets, with a preferred warm-up of 5000 steps. The second stage activates DPSG, where IRControlNet and DPSG jointly form the complete condition: , , .
[0077] The warm-up learning rate, joint phase learning rate, and DPSG learning rate of IRControlNet are set separately. In this embodiment, they are respectively adopted. , and The optimizer is AdamW, with two momentum parameters of 0.9 and 0.999, and a weight decay of 0.01.
[0078] Model prediction noise ,in Add noise to time step Latent variables of time, Provide all necessary control conditions. Construct a denoising loss function using mean square error: (44) in, Represents the denoising loss function. Representing the actual noise, formula (44) is used to measure the difference between the actual noise and the predicted noise. During backpropagation, the warm-up phase only updates... Joint Phase Update and .
[0079] From training T Uniform selection in each time step N A descending sequence is formed at intervals of time steps. Preferred T =1000、 N =50. For Perform the following steps in sequence: (1) By Calculate the 13 IRControlNet residuals; (2) DPSG for residual projection, grouping, and aggregation; (3) Calculate the three-stage dynamic prompts, spatial guidance features, and temporal gating; (4) Input the control residual and prompt status into the frozen U-Net for back diffusion; (5) Obtain the predicted noise ; (6) Calculate the sampler based on the sampler Estimate: ; (7) According to the posterior timeline The mean and variance of the sampled latent variable are used to sample the next latent variable; no random noise is added at the termination step.
[0080] Decoding and size recovery include decoding the final latent variables after completing all back-diffusion: (45) in, Indicates VAE decoder, Representing clear latent variables, This represents the clear image after decoding. Crop the padded area and resize the image back to its original size. And cut to Obtain the daytime image after dehazing. .
[0081] Example 6 This embodiment, based on Embodiment 1, further describes the method for synthesizing nighttime foggy images. For example... Figure 6 As shown, all the nighttime foggy images synthesized in this embodiment constitute the RINH nighttime fog and haze dataset. The synthesis method of the RINH nighttime fog and haze dataset takes clear daytime images, semantically labeled maps, and original depth maps as inputs, and sequentially performs regional low-light transformation, depth-related fog scattering, semantically perceptive artificial light source placement, direct illumination attenuation, surface reflected illumination, glow, and sensor noise simulation. The final output is a synthesized nighttime foggy image that is strictly paired with the original clear daytime image. This synthesis process, by simultaneously constraining the light source position, propagation direction, distance attenuation, and surface depth consistency, makes the synthesized degradation more closely resemble the real road and urban environment at night.
[0082] The process of combining clear daytime images and the original depth map to synthesize foggy nighttime images includes: S11: Obtain a clear daytime image, as well as a semantic label map and an original depth map corresponding to the clear daytime image; S12: Extract the sky mask based on the semantic tag map, reduce the brightness of the sky region corresponding to the sky mask, and use power exponent to reduce the brightness of the sky region and non-sky region to obtain a low-light image; S13: Generate a random number from a uniform distribution, and generate a green channel and a blue channel based on the random number; S14: Set the red channel to 1, combine the red, green and blue channels, and multiply by the ambient intensity factor to obtain the final atmospheric light; S15: Calculate the fog scattering depth map from the original depth map, calculate the transmittance map from the fog scattering depth map, and generate a haze image by combining the transmittance map, the final atmospheric light and low-light image. S16: Generate an artificial lighting map based on the original depth map, and combine the haze image, the artificial lighting map, and surface reflected light to generate a preliminary composite image; S17: Add glow and image noise to the preliminary synthesized image to obtain a nighttime foggy image.
[0083] For each clear daytime image Prepare semantic label graphs of the same spatial size. Semantic label maps can be directly provided by datasets such as Cityscapes and Mapillary; if labels are missing, they are generated using a semantic segmentation model. The original depth map is estimated using Depth Anything V2. ,right Perform min-max normalization to obtain the normalized depth map. D ( x ). Normalized depth map D ( x Cut to Since the DepthAnything output is used in this invention based on relative parallax, the fog scattering depth map is defined as follows: .
[0084] In this embodiment, the method of regional low-light transformation is based on semantic tag maps. Extract the sky mask using the semantic label graph. Y Distinguish between sky areas and non-sky areas to form a sky mask. : (46) First, reduce the brightness of the sky area: (47) in, This indicates a clear image after the sky brightness decreases. This represents the brightness adjustment factor, which is greater than 1 in this embodiment. .
[0085] Then use the regional correlation power index: (48) in, Represents a low-light image. Indicates the first power exponent. Indicates the second power exponent. To represent the third power exponent, in this embodiment, we use... , , For example.
[0086] In this embodiment, the random sampling of atmospheric light color uses color change modeling and uniform sampling. , For a random number, k =3.608, generating a green channel: (49) The blue channel represents: (50) Here, clip represents the numerical truncation operation. g Indicates a green channel. b This indicates the blue channel.
[0087] Resampling environmental intensity factor , The value of the red channel is 1. Represents colored atmospheric light. This represents the final atmospheric light. Indicates the environmental intensity factor. express A uniform distribution is used, and environmental intensity factors are randomly generated from this uniform distribution. Preferred... , .
[0088] Sampling coefficients are sampled from a preset interval: (51) (52) in, Represents the scattering coefficient. This represents the minimum value of the scattering coefficient. This represents the maximum value of the scattering coefficient. It represents a uniform distribution, from which a scattering coefficient is randomly generated. This represents a transmittance diagram. This represents an exponential function with the natural constant as the base, and * indicates multiplication. This represents a depth map showing fog scattering.
[0089] Haze images were calculated using an atmospheric scattering model: (53) in, Represents images of smog. This indicates element-wise multiplication. The scattering range is a configurable parameter used to control haze density; different data sources use different scattering ranges.
[0090] From semantic tags The connected regions of streetlights, lamp posts, and optional traffic lights are detected to form an artificial light source set. (1) If there is a streetlight category, the center of the lamp's connected region is used as the light source anchor point, and the light source depth is obtained from the depth statistics of the lamp region and its neighborhood. (2) If there is no lamp category but there is a lamp post category, the top position of each lamp post's connected region is extracted as a cone-shaped light source anchor point, and the direction of the upper part of the lamp post or the random sampling direction is recorded as the first. The main illumination direction of the cone-shaped light source (3) Perform non-maximum suppression on candidate light sources that are too close and have similar depths to avoid duplicate light sources. (4) If there is no semantic light source anchor point, randomly sample the light source position and type. It is preferable to generate point light sources with a probability of 0.7, otherwise generate cone light sources; the cone angle can be randomly sampled between 30 degrees and 60 degrees. (5) Add a small channel-by-channel perturbation to the color of each light source, and also sample neon light colors from the set of blue, green, purple, cyan or red with a small probability to improve lighting diversity. The perturbation is added in the following ways: , This indicates small perturbations on a channel-by-channel basis. This is the atmospheric light after cropping.
[0091] For pixels and the Each light source center Define two-dimensional distance: (54) in, Indicates the first i A two-dimensional distance.
[0092] For point light sources: (55) For a cone-shaped light source, define a unit vector pointing from the light source to the pixel: (56) in, Represents a unit vector. It is a positive number used to prevent the denominator from being 0.
[0093] Consistency of direction for: (57) make For the half-angle of the light cone, use a smoothing function. Soften the boundary: (58) Where max is the function for finding the maximum value. The illumination direction mask is used to perform three smoothing operations to form continuous edges and reduce hard boundary artifacts.
[0094] The light source attenuation weight is calculated using the following formula: (59) in, Indicates the light source attenuation weight. This represents the overall lighting intensity scaling factor, used to control the overall energy of a single light source. It is the relative depth index of the light source, used to adjust the intensity of direct illumination according to the depth of the light source. The constant coefficient is used to limit the maximum intensity at the center of the light source and to prevent the denominator from approaching zero. This is the coefficient for the first-order distance attenuation term, used to control the first-order attenuation rate of illumination intensity as the distance from the pixel to the light source increases. These are the coefficients for the second-order distance attenuation term, used to control the second-order attenuation rate in areas far from the light source. D ( center i ) is the th in the normalized depth map i The light source center is determined by the light source attenuation weight, which is calculated from the original depth map. The light source attenuation weight is related to the location of the streetlights and the sky in the semantic label map.
[0095] The formula for calculating artificial lighting diagrams is: (60) in, This represents an artificial lighting diagram, which is... Figure 6 The artificial light sources in the image are scaled according to their relative depth, and the illumination results of all light sources are summed and then cropped to the desired value. .
[0096] In real-world nighttime roads, lights illuminate roads, buildings, vehicles, and other surfaces. Simply superimposing direct atmospheric illumination cannot simulate this localized reflection. Therefore, a depth consistency factor is defined: (61) in, Indicates the depth of consistency factor. This is a depth representation of the surface matching construction.
[0097] The surface reflection weight is: (62) Considering the further attenuation of reflected light propagation by fog, the surface reflected light is: (63) in, Indicates surface reflected light. .
[0098] The haze image, the artificial lighting image, and the surface reflected light are added together to obtain a preliminary composite image. Glow and image noise are then added to this preliminary composite image to obtain a nighttime foggy image. This nighttime foggy image is the... Figure 6 The final synthesized image is formed by combining multiple nighttime foggy images synthesized in the above steps to create the RINH nighttime fog and haze dataset. The image noise follows a Gaussian distribution, and the synthesized nighttime foggy images and clear daytime images are saved as image pairs with the same name. The training process of this invention is based on nighttime foggy images and paired clear daytime images. Real-world datasets only contain images with fog and haze, which are not paired. The RINH nighttime fog and haze dataset constructed in this invention uses paired images. Real-world datasets lack strictly paired clear ground truth images, making them difficult to directly use for pixel-level supervised training. After image pairing, there is a better reference and measurement standard.
[0099] Example 7 This embodiment uses the real nighttime haze datasets RealNightHaze and NHRW, as well as the synthesized RINH nighttime haze dataset, to verify the image dehazing performance of the image synthesis-based nighttime fog image dehazing methods in Examples 1-6.
[0100] RealNightHaze is a real-world dataset of foggy nighttime images, but it lacks sharp ground truth images that are strictly paired with the foggy nighttime images. It is primarily suitable for testing purposes and difficult to use directly for supervised training. The RINH dataset generates foggy nighttime images through processes such as fog scattering, semantically aware artificial light sources, surface reflection, adding glow and noise, and retains the corresponding sharp daytime images as ground truth pairs. Its advantage is that it can provide a large-scale, pixel-level paired training sample set with controllable degradation, thus supporting supervised training of the model and covering different fog concentrations, light source colors, and non-uniform lighting conditions. Because it is difficult to collect paired sharp and foggy nighttime images in reality, RINH is mainly used to supplement the lack of paired ground truth images in real-world datasets, rather than replacing RealNightHaze.
[0101] Figure 7 This is a comparison of the image dehazing method provided by this invention with other image dehazing methods. Figure 8 These are paired daytime clear images and nighttime foggy images from the RINH nighttime haze dataset. Figure 7 It is evident that the method provided by this invention, compared to other methods, can restore nighttime foggy images to images with natural brightness, true colors, and clear structural details. Figure 8 It can be seen that the clear daytime images and foggy nighttime images in the RINH nighttime haze dataset synthesized by this invention are paired one-to-one, and both belong to the same scene.
[0102] This embodiment also provides an electronic device, including: A memory on which computer programs are stored; A processor is configured to execute the computer program in the memory to implement the image synthesis-based dehazing method for nighttime foggy images in Examples 1-6.
[0103] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0104] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for dehazing nighttime foggy images based on image synthesis, characterized in that, include: Combine clear daytime images with the original depth map to create a foggy nighttime image; The nighttime foggy image is mapped and input into a frozen VAE encoder for image encoding to obtain VAE conditional latent variables; The preset text is input into the frozen text encoder for text encoding to obtain fixed text conditions; Input the VAE conditional latent variables, fixed text conditions, noisy latent variables, and time steps into IRControlNet to generate M residual features; divide the M residual features into shallow feature groups, medium feature groups, and deep feature groups; Dynamic prompts and spatial guidance are generated based on shallow feature groups, mid-level feature groups and deep feature groups to produce dynamic prompt words and spatial guidance features. The dynamic and fixed prompts are concatenated to obtain all prompts; a complete attention map is generated based on all prompts. Extract the spatial dynamic attention subgraph corresponding to the second half of the dynamic prompt words from the complete attention map, and use the spatial dynamic attention subgraph to generate spatially perceptual dynamic prompt words from the spatial guidance features; Concatenate the fixed text value vector corresponding to the fixed prompt word, the non-spatial value vector corresponding to the first half of the dynamic prompt words, and the spatial value vector corresponding to the spatially aware dynamic prompt words along the token dimension to obtain the complete value vector. Multiply the complete value vector by the complete attention map and project it to obtain the cross-attention output; The cross-attention output is input into the frozen U-Net and backdiffusion is performed to obtain the dehazed daytime image.
2. The method for dehazing nighttime foggy images based on image synthesis according to claim 1, characterized in that, The method of generating dynamic prompts and spatial guidance based on shallow, medium, and deep feature groups, including: Obtain a set of learnable weights, and normalize the set of learnable weights to obtain a normalized set of weights. Global average pooling is performed on each aligned residual feature in the shallow feature group, the middle feature group and the deep feature group to obtain M spatial dimensional pooling vectors. Using the normalized weight set, within the same feature group, the pooled vectors of each spatial dimension in the shallow feature group, the middle feature group, and the deep feature group are weighted and summed to obtain the group-level global representation; the group-level global representation includes the shallow global representation, the middle global representation, and the deep global representation. Bilinear interpolation is performed on each aligned residual feature in the shallow, middle and deep feature groups to obtain the interpolated residual features. The interpolated residual features are weighted and summed using the normalized weight set to obtain a group-level spatial representation; the group-level spatial representation includes shallow spatial representation, medium spatial representation and deep spatial representation. Input the group-level global representation into the dynamic prompt generation module to generate dynamic prompt words; The group-level spatial representation is input into the spatial guided feature generation module to generate spatial guided features.
3. The method for dehazing nighttime foggy images based on image synthesis according to claim 2, characterized in that, The step of inputting group-level global representations into the dynamic prompt generation module to generate dynamic prompt words includes: The time step is subjected to sinusoidal position encoding, linear transformation and SiLU activation to obtain the time step embedding; The shallow global representation, the middle global representation, and the time step embedding are concatenated along the feature dimension, and then linear projection, encoding stage embedding, and layer normalization are performed to obtain the encoding stage features. The intermediate-level global representation, deep-level global representation, and time step embedding are concatenated along the feature dimension, and then linear projection, intermediate stage embedding, and layer normalization are performed to obtain the intermediate stage features. The shallow global representation, the middle global representation, the deep global representation, and the time step embedding are concatenated along the feature dimension, and then linear projection, decoding stage embedding, and layer normalization are performed to obtain the decoding stage features. Linear projections are performed on the features of the encoding stage, the intermediate stage, and the decoding stage, and the corresponding stage biases are added to obtain dynamic prompt words for the encoding stage, the intermediate stage, and the decoding stage.
4. The method for dehazing nighttime foggy images based on image synthesis according to claim 2, characterized in that, The module that inputs group-level spatial representations into spatial guiding feature generation to generate spatial guiding features includes: The following formulas are used to calculate the spatial guidance features of the encoding stage, the intermediate stage, and the decoding stage: ; ; ; in, This represents the spatial guidance features of the encoding stage. These represent the shallow fusion parameters during the encoding phase. This represents a shallow spatial characterization. This represents the mid-layer fusion parameters during the encoding phase. This represents the mid-level spatial characterization. Indicates the target resolution during the encoding stage. This indicates that the resolution of the mid-level spatial representation is scaled down to the target resolution during the coding stage; This indicates the spatial guidance characteristics of the intermediate stage. This represents the mid-layer fusion parameters in the intermediate stage. This represents the deep fusion parameters in the intermediate stage. Represents deep space characterization. This indicates the target resolution for the intermediate stage. This indicates scaling the resolution of the deep spatial representation to the target resolution of the intermediate stage; This represents the spatial guidance features of the decoding stage. These represent the shallow fusion parameters during the decoding phase. This represents the mid-layer fusion parameters during the decoding stage. Indicates the target resolution during the decoding stage. This indicates that the resolution of the mid-level spatial representation is scaled to the target resolution of the decoding stage. This represents the deep fusion parameters during the decoding stage. This indicates that the resolution of the deep spatial representation is scaled down to the target resolution of the decoding stage.
5. The method for dehazing nighttime foggy images based on image synthesis according to claim 1, characterized in that, The step of generating a complete attention map based on all the prompt words includes: By performing a linear projection on all the prompt words, a complete key vector is obtained; Grouping time gating is introduced into the attention mechanism, and the attention scores for fixed text key vectors, non-spatial key vectors, and spatial key vectors are calculated using the following formulas: ; ; ; in, The attention score represents the fixed text key vector. The attention score represents the non-spatial key vector. The attention score represents the spatial key vector; Represents the query vector. K t Represents a fixed text key vector. This represents the matrix transpose operation. This represents the channel dimension of the query vector; log represents the logarithm function with base 1000. Representation phase r Time-based gating of fixed prompt words t Indicates a time step; K reg Represents a non-spatial key vector. Representation phase r Temporal gating of non-spatial-aware dynamic prompts; K spa Represents a spatial key vector. Representation phase r Temporal gating of spatially perceived dynamic prompts; The attention scores of the fixed text key vector, the non-spatial key vector, and the spatial key vector are concatenated to obtain the concatenated attention score. The concatenated attention score is non-linearly activated along the token dimension to obtain the complete attention map.
6. The method for dehazing nighttime foggy images based on image synthesis according to claim 1, characterized in that, The step of generating spatially aware dynamic cue words from the spatial guidance features using the spatial dynamic attention subgraph includes: The spatial guidance features are scaled, flattened, and layer-normalized to obtain the layer-normalized spatial guidance features. Spatial normalization is performed on the spatial dynamic attention subgraph to obtain the normalized spatial dynamic attention subgraph; Multiply the normalized spatial guidance feature and the normalized spatial dynamic attention subgraph to obtain the spatial slot aggregation feature; Linear projection is performed on the spatial slot aggregation features to obtain spatially perceptive dynamic prompts.
7. The method for dehazing nighttime foggy images based on image synthesis according to claim 1, characterized in that, The process of combining clear daytime images and the original depth map to synthesize foggy nighttime images includes: Acquire a clear daytime image, as well as a semantic label map and a raw depth map corresponding to the clear daytime image; Extract the sky mask from the semantic tag map, reduce the brightness of the sky region corresponding to the sky mask, and use a power-law to down-adjust the brightness of the sky region and non-sky region to obtain a low-light image. A random number is generated from a uniform distribution, and a green channel and a blue channel are generated based on the random number; Set the red channel to 1, combine the red, green and blue channels, and multiply by the ambient intensity factor to obtain the final atmospheric light. A fog scattering depth map is calculated from the original depth map, and a transmittance map is calculated from the fog scattering depth map. A fog and haze image is generated by combining the transmittance map, the final atmospheric light, and the low-light image. An artificial lighting map is generated based on the original depth map, and a preliminary composite image is generated by combining the haze image, the artificial lighting map, and surface reflected light. Add glow and image noise to the preliminary synthesized image to obtain a nighttime foggy image.
8. The method for dehazing nighttime foggy images based on image synthesis according to claim 1, characterized in that, The process of mapping the nighttime foggy image and inputting it into a frozen VAE encoder for image encoding yields VAE conditional latent variables, including: The nighttime foggy image is preprocessed to obtain a preprocessed image. The preprocessing includes scaling and size padding. The pixel values of the pixels in the preprocessed image are greater than or equal to 0 and less than or equal to 1. Multiply the preprocessed image by 2 and then subtract 1 to obtain the mapped image; The mapped image is input into the frozen VAE encoder for image encoding to obtain VAE conditional latent variables.
9. The method for dehazing nighttime foggy images based on image synthesis according to claim 1, characterized in that, The step of dividing the M residual features into shallow feature groups, middle feature groups, and deep feature groups includes: M 1×1 convolutions are used to align the M residual features to obtain M aligned residual features; where M=13. Among the M aligned residual features, the first to the sixth aligned residual features from shallowest to deepest are taken as the shallow feature group; Among the M aligned residual features, the 7th to 9th aligned residual features from shallow to deep are taken as the middle layer feature group; Among the M aligned residual features, the 10th to 13th aligned residual features from shallow to deep are taken as the deep feature group.
10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1 to 9.
