Event feature combined image defogging method, device, equipment, medium and product

Abstract

The invention discloses an event feature combined image defogging method, device and equipment, a medium and a product, and the method comprises the steps: inputting an obtained foggy image into a preset encoder for encoding, and extracting potential feature representation; adding noise to the potential feature representation to obtain a noise representation; acquiring event features of event streams synchronously acquired by the foggy image; fusing the event features into a preset image segmentation network, and performing diffusion reverse denoising iteration on the noise representation to obtain a denoising result; and inputting the de-noising result into a preset decoder for decoding to obtain a defogged image. According to the scheme, the limitation of dynamic range compression in the defogging process can be reduced, and the image defogging effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image defogging method and device combining event features, equipment, medium and product. BACKGROUND

[0002] Haze weather seriously reduces the contrast and clarity of images through atmospheric scattering and absorption, especially at long distances and under strong light interference. Image defogging is a key computer vision task and is widely used in photography enhancement, unmanned aerial vehicle monitoring, intelligent transportation and autonomous driving scenarios.

[0003] Diffusion models are applied to image restoration fields such as super-resolution, denoising and image generation due to their strong generation capabilities. However, these methods still rely on RGB images as the only input, and RGB image sensors often cannot capture the structural details obscured by scattered light due to their limited dynamic range, resulting in irreversible loss of image information. Therefore, the existing image defogging methods relying on RGB images are naturally limited by dynamic range compression, especially under heavy haze conditions. SUMMARY

[0004] To solve the above problems, the present application provides an image defogging method and device combining event features, equipment, medium and product, which can reduce the limitation of the defogging process by dynamic range compression and improve the image defogging effect.

[0005] The present application provides an image defogging method combining event features, which comprises:

[0006] The obtained foggy image is input into a preset encoder for encoding to extract a latent feature representation;

[0007] Noise is added to the latent feature representation to obtain a noise representation;

[0008] The event features of the event stream synchronously collected by the foggy image are obtained;

[0009] The event features are fused into a preset image segmentation network to perform diffusion reverse denoising iteration on the noise representation to obtain a denoising result;

[0010] The denoising result is input into a preset decoder for decoding to obtain a defogging image.

[0011] Preferably, the obtained foggy image is input into a preset encoder for encoding to extract a latent feature representation, which comprises:

[0012] A foggy image is obtained;

[0013] inputting the obtained foggy image into a preset latent space encoder to compress the foggy image from a pixel space into a latent space and extract the latent feature representation.

[0014] Preferably, the obtaining event features of the event stream synchronously acquired with the foggy image comprises:

[0015] synchronously acquiring the event stream of the foggy image through an event camera;

[0016] dividing the event stream into a three-layer time pyramid and dividing each layer into two time periods to obtain two event voxel slices respectively;

[0017] combining the obtained event voxel slices to obtain a time pyramid representation of the event stream;

[0018] inputting the time pyramid representation into a preset event encoder to extract structural edge features to obtain the event features.

[0019] Preferably, the fusing the event features into a preset image segmentation network and iteratively performing diffusion reverse denoising on the noise representation to obtain a denoised result comprises:

[0020] fusing the event features into an intermediate layer of the image segmentation network through an event guiding module;

[0021] performing diffusion reverse denoising on the noise representation for several times through the fused image segmentation network to obtain the denoised result.

[0022] Preferably, the fusing the event features into a preset image segmentation network and iteratively performing diffusion reverse denoising on the noise representation to obtain a denoised result comprises:

[0023] in each iteration, taking the event features as a query condition, taking the image features output by the previous step as a key and a value, generating attention weights through a cross-attention mechanism, denoising the image features output by the previous step to obtain a noise representation output by the current step;

[0024] taking the image features output by the last step as the denoised result;

[0025] wherein, the noise representation output by the initial previous step is the noise representation.

[0026] Preferably, the method further comprises:

[0027] inputting the obtained training image into a preset to-be-trained encoder to perform encoding and extract a training latent feature representation;

[0028] adding noise to the training latent feature representation to obtain a training noise representation;

[0029] obtaining training event features of an event stream synchronously collected with the training image;

[0030] fusing the training event features into a preset image segmentation network to be trained, performing iterative diffusion reverse denoising on the training noise representation, and obtaining a training denoising result;

[0031] inputting the training denoising result into a preset decoder to be trained for decoding, and obtaining a predicted defogging image;

[0032] determining a loss function between the real image corresponding to the training image and the predicted defogging image through a pixel-level L1 loss and a perception loss weighted joint;

[0033] According to the loss function, an optimizer is used to perform gradient back propagation, and the network parameters of the encoder to be trained, the image segmentation network to be trained and the decoder to be trained are updated to minimize the loss value as the target, and the encoder, the image segmentation network and the decoder are obtained.

[0034] The embodiment of the application also provides an image defogging device combined with event features, the device comprising:

[0035] An encoding module is configured to input the obtained foggy image into a preset encoder for encoding to extract a latent feature representation.

[0036] A noise module is configured to add noise to the latent feature representation to obtain a noise representation.

[0037] An event module is configured to obtain event features of an event stream synchronously collected with the foggy image.

[0038] A denoising module is configured to fuse the event features into a preset image segmentation network to be trained, perform iterative diffusion reverse denoising on the noise representation, and obtain a denoising result.

[0039] A decoding module is configured to input the denoising result into a preset decoder to be trained for decoding to obtain a defogging image.

[0040] The embodiment of the application also provides an image defogging device combined with event features, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements an image defogging method combined with event features according to any one of the above embodiments when executing the computer program.

[0041] The embodiment of the present application further provides a computer readable storage medium, which comprises a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the image defogging method combining event features as any one of the above embodiments when the computer program runs.

[0042] The embodiment of the present application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the image defogging method combining event features as any one of the above embodiments.

[0043] The present application provides an image defogging method combining event features, device, equipment, medium and product, by inputting the obtained foggy image into the preset encoder for coding, extracting the potential feature representation; adding noise on the potential feature representation to obtain the noise representation; obtaining the event features of the event stream collected synchronously with the foggy image; fusing the event features into the preset image segmentation network, performing diffusion reverse denoising iteration on the noise representation to obtain the denoising result; inputting the denoising result into the preset decoder for decoding to obtain the defogging image. The application scheme can reduce the limitation of dynamic range compression in the defogging process, and improve the image defogging effect. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of an image defogging method combining event features provided by the embodiment of the present application;

[0045] Figure 2 is a principle diagram of an image defogging method combining event features provided by the embodiment of the present application;

[0046] Figure 3 is a comparison diagram of image defogging effects provided by the embodiment of the present application;

[0047] Figure 4 is a principle diagram of an image defogging method combining event features provided by the embodiment of the present application;

[0048] Figure 5 is a defogging result diagram of an image defogging method combining event features provided by the embodiment of the present application;

[0049] Figure 6 is a structure diagram of an image defogging device combining event features provided by the embodiment of the present application;

[0050] Figure 7 is a structure diagram of a terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0051] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] Referring to Figure 1 is a flowchart of an image defogging method combining event features provided by the embodiments of the present application, and the method comprises steps S1-S5.

[0053] S1, input the obtained foggy image into a preset encoder to encode and extract a latent feature representation;

[0054] S2, add noise to the latent feature representation to obtain a noise representation;

[0055] S3, obtain event features of an event stream synchronously collected with the foggy image;

[0056] S4, fuse the event features into a preset image segmentation network, perform diffusion reverse denoising iteration on the noise representation, and obtain a denoising result;

[0057] S5, input the denoising result into a preset decoder to decode and obtain a defogged image.

[0058] In the specific implementation of the present embodiment, based on the reverse denoising mechanism of the diffusion model Diffusion Model and combined with the high-frequency dynamic information of the event stream Event Stream, the defogging processing of the foggy image is realized.

[0059] The implementation of the scheme can be divided into two stages of model training and inference application, and the specific steps are as follows:

[0060] Pre-model training stage: collect a training data set of pairs of foggy images and clear images, such as synthetic fog images or real foggy scenes;

[0061] Synchronously collect event stream data, which needs to use an event camera to record the time, position and polarity information of pixel brightness changes;

[0062] The Prophesee event camera and the RealSense RGB camera are mounted on a drone for long-time synchronous collection, so as to avoid the problem that a static collection system is difficult to simulate real motion blur and fog change.

[0063] It should be noted that the synchronization of the event camera and the RGB camera can also use a combination of depth cameras, infrared cameras, or multispectral cameras to expand the adaptability of the system in extreme environments (night, fire, smoke).

[0064] The data is preprocessed to ensure that the image and event stream are time-aligned.

[0065] A convolutional neural network is used to map the input image to a latent feature vector.

[0066] Through an event feature extractor, an example of a specialized network including 2D convolution and temporal convolution, spatial-temporal features are extracted from the event stream.

[0067] Based on the U-Net structure, the input noisy latent features and event features are fused through an attention mechanism to predict noise and update features in each denoising iteration.

[0068] Through the decoder of the convolutional network symmetrical to the encoder, the denoised latent features are decoded into images.

[0069] Combined with pixel-level loss such as MSE, SSIM, to measure the difference between the dehazed image and the clear image, and diffusion process loss such as L2 loss of predicted noise and real noise.

[0070] The encoder, event feature extractor, denoising module, and decoder are jointly trained, and the parameters are optimized through backpropagation.

[0071] In the actual application stage, specifically:

[0072] Get the foggy image to be dehazed and the synchronized event stream data, and standardize the input to be consistent with the training data format.

[0073] Input the foggy image into the encoder to obtain the latent feature representation, and add noise to the latent feature according to the preset noise distribution, such as Gaussian noise, to generate an initial noise representation.

[0074] Input the event stream into the feature extractor to obtain the event feature, and input the noise representation and event feature into the image segmentation network for multi-step diffusion and reverse denoising:

[0075] In each iteration, the network predicts the added noise based on the current noise representation and event feature.

[0076] Update the feature based on the predicted noise to gradually reduce the noise intensity.

[0077] Repeat the iteration until the preset number of steps is reached to obtain the denoised latent feature.

[0078] Input the denoising result into the decoder to generate the final dehazed image.

[0079] The embodiment compresses the foggy image into low-dimensional latent features through an encoder, retains key structural information of the image such as contours and textures, and reduces processing complexity. A noise-adding and noise-removing process simulating a diffusion model is used to add noise to the latent features and then gradually recover clear features through iterative noise removal. Through event stream feature fusion, the event stream records the mutation information of the pixel brightness during image acquisition, provides additional constraints for the iterative noise removal, and improves the de-fogging accuracy. The de-noised latent features are restored to clear images through a decoder, and an end-to-end de-fogging process is completed. The de-fogging method provided in the application has stronger robustness, reduces the limitation of the de-fogging process on dynamic range compression, and in complex scenes such as dynamic objects and non-uniform fog concentration, the timing information of the event stream can assist the model to adapt to the spatial variation of the fog, avoid over-de-fogging or insufficient de-fogging, and improve the image de-fogging effect.

[0080] In another embodiment provided by the application, the input of the obtained foggy image into the preset encoder for encoding to extract the latent feature representation comprises:

[0081] obtaining a foggy image;

[0082] inputting the obtained foggy image into a preset latent space encoder to compress the foggy image from a pixel space into a latent space and extract the latent feature representation.

[0083] In the specific implementation of the embodiment, the foggy image is high-dimensional data in the pixel space, for example, a 256x256 RGB image contains 256x256x3=196,608 pixel values, which contain both useful information such as object structure and texture and redundant information such as noise caused by fog and repeated low-frequency components.

[0084] The latent space encoder uses a convolutional neural network (CNN) structure and gradually compresses the dimension through multiple nonlinear transformations to map the high-dimensional pixel information to a low-dimensional latent space (for example, the dimension can be reduced to 128 or 256). This process is not simply dimension reduction, but through training to learn how to retain key information and discard redundant information, and finally obtain an abstract and structured compression of the original image.

[0085] In the specific implementation, refer to Figure 2 is a principle diagram of the image de-fogging method combining event features provided by the embodiment of the application. The input foggy image I i is input into a pre-trained VQ-VAE encoder f E to extract its latent feature representation x hzThe encoder Encoder is in a frozen state and does not require additional training, and is designed to compress images from pixel space to latent space to improve the efficiency and quality of subsequent diffusion sampling.

[0086] The latent representation x hz Gaussian noise is added to the latent representation x An initial diffusion noise representation is obtained as the starting point of the reverse denoising process.

[0087] The preset latent space encoder is trained to make the latent space have semantic consistency: that is, images with similar semantics (such as images of the same object under different fog concentrations) are also closer in feature representation in the latent space. This feature provides a good foundation for subsequent diffusion denoising, which can focus on removing fog-related disturbances at the semantic level without damaging the core semantic structure of the image.

[0088] The latent space encoding technology realizes the mapping of images from high-dimensional pixel space to low-dimensional latent space, providing more efficient and robust feature representation for subsequent dehazing processing.

[0089] In yet another embodiment provided by the present application, the event features of the event stream synchronously acquired with the foggy image include:

[0090] The event stream of the foggy image is synchronously acquired by an event camera;

[0091] The event stream is divided into a three-layer time pyramid, and each layer is further divided into two time periods, to obtain two event voxel slices respectively;

[0092] The obtained event voxel slices are combined to obtain a time pyramid representation of the event stream;

[0093] The time pyramid representation is input into a preset event encoder to extract structural edge features, to obtain the event features.

[0094] In the specific implementation of the present embodiment, the event stream is multi-scale time modeled by the time pyramid structure, and the structured edge features are extracted by combining the event encoder, to provide high-frequency information of the dynamic scene for dehazing.

[0095] Events Stream E, which is inputted into the Events Representation M. The event stream E is captured by an event camera (Prophesee EVK4 HD). Unlike traditional image cameras, the event camera does not record images in frames, but records the brightness change of each pixel with microsecond-level precision (triggered when it exceeds a certain threshold). In this paper, the event camera and the RGB camera are synchronously captured, i.e. each frame of image Ii has a corresponding event stream E. Both have consistent view and time stamp alignment, which are used to construct the input pair of RGB+events to obtain the structural features x of high dynamic range e .

[0096] The event stream is composed of a large number of asynchronous events, recording the time, location and polarity of the pixel brightness mutation. It has a very high time resolution but a sparse spatial distribution. By constructing a time pyramid, the dynamic changes of events can be captured at different time scales:

[0097] Level L = 3, which means that the event stream is divided into a three-layer time pyramid, and different layers represent different time scales

[0098] Low layer: small time window (e.g. 10ms), capturing fast dynamics;

[0099] Middle layer: medium time window (e.g. 50ms), balancing dynamics and statics;

[0100] High layer: large time window (e.g. 200ms), capturing global structure.

[0101] Moments M = 2: each layer of time window is evenly divided into two sub-time periods, e.g. the low layer of 10ms is divided into the first 5ms and the last 5ms, and the events in each sub-time period are converted into event voxel slices: 2 event voxel slices are obtained. The division of each layer into two time periods forms a contrast, which can strengthen the temporal correlation of events (such as motion direction, change rate).

[0102] Stack the three layers of 6 event voxel slices (3 layers x 2 segments) according to the time scale to form the time pyramid representation of the event stream.

[0103] Events TPR, which represents the Temporal Pyramid Representation of events, i.e. the time pyramid representation E of the event stream TPR , with dimensions [L, M, H, W].

[0104] It should be noted that the event pyramid representation can be replaced by event frame accumulation, event stream convolution, or a Transformer-based event encoder to adapt to different hardware platforms.

[0105] The event stream is converted into event voxel slices, and the events within a time window are accumulated into a multi-channel tensor, such as a polarity and a timestamp, to convert the sparse events into a dense spatial feature matrix, and adapt the processing mode of the convolutional neural network.

[0106] The event stream is sensitive to brightness changes and is naturally good at capturing object edges and motion contours (not affected by fog). The event encoder f e These anti-fog structural features are extracted from the temporal pyramid representation through convolutional layers, especially edge detection-oriented convolutional kernels conv3 and pooling, to provide clear prior clues for subsequent denoising.

[0107] The event stream is not sensitive to fog, and the extracted structural edge features, such as vehicle contours and road edges, remain clear in heavy fog and can be directly used as anchor points for dehazing, avoiding the loss of structure caused by fog blurring in traditional methods. The event features and the latent features of the foggy image are complementary, and their fusion can greatly improve the dehazing accuracy.

[0108] In yet another embodiment provided by the present application, the event features are fused into a preset image segmentation network, and the noise representation is iteratively diffused and denoised to obtain a denoising result, including:

[0109] The event features are fused into the middle layer of the image segmentation network through an event guidance module.

[0110] The noise representation is iteratively diffused and denoised several times through the fused image segmentation network to obtain the denoising result.

[0111] In the specific implementation of the present embodiment, the anti-fog structural information provided by the event stream is injected into the reverse denoising process of the diffusion model through event-guided cross-modal feature fusion, providing precise structural constraints for denoising iteration, thereby improving the clarity and accuracy of the dehazing result.

[0112] Using the DDIM (Denoising Diffusion Implicit Models) framework, x T is iteratively denoised for T steps, and a denoising U-Net is used at each step to predict the denoising result of x t In the t-th step, the current state is xt , the U-Net network predicts the de-noised result x t-1 .

[0113] It should be noted that Figure 2 Denoising U-Net in the EvDehaze model is the core module that performs diffusion back sampling, and its role is: at each time step t, according to the current noise latent x t , the de-noised x t-1 is predicted. This part of the network essentially functions as a noise predictor. It can be replaced by other network structures with similar capabilities. For example, a de-noising network based on ResNet; a deeper or shallower U-Net variant; a backbone network with a self-attention mechanism such as Swin Transformer, ConvNeXt, etc. The newly replaced model can perform the task of conditional noise estimation in the diffusion step. The core working principle remains unchanged after replacement, that is, starting from the noisy latent xT, the clear latent representation x0 is gradually generated through the reverse de-noising process of the diffusion model, and the final image I o is generated through the VQ-VAE decoder.

[0114] In order to adapt to the new noise predictor structure, the following adaptive adjustments need to be made: network structure adaptation: the new network replaced should support the embedding input of time step t (for example, through Time Embedding); the size and channel number of the input and output should be consistent with the original x t , in order to be compatible with the diffusion calculation process. Training method adaptation: keep consistent with the original training target, that is, minimize the diffusion loss function; if the convergence speed or training stability of the new model is different, it may be necessary to adjust the learning rate, loss weighting, regularization strategy and other hyperparameters. Event guiding mechanism adaptation, if the new structure does not use the U-Net encoding-decoding path, the insertion method of the event guiding module needs to be redesigned.

[0115] In the de-noising process, the event stream E is input into the event representation encoder f_e to obtain the high dynamic range structural feature xe. The event feature is fused into the middle layer of the U-Net through the event guiding module f eg . In the de-noising process, the event feature xe is introduced as a condition to guide the structure recovery of the dehazing through the fusion of the event guiding module feg and the image latent feature.

[0116] where T represents T steps of iteration.

[0117] q(x t |x t-1 ) represents the probability distribution of the forward diffusion process, which is used for noise modeling.

[0118] pθ (x t-1 |x t ,x hz ,x e ) denotes the distribution of the reverse denoising process, and the U-Net predicts x t , the original latent x hz and the event feature x e according to the current x t-1 .

[0119] The latent feature representation with added noise (noise representation) is taken as the initial input of the denoising iteration.

[0120] The fused image segmentation network receives the current noise representation and the iteration step information, encodes through the time embedding, and outputs the predicted value of the current noise.

[0121] According to the reverse formula of the predicted noise and the diffusion model, the denoised feature representation is calculated:

[0122] In each iteration step, the event guiding module recalculates the fusion of the event feature and the current intermediate layer feature, ensuring that the event information continues to play a guiding role at different noise levels.

[0123] From high noise steps to low noise, until all iterations are completed, the denoised latent feature representation is obtained.

[0124] The structural anchor points provided by the event feature effectively constrain the denoising process, so that the final defogging image can accurately preserve key structures such as object edges and dynamic trajectories.

[0125] It should be noted that the DDIM (Denoising Diffusion Implicit Model) 15-step reverse sampling strategy is adopted in the embodiment, which greatly reduces the sampling delay while maintaining image quality, and the inference speed is increased by 3-5 times compared with the standard DDPM model. In other embodiments, the diffusion backbone network can be replaced by other generative modeling architectures, such as Score-based Model, Transformer-based Diffusion (such as PixArt-α), to realize an equivalent step-by-step sampling defogging process.

[0126] The application scheme encodes the input fog image into a latent representation by a diffusion model dehazing backbone and a VQ-VAE latent space encoder; then, Gaussian noise is added in the diffusion modeling, and a clear latent representation is gradually recovered through a denoising network; finally, the dehazed image is recovered by a decoder. Through the Event Representation Module, the input event stream is encoded into a multi-scale voxel tensor through the Temporal Pyramid Representation (TPR), and the structural edge features are extracted through a convolutional neural network to obtain a high dynamic range event feature map. Through the Event-Guided Diffusion Module, the event features are injected into the intermediate layer of the U-Net through the Cross-Attention mechanism at each step of the diffusion model denoising, guiding the model to sample in the latent space along the structural consistent direction, so as to generate an image with clear structure and enhanced contrast. It can be applied to any intelligent perception system with event cameras and RGB imaging components, especially in scenes with fog interference, significantly improving the perception robustness and imaging quality.

[0127] In yet another embodiment provided by the application, the step of fusing the event features into a preset image segmentation network and diffusing and denoising the noise representation iteratively to obtain a denoised result comprises:

[0128] In each iteration, the event features are taken as a query condition, the image features output by the previous step are taken as a key and a value, attention weights are generated through a cross-attention mechanism, the image features output by the previous step are denoised, and a noise representation output by the current step is obtained;

[0129] The image features output by the last step are taken as the denoised result.

[0130] The initial noise representation output by the previous step is the noise representation.

[0131] In the specific implementation of the embodiment, the cross-attention mechanism is used to establish an accurate association between the event features and the image features, and the optimization direction of the image features is dynamically adjusted in each denoising iteration through event feature navigation, so that the gradual approximation from the noise representation to the clear features is realized.

[0132] In the cross-attention mechanism, the event features are taken as a query, and the image features are taken as a key and a value. e(Q), the image feature output in the previous step as the key Key and the value Value, that is, V and K in the figure. This design enables the event feature to actively retrieve the region in the image feature that matches itself. For example, the vehicle edge in the event feature serves as a query to locate and focus on the corresponding fuzzy region in the image feature, and the attention weight generated can quantify the semantic correlation strength between the two, providing clear correction coordinates for denoising.

[0133] The Cross-Attention fusion mechanism selects a multi-head attention block (Multi-head Cross Attention) embedded in the middle layer of the diffusion model U-Net to inject event features in a multi-scale manner, significantly enhancing the edge preservation capability.

[0134] It should be noted that the injection of event information is not limited to the Cross-Attention form, and can also use Channel-wise Fusion, Gated Fusion, or Conditional Normalization to fuse the event feature into the U-Net.

[0135] The event feature is fused into the middle layer of the U-Net through an event guiding module f eg The Cross-Attention mechanism is used to realize structural guidance for each step of denoising. In Cross-Attention, the event feature is used as the Query, and the image feature is used as the Key and Value to generate attention weights, thereby enhancing the model's perception of edges and contours.

[0136] The initial noise representation contains a large amount of fog-related noise, and each iteration injects structural constraints from the event feature into the image feature through cross-attention: regions with high attention weights, such as edge positions marked by the event, are preferentially denoised, while regions with low weights, such as pure fog regions, are moderately retained to preserve basic information. Through dynamic guidance, the denoising process is transformed from blind noise suppression to targeted structure recovery, gradually removing fog interference and strengthening the effective structures indicated by the event feature.

[0137] The image feature output in the previous step is used as the key and value for the next step, meaning that each denoising result inherits the optimization information from the previous iteration. For example, the first iteration focuses on repairing the key edges marked by the event feature, and the second iteration further optimizes the surrounding texture based on the repaired edges through cross-attention, forming a progressive recovery from core structures to detailed filling. The final denoising result has both structural integrity and detail richness.

[0138] In implementation, the total number of iterations is determined, such as T steps, and the noise representation of the initial last step output is added to the latent feature representation, denoted as F0.

[0139] The event feature (denoted as E) and the image feature (denoted as F t-1 , where t is the current iteration step, F0 is the initial noise representation when t = 1, and the dimension uniformity processing is performed, such as adjusting the channel number to the same dimension D through 1x1 convolution, to ensure that the cross attention can be normally calculated.

[0140] The projection matrix of the query, the key, and the value is calculated, and the scaled dot product attention formula is used to calculate the attention score matrix.

[0141] The score matrix S is row normalized to obtain the weight matrix.

[0142] The value V is weighted and summed through the attention weight matrix to obtain the attention feature A x V, which highlights the image region with strong correlation with the event feature.

[0143] The attention feature is combined with the image feature F t-1 of the last step to obtain F t .

[0144] The F t is convoluted and smoothed (such as 3x3 convolution) to further weaken the noise in the low weight region and retain the core structural information.

[0145] The F t is taken as the output of the tth step, which is used as the key and the value for the next iteration (t+1 step).

[0146] The above single step process is repeated from t = 1 to t = T, and the event feature is taken as the query, and the image feature of the previous output is taken as the key and the value, to gradually optimize the feature quality.

[0147] After the iteration is completed, the image feature F t output in the last step is taken as the denoising result, and the feature has been guided by the cross attention for multiple rounds to maximize the stripping of the fog noise and the retention of the structure indicated by the event feature.

[0148] Finally, the final x0 is decoded into the output image Io after defogging through the VQ-VAE decoder fD.

[0149] The cross attention can capture the fine-grained correlation between the event feature and the image feature, accurately locate the position that needs to be repaired in the denoising process, and avoid the structural deviation caused by the global weighting in the traditional fusion method.

[0150] In another embodiment provided by the application, the method further comprises:

[0151] input the obtained training image into a preset to-be-trained encoder for encoding to extract a training latent feature representation;

[0152] add noise to the training latent feature representation to obtain a training noise representation;

[0153] obtain a training event feature of an event stream synchronously collected with the training image;

[0154] fuse the training event feature into a preset to-be-trained image segmentation network, and perform iterative diffusion reverse denoising on the training noise representation to obtain a training denoising result;

[0155] input the training denoising result into a preset to-be-trained decoder for decoding to obtain a predicted defogging image;

[0156] determine a loss function between a real image corresponding to the training image and the predicted defogging image through a pixel-level L1 loss and a perception loss weighted combination;

[0157] According to the loss function, an optimizer is adopted to perform gradient back propagation to update network parameters of the to-be-trained encoder, the to-be-trained image segmentation network and the to-be-trained decoder, so as to obtain the encoder, the image segmentation network and the decoder.

[0158] In the specific implementation of the embodiment, through supervised learning and multi-loss function joint optimization, the model learns the mapping rule from a foggy image to a clear image in the simulation defogging process, and the structural prior of the event stream is used to improve the defogging accuracy and generalization ability of the model

[0159] The training image is encoded by the to-be-trained encoder, noise is added to simulate interference, the training event feature is fused for reverse denoising iteration, and finally the predicted defogging image is generated by the to-be-trained decoder. Through the whole process of encoding, adding noise, denoising and decoding, the model can learn the cooperative relationship of each link in a unified framework to ensure the process consistency during inference.

[0160] The pixel-level L1 loss and the perception loss are combined by weighting to realize double constraints on the defogging result: the L1 loss focuses on the direct difference of pixel values to ensure the consistency of the predicted defogging image and the real image in basic visual features such as brightness and color, avoiding overall color tone shift; the perception loss, usually based on a pre-trained image classification network, measures the difference of high-level semantic features to guide the model to preferentially restore the structural information of the image, avoiding visual distortion caused by only optimizing pixel values. The combination of the two makes the model not only ensure the pixel-level accuracy, but also generate clear images consistent with human visual perception.

[0161] In model training, pixel-level L1 loss and perceptual loss are introduced to jointly optimize the color fidelity and structural semantic consistency, effectively preventing the generation of fake textures.

[0162] In model training, specifically:

[0163] Collect large-scale paired training images and corresponding real images data, and synchronize the acquisition of event stream data corresponding to each training image, recorded by event cameras; the data should cover different fog concentrations (light fog, medium fog, and thick fog), scene types (urban road, rural area, and highway), and dynamic conditions (static, low-speed motion, and high-speed motion).

[0164] Crop (e.g., 256x256), normalize (pixel values scaled to [0,1]), and perform random flipping / rotation data augmentation on training images and real images to improve model generalization;

[0165] Generate training event features according to the method in the inference phase, time pyramid representation, and event encoder extraction to ensure spatial resolution matching with training images; match training images, real images, and event stream through timestamps to ensure time synchronization.

[0166] Input the training image into the encoder to be trained to obtain the training latent feature representation; add noise to the training latent feature according to the preset noise scheduler to generate the training noise representation.

[0167] Input the training event feature into the image segmentation network to be trained, fuse it with the training noise representation through cross-attention mechanism, and perform T-step diffusion reverse denoising iteration to obtain the training denoising result.

[0168] Input the training denoising result into the decoder to be trained to output the predicted defogging image.

[0169] Based on the prediction process, the loss function is calculated, and a joint loss function is used during training:

[0170] L total =λ pix ×L pix +λ perc ×L perc ;

[0171] L pix is the pixel-level L1 loss, which measures the difference between the output image I o and the target image I gt at the pixel level.

[0172] L pix =L1(I o ,I gt )=mean(abs(Io -I gt ))

[0173] L perc For perceptual loss, the difference between the output image and the target image in high-level semantic features is calculated.

[0174] Feature extraction is usually based on the intermediate layers of the pre-trained VGG network (such as the relu3_3 layer).

[0175] L perc = mean(abs(φ(I o )-φ(I gt ))) where φ(·) represents the VGG extracted feature representation.

[0176] λ pix and λ perc are loss weighting coefficients

[0177] Parameter optimization and iterative training, the total loss L is calculated by automatic differentiation to the gradient of all trainable parameters (weights of the encoder, image segmentation network, decoder).

[0178] Optimizers such as AdamW are used to update parameters by gradient backpropagation, with an initial learning rate of 1e-4, which is linearly decayed to 1e-6 with training rounds (such as 100 rounds).

[0179] Repeat the process of forward propagation, loss calculation and parameter update, use mini-batch data for each training round, and continue training until the loss converges, such as the validation set loss does not decrease for 10 consecutive rounds.

[0180] After training, save the final converged parameters to obtain the encoder, image segmentation network and decoder used in the inference stage.

[0181] Through this joint optimization strategy, the model can not only guarantee the accuracy of low-level pixel restoration, but also guarantee the consistency of high-level structure and semantics, so as to realize more real and detailed defogging results.

[0182] The application scheme introduces an event camera for image defogging. The event camera has a very high dynamic range (120 dB) and a microsecond-level time resolution. Compared with a traditional APS RGB camera (dynamic range of about 60 dB), it can capture more structural details and brightness changes in a foggy environment, significantly alleviating the information loss problem in the defogging process. An event-guided diffusion generation framework is designed: based on the strong generation ability of the diffusion model, combined with the structural features in the event stream, the event information is gradually injected into the sampling path of the latent space through the cross-attention mechanism, so as to generate defogging images with real structure and contrast under the premise of no real paired data supervision. A real event-image synchronous acquisition dataset is constructed to acquire unmanned aerial vehicle flight data from real foggy scenes, equipped with a Prophesee event camera and a RGB sensor, which realizes event-image synchronous acquisition and calibration in foggy environment for the first time, laying a data foundation for subsequent research. Moreover, the event camera is small in size and low in power consumption, and can be connected to the existing visual system as an external plug-in, reducing the engineering integration threshold.

[0183] In yet another embodiment provided by the application, to verify the effectiveness of the event-guided image defogging method proposed by the application, the method is specifically implemented by taking the acquisition of real foggy images by a UAV equipped with an event camera and a RGB camera as an example:

[0184] An event-image synchronous acquisition system is deployed on a UAV platform to acquire training and test data under real outdoor heavy fog conditions, and to complete defogging image reconstruction.

[0185] The UAV platform has high-precision flight and stable hovering capability; the load part integrates two image sensors vertically stacked, and the overlapping area of the view angle covers more than 95%; the event camera has a resolution of 1280x720 and supports a dynamic range of up to 120 dB with a sampling frequency of up to 100 kHz; the RGB camera has an effective resolution of 1280x720 and a frame rate of 30 FPS;

[0186] The installation distance between the two cameras is about 3.6 cm (achieved by a 3D printed bracket), and the calibration has been completed through a time synchronization module and a calibration board with an error of less than 0.2°.

[0187] The collection site is selected on a suburban road and a building area with serious air pollution (AQI>300);

[0188] The flight height of the UAV is set to 80-120 meters, and the trajectory covers straight routes and rotating paths around buildings; the collection time is about 30 minutes, and about 80,000 frames of RGB image and event stream data are collected, and the original event data is stored in RAW format.

[0189] The collected RGB images are used as input I i, event data E is encoded into event features x by event pyramid representation module e ;

[0190] Fog image I i Encoded into latent space features x by VQ-VAE hz , and Gaussian noise is added to generate x T ;

[0191] Using the DDIM reverse sampling process, x e is injected into the U-Net at each step; finally, the latent space representation x0 is restored, and the dehazed image I is output through the decoder o .

[0192] Referring to Figure 3 , the comparative diagram of the image dehazing effect provided by the embodiment of the application, Figure 3 , in (a), Input Frames represent the input image, i.e. the original foggy image (frames affected by severe light or atmospheric degradation), which is usually the input of the model.

[0193] Figure 3 , (b), DehazeFormer-B represents the dehazing result processed by the DehazeFormer-B model, which is a Transformer-based image dehazing method.

[0194] Figure 3 , (c), Restormer represents the output result of the Restormer model, which is a high-performance Transformer network suitable for various image restoration tasks.

[0195] Figure 3 , (d), ResShift represents the dehazing result of the baseline diffusion method ResShift, representing the performance of the current mainstream diffusion model without event input.

[0196] Figure 3 , (e), EvDehaze (Ours) represents the output result of the EvDehaze model proposed in this paper, which introduces event information for guidance, and the restored image has clearer structure and stronger contrast.

[0197] Figure 3 , (f), Ground Truth original image, true image, represents the undegraded clear image, which is used as the evaluation reference standard.

[0198] Referring to Figure 4 , the principle diagram of the image dehazing method combined with event features provided by the embodiment of the application.

[0199] Figure 4 Input Frames in (a), represents the input image, i.e. the original foggy image (frames affected by severe illumination or atmospheric degradation), which is usually the input of the model.

[0200] Figure 4 DehazeFormer-B in (b), represents the dehazing result after processing with DehazeFormer-B network, which is a Transformer-based image dehazing method.

[0201] Figure 4 Restormer in (c), represents the output result of Restormer model, which is a high-performance Transformer network suitable for various image restoration tasks.

[0202] Referring to Figure 5 is the dehazing result schematic diagram of the image dehazing method combining event features provided by the embodiment of the application. RGB Histogram represents the histogram of RGB color space, the horizontal axis (X axis): represents the pixel intensity PixelIntensity, ranging from 0 to 255, corresponding to the brightness value of the pixel in the image. The vertical axis (Y axis): represents the number of pixels Number ofPixels, i.e. the number of pixels corresponding to the pixel intensity in the image. The red curve R channel represents the pixel intensity distribution of the red channel. The green curve G channel represents the pixel intensity distribution of the green channel. The blue curve B channel represents the pixel intensity distribution of the blue channel.

[0203] Figure 5 a) in (a) represents the dehazing result of the baseline diffusion method ResShift, representing the performance of the current mainstream diffusion model without event input.

[0204] Figure 5 b), EvDehaze (Ours) in (b) represents the output result of the EvDehaze model proposed in this paper, which introduces event information for guidance, and shows a clearer and stronger contrast recovery image.

[0205] On real unmanned aerial vehicle collected data, the method of the application significantly improves the image visual quality;

[0206] Compared with the untreated image and the dehazing result, the histogram contrast, edge preservation, detail clarity and other aspects are obviously improved; in the area with high concentration of haze (such as distant buildings, traffic signs), the structure texture can still be restored, proving the high dynamic range reconstruction ability of the method.

[0207] The system structure of the embodiment has high universality and can also be deployed on a vehicle-mounted system, a ground robot, an edge monitoring device or the like, as long as time synchronization and visual angle overlap between an event sensor and an image sensor are maintained.

[0208] In another embodiment provided by the present application, referring to Figure 6 is a structural schematic diagram of an image defogging device combined with event features provided by an embodiment of the present application, and the device comprises:

[0209] The encoding module is configured to input the acquired foggy image into a preset encoder to encode and extract a latent feature representation.

[0210] The noise module is configured to add noise to the latent feature representation to obtain a noise representation.

[0211] The event module is configured to acquire an event feature of an event stream synchronously collected by the foggy image.

[0212] The denoising module is configured to fuse the event feature into a preset image segmentation network, perform iterative diffusion reverse denoising on the noise representation, and obtain a denoising result.

[0213] The decoding module is configured to input the denoising result into a preset decoder to decode and obtain a defogged image.

[0214] The image defogging device combined with event features provided by the embodiment can perform all steps and functions of the image defogging method combined with event features provided by any of the above embodiments, and the specific functions of the device will not be repeated here.

[0215] In another embodiment provided by the present application, referring to Figure 7 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. The terminal device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, for example, an image defogging program combined with event features. The processor implements the steps in each of the above image defogging methods combined with event features when executing the computer program, for example, steps S1-S5 shown in the figure. Figure 1 Alternatively, the processor implements the functions of each module in the above device embodiments when executing the computer program.

[0216] For example, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the terminal device. For example, the computer program can be divided into modules, and the specific functions of each module have been described in detail in the image defogging method combined with event features provided in any of the above embodiments. Here, the specific functions of the terminal device are not described again.

[0217] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The terminal device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of the terminal device, and does not constitute a limitation on the terminal device, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the terminal device can also include an input / output device, a network access device, a bus and the like.

[0218] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and is connected with all parts of the terminal device through various interfaces and lines.

[0219] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the terminal device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0220] The modules integrated in the terminal device can be stored in a computer-readable storage medium in the form of a software function unit and sold or used as an independent product. Based on this understanding, all or part of the processes in the above-mentioned embodiments can also be implemented by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0221] It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. An image defogging method combining event features, characterized by, The method comprises: inputting the obtained foggy image into a preset encoder for encoding to extract a latent feature representation; adding noise to the latent feature representation to obtain a noise representation; obtaining event features of an event stream synchronously collected with the foggy image; fusing the event features into a preset image segmentation network to perform diffusion reverse denoising iteration on the noise representation to obtain a denoised result; inputting the denoised result into a preset decoder for decoding to obtain a defogged image.

2. The image defogging method of claim 1, wherein, The method comprises: obtaining a foggy image; inputting the obtained foggy image into a preset latent space encoder to compress the foggy image from a pixel space into a latent space to extract the latent feature representation.

3. The method of claim 1, wherein, The method comprises: synchronously obtaining an event stream of the foggy image through an event camera; dividing the event stream into a three-layer time pyramid, and dividing each layer into two time periods to obtain two event voxel slices respectively; combining the obtained event voxel slices to obtain a time pyramid representation of the event stream; inputting the time pyramid representation into a preset event encoder to extract structural edge features to obtain the event features.

4. The method of claim 1, wherein, The method comprises: fusing the event features into an intermediate layer of the image segmentation network through an event guiding module; performing diffusion reverse denoising iteration on the noise representation several times through the fused image segmentation network to obtain the denoised result.

5. The image defogging method of claim 4, wherein, The method comprises: in each iteration step, taking the event features as a query condition, taking the image features output by the previous step as a key and a value, generating attention weights through a cross-attention mechanism, denoising the image features output by the previous step to obtain the noise representation output by the current step; taking the image features output by the last step as the denoised result; wherein the initial noise representation output by the previous step is the noise representation.

6. The method of claim 1, wherein, The method further comprises: inputting an obtained training image into a preset to-be-trained encoder for encoding to extract a training latent feature representation; adding noise to the training latent feature representation to obtain a training noise representation; obtaining training event features of an event stream synchronously collected with the training image; fusing the training event features into a preset to-be-trained image segmentation network to perform diffusion reverse denoising iteration on the training noise representation to obtain a training denoised result; inputting the training denoised result into a preset to-be-trained decoder for decoding to obtain a predicted defogged image; determining a loss function between a real image corresponding to the training image and the predicted defogged image through a pixel-level L1 loss and a perceptual loss weighted joint. According to the loss function, an optimizer is adopted to update network parameters of the to-be-trained encoder, the to-be-trained image segmentation network and the to-be-trained decoder by gradient back propagation, so as to minimize the loss value, and obtain the encoder, the image segmentation network and the decoder.

7. An image defogging device combining event features, characterized by, The device comprises: An encoding module is configured to input the obtained foggy image into a preset encoder to encode and extract a latent feature representation; A noise module is configured to add noise to the latent feature representation to obtain a noise representation; An event module is configured to obtain an event feature of an event stream synchronously collected by the foggy image; A denoising module is configured to fuse the event feature into a preset image segmentation network, and perform iterative diffusion reverse denoising on the noise representation to obtain a denoising result; A decoding module is configured to input the denoising result into a preset decoder to decode and obtain a defogged image.

8. A terminal device, comprising: The computer program is configured to be executed by the processor, and the processor implements the image defogging method with event features according to any one of claims 1 to 6 when the computer program is executed.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the image defogging method with event features according to any one of claims 1 to 6 when the computer program is executed.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

Images