Progressive depth video defogging method and system applied to hazy weather

Abstract

The invention belongs to the technical field of image processing, and designs an end-to-end video deblurring network comprising four fusion networks and one refining network, in the first fusion stage, the three fusion networks (Nets1, 2, 3) have the same network structure and parameters and are used for processing continuous adjacent frames, network parameters are reduced, and the fusion efficiency is improved. The second fusion network (Net4) is used for fusing the result of the first fusion stage to realize more accurate alignment and preliminary recovery, the refinement network (Net5) is used for further removing haze and recovering more details of the reference frame, the method adopts an implicit multi-stage alignment mode, the adjacent frames and the reference frame are gradually aligned through the fusion network, and in the alignment process, the adjacent frames and the reference frame are subjected to image processing. Optical flow estimation or deformable convolution is not used, the complexity of the alignment process is simplified, the recovery process is realized in the alignment process, and the defogging performance of the whole network is improved by using the refined network.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular to a progressive deep video defogging method and system applied in haze weather. Background Art

[0002] There are currently two main video restoration methods: solving restoration and alignment separately and solving restoration and alignment jointly.

[0003] The methods for solving restoration and alignment separately first use a variety of alignment methods such as Markov random fields, optical flow, deformable convolution, recurrent neural networks, etc. to wrap adjacent frames into a reference frame, and then use the restoration method to restore the aligned reference frame. For example, Cai et al. first constructed a Markov random field (MRF) model based on the intensity value, and then processed the spatiotemporal information of the three adjacent frames. Then, they restored the reference frame by optimizing the MRF likelihood function. (Reference 1: Cai B, Xu X, Tao D (2016) Real-time video dehazing based on spatial-temporal MRF. In: Advances in multimedia information processing, pp315–325) Xiang et al. first used optical flow to estimate the motion information between two frames of R. Li and L. Chen, and warped it to the reference frame, and then used the residual network to restore the aligned reference frame. This method Not only does it increase the model parameters generated by the optical flow estimation network, but it also reduces the performance of the trained model when the motion information is estimated inaccurately. (Reference 2: Xiang X, Wei H, Pan J (2020) Deep video deblurring using sharpness features from exemplars. IEEE Trans Image Process 29: 8976–8987) Wang et al. proposed using enhanced deformable convolution and offset estimation to align adjacent frames to the reference frame in the feature space, and use the spatiotemporal attention module to fuse these features. Then, they used the reconstruction module to restore the reference frame. However, when large motion or non-uniform haze occurs, this method has inaccurate estimates of the offset. (Reference 3: Wang X, Chan KCK, Yu K, Dong C, Loy CC (2019) EDVR: video restoration with enhanced haze nced deformable convolutional networks. In: IEEE Conference on computer vision and pattern recognition workshops, pp1954–1963) Zhong et al. [7] first used a recurrent neural network and a global spatiotemporal attention module to align adjacent frames to a reference frame, and then used a reconstruction network to restore the aligned reference frame. (Reference 4: Zhong Z, Gao Y, Zheng Y, Zheng B (2020) Efficient spatiotemporal recurrent neural network for video deblurring.In: European conference on computer vision, pp191–207). .

[0004] Another method is to jointly solve restoration and alignment, align adjacent frames to the reference frame, and restore the reference frame in the same process or through an end-to-end network. For example, Zhang et al. used 3D convolution to jointly process spatiotemporal information by slowly fusing multiple consecutive video frames, but 3D convolution significantly increases the computational complexity and memory, and reduces the efficiency of video processing. (Reference 5: Zhang K, Luo W, Zhong Y, Ma L, Liu W, Li H (2019) Adversarial spatio-temporal all earning for video deblurring. IEEE Trans Image Process 28 (1): 291–301) Li et al. proposed to transfer three adjacent frames into the feature space through three different convolutional layers, cascade these features, and then send them to the dehazing network. (Reference 6: Li B, Peng X, Wang Z, Xu J, Feng D (2018) End-to-end united video dehazing and dehazing. In: Proceedings of the conference on artificial intelligence, p p7016–7023) This method combines the adjacent frame alignment process and the reference frame restoration process into an end-to-end trainable network, but the single-stage process cannot restore severely degraded haze videos. Therefore, to address the above problem, a progressive deep video defogging method and system for haze weather is proposed. Summary of the invention

[0005] In order to make up for the deficiencies of the prior art and solve at least one technical problem raised in the background technology, the present invention proposes a progressive deep video defogging method and system for use in haze weather.

[0006] The technical solution adopted by the present invention to solve its technical problem is: the invention designs an end-to-end video deblurring network including four fusion networks and one refinement network. In the first fusion stage, the three fusion networks (Nets1, 2, 3) have the same network structure and parameters, which are used to process continuous adjacent frames and reduce network parameters. The second fusion network (Net4) is used to fuse the results of the first fusion stage to achieve more accurate alignment and preliminary restoration. The refinement network (Net5) further removes haze and restores more details of the reference frame.

[0007] This method adopts an implicit multi-stage alignment method to gradually align adjacent frames with the reference frame through a fusion network. During the alignment process, no optical flow estimation or deformable convolution is used, which simplifies the complexity of the alignment process.

[0008] The restoration process is not only implemented under the alignment process, but also uses a refinement network to improve the dehazing performance of the entire network.

[0009] During the training process, a smooth L1 loss function and a perceptual loss function are used to jointly constrain the training process of the multi-stage network. The smooth L1 loss function is used for pixel-level constraints to improve the quality of the restored image. The perceptual loss function is used to constrain the training process of the feature space, which helps to improve the deblurring performance of the network. In order to verify the effectiveness of this method, experiments are conducted on a real haze video dataset and compared with other state-of-the-art image and video deblurring methods to evaluate the performance of this method.

[0010] Beneficial effects of the present invention:

[0011] The end-to-end video dehazing method proposed in this invention can significantly improve the dehazing effect and make the generated video frames clearer and more realistic.

[0012] By using three fusion networks with the same parameters in the first fusion stage, the invention effectively reduces the network parameters, reduces the amount of calculation and improves the processing speed.

[0013] Enhanced robustness: This method can handle video defogging problems in various complex scenarios, including those with rapid scene changes and large amounts of correlation, and has strong robustness. It can be widely used in monitoring, autonomous driving and other fields, providing clearer and more accurate video information for these fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0015] In the attached picture:

[0016] Figure 1 is a flow chart of the method of the present invention;

[0017] Figure 2 is the UNet network architecture of the model;

[0018] Figure 3 The ResNet network architecture of the model;

[0019] Figure 4 This is a comparison of real frames and blurred frames from the REVIDE dataset;

[0020] Figure 5 This is a schematic diagram of the experimental results of different frame dehazing methods on the REVIDE test dataset;

[0021] Figure 6 This is a schematic diagram of the experimental results of different frame defogging methods on consecutive adjacent frames;

[0022] Figure 7 It is a schematic diagram of the present invention using different baseline methods to perform frame defogging on the REVIDE test dataset. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Specific examples are given below.

[0025] Step 1: Input continuous video frames and select one of them as a reference frame. In this embodiment, five continuous frames of images in the video are selected as input, and the middle frame f t These five frames constitute a time unit to capture the coherence in space and time. Let these five frames be f t-2 , f t-1 , f t , f t+1 , f t+2 .

[0026] Step 2: Build a fusion network. The fusion network includes multiple convolutional layers, pooling layers, upsampling layers, and attention mechanism modules. Specifically, the structure of the fusion network is as follows:

[0027] Input layer: receives adjacent frames as input;

[0028] Convolutional layer 1: 3x3 convolution kernel is used for feature extraction, and the number of output channels is 64; the mathematical expression is: F 1 =Conv(f,3×3,64);

[0029] Pooling layer 1: Use 2x2 maximum pooling for downsampling; P 1 =MaxPool(F 1 , 2×2);

[0030] Attention mechanism module 1: Introduce the attention mechanism to increase the receptive field of the convolutional layer; let the output of the attention mechanism be A 1, then: A 1 =Attention(P 1 );

[0031] Convolutional layer 2: 3x3 convolution kernel is used for feature extraction, and the number of output channels is 128; the mathematical expression is: F 2 =Conv(A 1 , 3×3,128);

[0032] Pooling layer 2: Use 2x2 maximum pooling for downsampling; P 2 =MaxPool(F 2 , 2×2);

[0033] Attention mechanism module 2: Introduce the attention mechanism to increase the receptive field of the convolutional layer; let the output of the attention mechanism be A 2 , then: A 2 =Attention(P 2 );

[0034] Upsampling layer 1: Bilinear interpolation is used for upsampling to restore the resolution of the image; the mathematical expression is: U 1 =UPsample(A 2 , 2×2);

[0035] Convolutional layer 3: 3x3 convolution kernel is used for feature extraction, and the number of output channels is 64; the mathematical expression is: F 3 =Conv(U 1 , 3×3, 64);

[0036] Output layer: Output aligned frames and preliminarily dehazed frames. The mathematical expression is:

[0037] O 1 =OutputLayer(F 3 );

[0038] Step 3: Input the adjacent frames into the fusion network to obtain the aligned frames and the frames after preliminary defogging. Specifically, the adjacent frames are respectively input into the fusion network. After being processed by the convolution layer, the pooling layer, the attention mechanism module and the upsampling layer, the aligned frames and the frames after preliminary defogging are obtained. Let the output be O 1，t-1 , O 1，t , O 1，t+1 .

[0039] Step 4: Build a refined network. The refined network includes multiple convolutional layers, skip connections, and activation function modules. Specifically, the structure of the refined network is as follows:

[0040] Input layer: receives the preliminary dehazed frame and the reference frame as input;

[0041] Convolutional layer 1: 3x3 convolution kernel is used for feature extraction, and the number of output channels is 64; the mathematical expression is: F 1 =Conv(O 2，t , 3×3, 64);

[0042] Skip connection 1: connect the input layer to the output of convolutional layer 1; the mathematical expression is: C 1 =Concat(O 2，t , F 1 );

[0043] Activation function module 1: ReLU activation function is used for nonlinear transformation; the mathematical expression is: A 1 =ReLU(C 1 );

[0044] Convolutional layer 2: 3x3 convolution kernel is used for feature extraction, and the number of output channels is 128; the mathematical expression is: F 2 =Conv(A 1 , 3×3,128);

[0045] Skip connection 2: connect the output of convolutional layer 1 with the output of convolutional layer 2; the mathematical expression is: C 2 =Concat(F 1 , F 2 );

[0046] Activation function module 2: Use ReLU activation function for nonlinear transformation; the mathematical expression is: A 2 =ReLU(C 2 );

[0047] Convolutional layer 3: 3x3 convolution kernel is used for feature extraction, and the number of output channels is 64; the mathematical expression is: F 3 =Conv(A 2 , 3×3, 64);

[0048] Output layer: Output the final defogging frame; the mathematical expression is O 3 =OutputIayer(F 3 ).

[0049] Step 5: The preliminary dehazed frame and the reference frame are input into the refinement network respectively. After being processed by the convolution layer, skip connection and activation function module, the final dehazed frame is obtained.

[0050] Step 6: Repeat steps 2 to 5 to process each frame in the video to obtain the defogging video. During the training process, we use the same loss function to constrain the initial defogging result O 2，t And the final dehazing result O3，t , the specific loss function is as follows:

[0051] SmoothL1 loss function is used for pixel-level constraints, and its mathematical expression is:

[0052]

[0053] in

[0054]

[0055] here, represents the intensity of pixel x in the i-th channel of the dehazed frame, J i (x) represents the true value of the reference frame, and N is the total number of pixels.

[0056] Perceptual loss: used to improve visual effects, the mathematical expression is:

[0057]

[0058] Among them, φ i represents the i-th layer feature map extracted from the pre-trained VGG19, M is the total number of feature layers, and λ i is the weight of the feature map of the i-th layer.

[0059] The total loss function, mathematical expression is:

[0060] L=αL SmoothL1 +βL P ;

[0061] Among them, α and β are balance weights.

[0062] Through the above steps and the definition of the loss function, the method of the present invention can effectively perform defogging on the video and improve the quality and clarity of the video.

[0063] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A progressive deep video defogging method for use in haze weather, characterized by: The following steps are involved: S1: Take consecutive adjacent frames in the foggy video as input; S2: In the first stage, five consecutive adjacent frames are divided into three groups in turn, and then each group of frames is cascaded and sent to three fusion networks to achieve alignment processing; S3: In the second stage, the three adjacent frames in the first stage are sent to another fusion network to implement the alignment process and preliminary recovery process; S4: In the third stage, RefinementNetwork further removes the residual haze and obtains the preliminary dehazing results; S5: Loss function constraint. During the training process, a specific loss function is used to constrain the network training to ensure that the network can learn effective alignment and restoration strategies. These loss functions include but are not limited to perceptual loss and smoothness loss. They work together to ensure the clarity and quality of the output video.

2. The method for progressive deep video defogging in haze weather according to claim 1, characterized in that: The progressive alignment method is achieved by constructing a fusion network based on a U-Net structure, which can automatically learn the alignment relationship between adjacent frames and reference frames and align them in the feature space.

3. The method for progressive deep video defogging in haze weather according to claim 2 is characterized in that: The refinement network adopts a deep convolutional neural network structure to further process the aligned and defogged frames output by the fusion network to improve the defogging performance and reduce flickering and artifact phenomena.

4. The method for progressive deep video defogging in haze weather according to claim 3 is characterized in that: The following steps are involved: In the training phase, a dataset containing blurry video frames and corresponding clear frames is used for training, and the dehazing performance of the model is improved by optimizing the loss function; During the testing phase, the trained model is used to dehaze new blurry videos.

5. The method for progressive deep video defogging in haze weather according to claim 4 is characterized in that: The loss function includes alignment loss and restoration loss, which are used to measure the alignment effect and dehazing effect respectively, so as to guide the training and optimization of the model.

6. A progressive deep video defogging system for use in haze weather, characterized by: The system is applicable to a progressive deep video defogging method proposed in rights 1 to 6 for use in haze weather; it is characterized by comprising a data input module, an alignment module, a restoration module, a refinement module and an output module.

Images