A video dehazing method and system based on dual constraints of color domain and frequency domain

Abstract

This invention discloses a video dehazing method and system based on dual constraints in the color domain and frequency domain. It includes: synthesizing a video haze dataset simulating a real-world haze scene; constructing a video dehazing network with dual constraints in the color domain and frequency domain; training the video dehazing network using the video haze dataset, and applying loss functions L1 and L2. c The network training is constrained to update in a natural dehazing direction. During the network fixing phase, the trained network weights are loaded, and a real haze test dataset is input into the video dehazing network to obtain predicted haze-free frames. The dehazing effect is evaluated, and if the performance is good, the parameters are fixed to form the final video dehazing network. In the network application phase, the user inputs the haze video to be processed into the final video dehazing network to obtain the dehazed video. This invention can remove haze from real video scenes. Compared with existing video dehazing methods, it has significant advantages in dehazing quality evaluation and dehazing speed.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and specifically to a video dehazing method and system based on dual constraints of color domain and frequency domain. Background Technology

[0002] Haze reduces visibility, resulting in insufficient image and video clarity when cameras capture images and videos outdoors, severely impacting the performance of outdoor vision systems. Video dehazing aims to utilize atmospheric optical models and advanced computational methods to effectively estimate physical factors such as fog transmittance, atmospheric light intensity, and spatial visibility, achieving efficient restoration of scene information and video frame details. This is crucial for the robustness of modern intelligent vision systems.

[0003] Depending on the input type, dehazing methods can be broadly categorized into image dehazing and video dehazing. Compared to image dehazing methods that rely solely on the texture appearance of a single frame, video dehazing presents a greater challenge because it must consider how to model and utilize the inherent temporal correlations between consecutive frames. Classical video dehazing methods extend single-frame-based dehazing approaches by leveraging temporal consistency information between frames. With the development of deep networks, various neural networks have been introduced to address the video dehazing problem. Early learning-based video dehazing methods typically modeled temporal consistency by simply connecting input frames or feature maps. By training on large-scale synthetic datasets, these methods can extract effective features to directly estimate the transmittance map. Currently, some researchers have proposed using matching costs and confidence scores to align and acquire temporal consistency information and perform video dehazing in an end-to-end manner.

[0004] One existing technology is the patent "Video Dehazing Model and Method Based on Phase Feature Fusion using Memory Network (CN115471418A)". This method first uses a color query encoder and a phase query encoder to extract color query features and phase query features from hazy video frames. Second, it uses a color memory encoder and a phase memory encoder to extract color memory features and phase memory features from the past n frames. Third, it employs a cross-modal memory reading module to fuse the extracted color query features, phase query features, color memory features, and phase memory features. Finally, it inputs the obtained fused features into a decoder for result prediction. The drawback of this method is that, due to the use of a relatively complex deep network structure, it suffers from a large number of model parameters, high computational complexity, and an inability to remove haze from complex open scenes in the real world in real time.

[0005] The second existing technology is a multi-range temporal alignment video dehazing network with physical prior proposed in the paper "Video Dehazing via a Multi-Range TemporalAlignment Network with Physical Prior". This method first designs a memory-based physical prior guidance module to encode prior-related features into long-range memory. Then, a multi-range scene brightness recovery module is designed to capture spatiotemporal dependencies across multiple spatiotemporal ranges, and further effectively aggregates temporal information from adjacent frames. The drawback of this method is that it cannot effectively dehaze videos with dense fog due to a lack of sufficient prior knowledge, and it still cannot meet real-time requirements. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing methods and propose a video dehazing method and system based on dual constraints of the color domain and frequency domain. The main problem addressed by this invention is how to improve dehazing quality and speed to achieve rapid removal of fog and haze from realistic, complex, and open scenes.

[0007] To address the aforementioned problems, this invention proposes a video dehazing method based on dual constraints of color domain and frequency domain, the method comprising:

[0008] Synthesize a low-light video haze dataset to simulate a real haze scene;

[0009] Construct a video dehazing network with dual constraints in both the color domain and the frequency domain;

[0010] The video dehazing network, constrained by both color domain and frequency domain, was trained using the aforementioned video haze dataset, and both the similarity comparison loss function L1 and the perceptual contrast loss function L... c This constrains the network's training to update in a more natural, de-fogging direction;

[0011] In the network fixing phase, the trained network weights are loaded, and the test dataset based on real haze videos is input into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames. The dehazing effect is evaluated using a no-reference graph quality evaluation index. When the performance on the evaluation index is good, the parameters are fixed to form the final video dehazing network with dual constraints in the color domain and frequency domain (CFVD-Net). In the network application phase, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video.

[0012] Preferably, a low-light video haze dataset simulating a real haze scene is synthesized, specifically as follows:

[0013] Following Occam's razor principle and based on Retinex theory, the original atmospheric light scattering model is improved. The improved model formula is as follows:

[0014] I(x)≈{m α}(Rg(x)α+1-g(x)α),

[0015] Where m α <m represents the dark light constraint term, Rg(x)α+1-g(x)α represents the haze constraint term, I(x) is the synthesized haze frame, m is the global atmospheric light value, α is a random value less than 1, R is the reflection component information of the object, x is the pixel value, and g=e -βt The transmittance map is given, where β is a random number in the range [0.1, 0.5], and t is the scene depth.

[0016] Based on this formula, a low-light video haze dataset simulating a real haze scene is synthesized.

[0017] Preferably, a video dehazing network with dual constraints in both the color domain and frequency domain is constructed, specifically as follows:

[0018] For the design of the color domain constrained network branch, an encoder-decoder network is first constructed to extract high-quality spatial haze-related features, directly extracting high-level spatial features from the input haze frames. Then, these representation features are input into a bidirectional recurrent long short-term memory network. The construction process is as follows:

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] h (s) =Conv(Relu(Conv(o (s) )), tanhC (s) ),

[0025]

[0026] Where f (s) This represents the forget gate information of frame s, μ represents the Sigmoid activation function, and w (f) It is the weight matrix from hidden layer to hidden layer in the forget gate, w (i) It is the weight matrix from hidden layer to hidden layer in the input gate, w (c)It is the weight matrix from hidden layer to hidden layer in the cell, w (o) It is the weight matrix from hidden layer to hidden layer in the output gate. and h represents the spatial features extracted from frame s and frame s-1 by the bidirectional recurrent long short-term memory network, respectively. (s) H represents the restored result of frame s. (s-1) This represents the restored result of frame s-1. This represents the reverse reconstruction result of frame s, v (f) It is the bias vector in the forget gate update process, v (i) It is the bias vector during the input gate update process, v (c) It is the bias vector during the cell update process, v (o) It is the bias vector during the output gate update process, i (s) C represents the input gate information of the s-frame. (s) It is the cell information of the s-frame, C (s-1) It is the cell information of frame s-1. Indicates updated cell information, o (s) This indicates the output gate information of the s-frame. This indicates a more refined dehazing result for frame s.

[0027] ⊙ indicates pixel multiplication, * indicates convolution; and The data is concatenated and input into a bidirectional recurrent long short-term memory network for information updates, and then the input gate is... (s) The information from the updated bidirectional recurrent long short-term memory network is connected and fed into the convolutional layer to obtain the restored result h. (s) Simultaneously, the reverse restoration result is obtained by preserving the order of each frame in reverse. h (s) and Simultaneous input to three convolutional layers yields a more refined dehazing result.

[0028] In the spatial feature extraction stage, four downsampling modules Block1, Block2, Block3 and Block4 are constructed in the encoder based on ResNet, and four upsampling modules UP1, UP2, UP3 and UP4 are constructed in the decoder using Pixel Shuffle, pixel attention and channel attention.

[0029] The design of the frequency domain constraint network branch involves swapping the high-frequency and low-frequency components of each hazy video frame and the corresponding clear frame. The low-frequency components in the hazy frame contain more hazy background features and less hazy information. Based on this prior image information, we decompose each frame into low-frequency and high-frequency components using two-dimensional discrete wavelet transform, and thus construct a spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition.

[0030] Based on the color domain constrained network branch and the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition, a video dehazing network based on dual constraints in the color domain and frequency domain is constructed. Spatial features are extracted using the color domain constrained network branch, and spatial frequency features are extracted using the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition. To fuse the features extracted from multiple domains, a cross-domain fusion operation is performed, as shown in the following formula:

[0031]

[0032] in A learnable factor whose value is determined by the sigmoid operation on the parameter θ. Decision, and F1 and F2 are the output features of the spatial frequency domain constraint network branch and the color domain constraint network branch based on two-dimensional discrete wavelet decomposition, respectively.

[0033] Preferably, the video dehazing network with dual constraints in the color domain and frequency domain is trained using the video haze dataset, and the similarity comparison loss function L1 and the perceptual contrast loss function L2 are used. c To constrain the network's training towards a more natural, dehazing-free update process, specifically:

[0034] First, the L1 loss function is used to measure the similarity between the dehazed image and the clear sample. The formula is as follows:

[0035]

[0036] Where y pre y gt These represent the predicted fog-free frame output and the true clear label, respectively, where N is the total number of pixels;

[0037] Meanwhile, a perceptual contrast loss function is used to constrain the haze-free image restored by the network to be closer to the clear sample image. The formula for the target loss function is:

[0038]

[0039] Where w is the penalty factor, λ n y is the weighting coefficient. p y fLet represent the distances between positive and negative sample pairs in the metric space, respectively, and the specific formulas are as follows:

[0040]

[0041] Where, θ n For the nth hidden layer feature extracted from the VGG-16 model with fixed pre-trained weights, y i The original haze frame is represented by σ, which is the kernel coefficient used in the loss function.

[0042] In summary, the total loss function is:

[0043] L = aL1 + bL c ,

[0044] Where a and b are weighting coefficients, a = 1, b = 0.04.

[0045] Preferably, in the network fixing phase, the trained network weights are loaded, and the test dataset based on real haze videos is input into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames. The dehazing effect is evaluated using a no-reference-map quality evaluation metric. When the performance on the evaluation metric is good, the parameters are fixed to form the final video dehazing network (CFVD-Net) with dual constraints in the color domain and frequency domain. Specifically:

[0046] Each video in the test dataset, composed of real smog videos, is divided into several reasonable frames. The size of each frame in the same test dataset is consistent, while the size of each frame in different test datasets is not exactly the same.

[0047] Load the trained network weights and input the test video frame into the video dehazing network with dual constraints in the color domain and frequency domain to obtain the predicted haze-free frame;

[0048] The dehazing effect of each frame is evaluated using a no-reference-map quality evaluation metric. The parameters of the high-performing network are fixed to obtain the weights of the trained network, forming the final video dehazing network (CFVD-Net) with dual constraints in the color domain and frequency domain.

[0049] Accordingly, the present invention also provides a video dehazing system based on dual constraints of color domain and frequency domain, comprising:

[0050] The Synthetic Haze Dataset Unit is used to synthesize a low-light video haze dataset that simulates a real haze scene.

[0051] A video dehazing network building unit is used to construct a video dehazing network with dual constraints in the color domain and frequency domain.

[0052] The loss function constraint unit is used to train the video dehazing network with dual constraints in the color domain and frequency domain using the video haze dataset, and uses the similarity comparison loss function L1 and the perceptual contrast loss function L... c This constrains the network's training to update in a more natural, de-fogging direction;

[0053] The network fixing and application unit is used in the network fixing stage to load the trained network weights, input the test dataset based on real haze videos into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames, evaluate the dehazing effect using a no-reference graph quality evaluation index, and fix the parameters when the performance on the evaluation index is good to form the final video dehazing network with dual constraints in the color domain and frequency domain (CFVD-Net); in the network application stage, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video.

[0054] Implementing this invention has the following beneficial effects:

[0055] This invention proposes a video dehazing network based on dual constraints in the color and frequency domains. This method can remove haze from realistic video hazy scenes. Compared to existing video dehazing methods, it has significant advantages in dehazing quality evaluation and speed. In this invention, the inputs and outputs of each part of the network are interconnected, extracting and integrating different features, improving efficiency and making the restored haze-free frames more natural. This provides a foundation for tasks such as mid-level video segmentation and high-level target detection in later applications such as autonomous driving, drone aerial photography, and nearshore marine monitoring. Attached Figure Description

[0056] Figure 1 This is a flowchart of a video dehazing method based on dual constraints of color domain and frequency domain according to an embodiment of the present invention;

[0057] Figure 2 This is a branch structure diagram of a spatial frequency domain constraint network based on two-dimensional discrete wavelet decomposition.

[0058] Figure 3 This is a diagram of the branch structure of the color domain constraint network;

[0059] Figure 4 This is the overall network structure diagram;

[0060] Figure 5 This is a structural diagram of a video dehazing system based on dual constraints of color domain and frequency domain according to an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Figure 1 This is a flowchart of a video dehazing method based on dual constraints of color domain and frequency domain according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0063] S1, synthesize a low-light video haze dataset that simulates a real haze scene;

[0064] S2, constructing a video dehazing network with dual constraints in the color domain and frequency domain;

[0065] S3, the video dehazing network with dual constraints in the color domain and frequency domain is trained using the video haze dataset, and the similarity comparison loss function L1 and the perceptual contrast loss function L2 are used. c This constrains the network's training to update in a more natural, de-fogging direction;

[0066] S4. In the network fixing stage, the trained network weights are loaded, and the test dataset based on real haze videos is input into the video dehazing network with dual constraints in color domain and frequency domain to obtain predicted haze-free frames. The dehazing effect is evaluated using the no-reference graph quality evaluation index. When the performance on the evaluation index is good, the parameters are fixed to form the final video dehazing network (CFVD-Net) with dual constraints in color domain and frequency domain.

[0067] S5: In the network application stage, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in color domain and frequency domain to obtain the dehazed video.

[0068] Step S1 is as follows:

[0069] S1-1, following Occam's razor and based on Retinex theory, improves the original atmospheric light scattering model. The improved model formula is as follows:

[0070] I(x)≈{m α}(Rg(x)α+1-g(x)α),

[0071] Where m α <m represents the dark light constraint term, Rg(x)α+1-g(x)α represents the haze constraint term, I(x) is the synthesized haze frame, m is the global atmospheric light value, α is a random value less than 1, R is the reflection component information of the object, x is the pixel value, and g=e-βt The transmittance map is given, where β is a random number in the range [0.1, 0.5], and t is the scene depth.

[0072] Based on this formula, a low-light video haze dataset simulating a real haze scene is synthesized.

[0073] Step S2 is as follows:

[0074] S2-1, for the design of the color domain constrained network branch, firstly, an encoder-decoder network is constructed for high-quality extraction of spatial haze-related features, directly extracting high-level spatial features from the input haze frames, and then these representation features are input into a bidirectional recurrent long short-term memory network. The construction process is as follows:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] h (s) =Conv(Relu(Conv(o (s) )), tanhC (s) ),

[0081]

[0082] Where f (s) This represents the forget gate information of frame s, μ represents the Sigmoid activation function, and w (f) It is the weight matrix from hidden layer to hidden layer in the forget gate, w (i) It is the weight matrix from hidden layer to hidden layer in the input gate, w (c) It is the weight matrix from hidden layer to hidden layer in the cell, w (o) It is the weight matrix from hidden layer to hidden layer in the output gate. and h represents the spatial features extracted from frame s and frame s-1 by the bidirectional recurrent long short-term memory network, respectively. (s) H represents the restored result of frame s. (s-1) This represents the restored result of frame s-1. This represents the reverse reconstruction result of frame s, v (f) It is the bias vector in the forget gate update process, v (i) It is the bias vector during the input gate update process, v (c)It is the bias vector during the cell update process, v (o) It is the bias vector during the output gate update process, i (s) C represents the input gate information of the s-frame. (s) It is the cell information of the s-frame, C (s-1) It is the cell information of frame s-1. Indicates updated cell information, o (s) This indicates the output gate information of the s-frame. This indicates a more refined dehazing result for frame s, ⊙ represents pixel multiplication, and * represents convolution. and The data is concatenated and input into a bidirectional recurrent long short-term memory network for information updates, and then the input gate is... (s) The information from the updated bidirectional recurrent long short-term memory network is connected and fed into the convolutional layer to obtain the restored result h. (s) Simultaneously, the reverse restoration result is obtained by preserving the order of each frame in reverse. h (s) and Simultaneous input to three convolutional layers yields a more refined dehazing result.

[0083] In the spatial feature extraction stage, four downsampling modules (Block1, Block2, Block3, and Block4) are constructed in the encoder based on ResNet. In the decoder, four upsampling modules (UP1, UP2, UP3, and UP4) are constructed using Pixel Shuffle, pixel attention, and channel attention. The color gamut constrained network branch structure diagram is shown below. Figure 3 As shown;

[0084] S2-2, the design of the frequency domain constraint network branch, involves swapping the high-frequency and low-frequency components of each hazy frame and its corresponding clear frame. The low-frequency components in the hazy frame contain more hazy background features and less hazy information. Based on this prior image information, we decompose each frame into low-frequency and high-frequency components using two-dimensional discrete wavelet transform, and thus construct a spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition, as follows: Figure 2 As shown;

[0085] S2-3, based on the color domain constraint network branch and the spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition, a video dehazing network based on dual constraints of color domain and frequency domain is constructed, such as... Figure 4As shown, (1) represents the spatial frequency domain constraint network branch of the two-dimensional discrete wavelet decomposition, and (2) represents the color domain constraint network branch. Spatial features are extracted using the color domain constraint network branch, and spatial frequency features are extracted using the spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition. In order to fuse the features extracted from multiple domains, a cross-domain fusion operation is performed, and the formula is as follows:

[0086]

[0087] in A learnable factor whose value is determined by the sigmoid operation on the parameter θ. Decision, and F1 and F2 are the output features of the spatial frequency domain constraint network branch and the color domain constraint network branch based on two-dimensional discrete wavelet decomposition, respectively.

[0088] Step S3 is as follows:

[0089] S3-1, First, the L1 loss function is used to measure the similarity between the dehazed image and the clear sample. The formula is:

[0090]

[0091] Where y pre ,y gt These represent the predicted fog-free frame output and the true clear label, respectively, where N is the total number of pixels;

[0092] Meanwhile, a perceptual contrast loss function is used to constrain the haze-free image restored by the network to be closer to the clear sample image. The formula for the target loss function is:

[0093]

[0094] Where w is the penalty factor, λ n y is the weighting coefficient. p ,y f Let represent the distances between positive and negative sample pairs in the metric space, respectively, and the specific formulas are as follows:

[0095]

[0096] Where, θ n For the nth hidden layer feature extracted from the VGG-16 model with fixed pre-trained weights, y i The original haze frame is represented by σ, which is the kernel coefficient used in the loss function.

[0097] In summary, the total loss function is:

[0098] L = aL1 + bL c ,

[0099] Where a and b are weighting coefficients, a = 1, b = 0.04.

[0100] Step S4 is as follows:

[0101] S4-1: Each video in the test dataset composed of real haze videos is decomposed into several reasonable frames. The size of each frame in the same test dataset is consistent, while the size of each frame in different test datasets is not exactly the same.

[0102] S4-2, Load the trained network weights, and input the test video frame into the video dehazing network with dual constraints in the color domain and frequency domain to obtain the predicted haze-free frame;

[0103] S4-3 uses no-reference-map quality evaluation metrics such as NIQE, NIMA, and FADE to evaluate the dehazing effect of each frame, fixes the parameters of the well-performing network, obtains the training network weights, and forms the final video dehazing network (CFVD-Net) with dual constraints in the color domain and frequency domain.

[0104] Accordingly, the present invention also provides a video dehazing method system based on dual constraints of color domain and frequency domain, such as... Figure 5 As shown, it includes:

[0105] Synthetic Haze Dataset Unit 1 is used to synthesize a low-light video haze dataset that simulates a real haze scene.

[0106] Specifically, following Occam's razor principle and based on Retinex theory, the original atmospheric light scattering model is improved, and the improved model formula is as follows:

[0107] I(x)≈{m α}(Rg(x)α+1-g(x)α),

[0108] Where m α <m represents the dark light constraint term, Rg(x)α+1-g(x)α represents the haze constraint term, I(x) is the synthesized haze frame, m is the global atmospheric light value, α is a random value less than 1, R is the reflection component information of the object, x is the pixel value, and g=e -βt The transmittance map is given, where β is a random number in the range [0.1, 0.5], and t is the scene depth.

[0109] Based on this formula, a low-light video haze dataset simulating a real haze scene is synthesized.

[0110] Video Dehazing Network Building Unit 2 is used to construct a video dehazing network with dual constraints in the color domain and frequency domain.

[0111] Specifically, for the design of the color domain constrained network branch, an encoder-decoder network is first constructed to extract high-quality spatial haze-related features, directly extracting high-level spatial features from the input haze frames. Then, these representation features are input into a bidirectional recurrent long short-term memory network, the construction process of which is as follows:

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] h (s) =Conv(Relu(Conv(o (s) )), tanhC (s) ),

[0118]

[0119] Where f (s) This represents the forget gate information of frame s, μ represents the Sigmoid activation function, and w (f) It is the weight matrix from hidden layer to hidden layer in the forget gate, w (i) It is the weight matrix from hidden layer to hidden layer in the input gate, w (c) It is the weight matrix from hidden layer to hidden layer in the cell, w (o) It is the weight matrix from hidden layer to hidden layer in the output gate. and h represents the spatial features extracted from frame s and frame s-1 by the bidirectional recurrent long short-term memory network, respectively. (s) H represents the restored result of frame s. (s-1) This represents the restored result of frame s-1. This represents the reverse reconstruction result of frame s, v (f) It is the bias vector in the forget gate update process, v (i) It is the bias vector during the input gate update process, v (c) It is the bias vector during the cell update process, v (o) It is the bias vector during the output gate update process, i (s) C represents the input gate information of the s-frame. (s) It is the cell information of the s-frame, C (s-1) It is the cell information of frame s-1. Indicates updated cell information, o (s) This indicates the output gate information of the s-frame. This indicates a more refined dehazing result for frame s, ⊙ represents pixel multiplication, and * represents convolution. and The data is concatenated and input into a bidirectional recurrent long short-term memory network for information updates, and then the input gate is... (s) The information from the updated bidirectional recurrent long short-term memory network is connected and fed into the convolutional layer to obtain the restored result h. (s) Simultaneously, the reverse restoration result is obtained by preserving the order of each frame in reverse. h (s) and Simultaneous input to three convolutional layers yields a more refined dehazing result.

[0120] In the spatial feature extraction stage, four downsampling modules Block1, Block2, Block3 and Block4 are constructed in the encoder based on ResNet, and four upsampling modules UP1, UP2, UP3 and UP4 are constructed in the decoder using Pixel Shuffle, pixel attention and channel attention.

[0121] The design of the frequency domain constraint network branch involves swapping the high-frequency and low-frequency components of each hazy video frame and the corresponding clear frame. The low-frequency components in the hazy frame contain more hazy background features and less hazy information. Based on this prior image information, we decompose each frame into low-frequency and high-frequency components using two-dimensional discrete wavelet transform, and thus construct a spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition.

[0122] Based on the color domain constrained network branch and the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition, a video dehazing network based on dual constraints in the color domain and frequency domain is constructed. Spatial features are extracted using the color domain constrained network branch, and spatial frequency features are extracted using the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition. To fuse the features extracted from multiple domains, a cross-domain fusion operation is performed, as shown in the following formula:

[0123]

[0124] in A learnable factor whose value is determined by the sigmoid operation on the parameter θ. Decision, and F1 and F2 are the output features of the spatial frequency domain constraint network branch and the color domain constraint network branch based on two-dimensional discrete wavelet decomposition, respectively.

[0125] Loss function constraint unit 3 is used to train the video dehazing network with dual constraints in the color domain and frequency domain using the video haze dataset, and uses the similarity comparison loss function L1 and the perceptual contrast loss function L... c This constrains the network's training to update in a more natural, de-fogging direction;

[0126] Specifically, firstly, the L1 loss function is used to measure the similarity between the dehazed image and the clear sample, as shown in the formula:

[0127]

[0128] Where y pre y gt These represent the predicted fog-free frame output and the true clear label, respectively, where N is the total number of pixels;

[0129] Meanwhile, a perceptual contrast loss function is used to constrain the haze-free image restored by the network to be closer to the clear sample image. The formula for the target loss function is:

[0130]

[0131] Where w is the penalty factor, λ n y is the weighting coefficient. p y f Let represent the distances between positive and negative sample pairs in the metric space, respectively, and the specific formulas are as follows:

[0132]

[0133] Where, θ n For the nth hidden layer feature extracted from the VGG-16 model with fixed pre-trained weights, y i The original haze frame is represented by σ, which is the kernel coefficient used in the loss function.

[0134] In summary, the total loss function is:

[0135] L = aL1 + bL c ,

[0136] Where a and b are weighting coefficients, a = 1, b = 0.04.

[0137] Network fixing and application unit 4, in the network fixing stage, is used to load the trained network weights, input the test dataset based on real haze videos into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames, evaluate the dehazing effect using a no-reference graph quality evaluation index, and fix the parameters when the performance on the evaluation index is good, forming the final video dehazing network (CFVD-Net) with dual constraints in the color domain and frequency domain; in the network application stage, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video;

[0138] Specifically, each video in the test dataset, which consists of real smog videos, is decomposed into several reasonable frames. The size of each frame in the same test dataset is consistent, while the size of each frame in different test datasets is not exactly the same.

[0139] Load the trained network weights and input the test video frame into the video dehazing network with dual constraints in the color domain and frequency domain to obtain the predicted haze-free frame;

[0140] The dehazing effect of each frame is evaluated using no-reference-map quality evaluation metrics such as NIQE, NIMA, and FADE. The parameters of the high-performing network are fixed to obtain the training network weights, forming the final video dehazing network (CFVD-Net) with dual constraints in the color domain and frequency domain.

[0141] In the network application phase, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video.

[0142] Therefore, this invention can remove haze from real-world hazy video scenes. Compared to existing video dehazing methods, it has significant advantages in terms of dehazing quality evaluation and speed. The network components in this invention are interconnected, extracting and integrating different features, thus improving efficiency and making the restored haze-free frames more natural. This provides a foundation for tasks such as mid-level video segmentation and high-level target detection in later applications such as autonomous driving, drone aerial photography, and nearshore marine monitoring.

[0143] The video dehazing method and system based on dual constraints of color domain and frequency domain provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A video dehazing method based on dual constraints of color domain and frequency domain, characterized in that, The method includes: Synthesize a low-light video haze dataset to simulate a real haze scene; Construct a video dehazing network with dual constraints in both the color domain and the frequency domain; The video dehazing network, constrained by both color domain and frequency domain, was trained using the aforementioned video haze dataset, and both the similarity comparison loss function L1 and the perceptual contrast loss function L... c This constrains the network's training to update in a more natural, de-fogging direction; In the network fixing phase, the trained network weights are loaded, and the test dataset based on real haze videos is input into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames. The dehazing effect is evaluated using a no-reference graph quality evaluation index. When the performance on the evaluation index is good, the parameters are fixed to form the final video dehazing network with dual constraints in the color domain and frequency domain. In the network application phase, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video.

2. The video dehazing method based on dual constraints of color domain and frequency domain as described in claim 1, characterized in that, The process of synthesizing a low-light video haze dataset that simulates a real haze scene specifically involves: Following Occam's razor principle and based on Retinex theory, the original atmospheric light scattering model is improved. The improved model formula is as follows: I(x)≈{m α }(Rg(x)α+1-g(x)α), Where m α <m represents the dark light constraint term, Rg(x)α+1-g(x)α represents the haze constraint term, I(x) is the synthesized haze frame, m is the global atmospheric light value, α is a random value less than 1, R is the reflection component information of the object, x is the pixel value, and g=e -βt The transmittance map is given, where β is a random number in the range [0.1, 0.5], and t is the scene depth. Based on this formula, a low-light video haze dataset simulating a real haze scene is synthesized.

3. The video dehazing method based on dual constraints of color domain and frequency domain as described in claim 1, characterized in that, The construction of the video dehazing network with dual constraints in the color domain and frequency domain is specifically as follows: For the design of the color domain constrained network branch, an encoder-decoder network is first constructed to extract high-quality spatial haze-related features, directly extracting high-level spatial features from the input haze frames. Then, these representation features are input into a bidirectional recurrent long short-term memory network. The construction process is as follows: h (s) =Conv(Relu(Conv(o (s) )),tanhC (s) ), Where f (s) This represents the forget gate information of frame s, μ represents the Sigmoid activation function, and w (f) It is the weight matrix from hidden layer to hidden layer in the forget gate, w (i) It is the weight matrix from hidden layer to hidden layer in the input gate, w (c) It is the weight matrix from hidden layer to hidden layer in the cell, w (o) It is the weight matrix from hidden layer to hidden layer in the output gate. and h represents the spatial features extracted from frame s and frame s-1 by the bidirectional recurrent long short-term memory network, respectively. (s) H represents the restored result of frame s. (s-1) This represents the restored result of frame s-1. This represents the reverse reconstruction result of frame s, v (f) It is the bias vector in the forget gate update process, v (i) It is the bias vector during the input gate update process, v (c) It is the bias vector during the cell update process, v (o) It is the bias vector during the output gate update process, i (s) C represents the input gate information of the s-frame. (s) It is the cell information of the s-frame, C (s-1) It is the cell information of frame s-1. This represents the cell information updated in frame s, o (s) This indicates the output gate information of the s-frame. This indicates a more refined dehazing result for frame s, ⊙ represents pixel multiplication, and * represents convolution. and The data is concatenated and input into a bidirectional recurrent long short-term memory network for information updates, and then the input gate is... (s) The information from the updated bidirectional recurrent long short-term memory network is connected and fed into the convolutional layer to obtain the restored result h. (s) Simultaneously, the reverse restoration result is obtained by preserving the order of each frame in reverse. h (s) and Simultaneous input to three convolutional layers yields a more refined dehazing result. In the spatial feature extraction stage, four downsampling modules Block1, Block2, Block3 and Block4 are constructed in the encoder based on ResNet, and four upsampling modules UP1, UP2, UP3 and UP4 are constructed in the decoder using Pixel Shuffle, pixel attention and channel attention. The design of the frequency domain constraint network branch involves swapping the high-frequency and low-frequency components of each hazy video frame and the corresponding clear frame. The low-frequency components in the hazy frame contain more hazy background features and less hazy information. Based on this prior image information, we decompose each frame into low-frequency and high-frequency components using two-dimensional discrete wavelet transform, and thus construct a spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition. Based on the color domain constrained network branch and the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition, a video dehazing network based on dual constraints in the color domain and frequency domain is constructed. Spatial features are extracted using the color domain constrained network branch, and spatial frequency features are extracted using the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition. To fuse the features extracted from multiple domains, a cross-domain fusion operation is performed, as shown in the following formula: in A learnable factor whose value is determined by the parameter-oriented sigmoid operation. Decision, and F1 and F2 are the output features of the spatial frequency domain constraint network branch and the color domain constraint network branch based on two-dimensional discrete wavelet decomposition, respectively.

4. The video dehazing method based on dual constraints of color domain and frequency domain as described in claim 1, characterized in that, The video dehazing network, constrained by both color domain and frequency domain, is trained using the video haze dataset, and a similarity comparison loss function L1 and a perceptual contrast loss function L2 are used. c To constrain the network's training towards a more natural, dehazing-free update process, specifically: First, the L1 loss function is used to measure the similarity between the dehazed image and the clear sample. The formula is as follows: Where y pre y gt These represent the predicted fog-free frame output and the true clear label, respectively, where N is the total number of pixels; Meanwhile, a perceptual contrast loss function is used to constrain the haze-free image restored by the network to be closer to the clear sample image. The formula for the target loss function is: Where w is the penalty factor, λ n y is the weighting coefficient. p y f Let represent the distances between positive and negative sample pairs in the metric space, respectively, and the specific formulas are as follows: in, For the nth hidden layer feature extracted from the VGG-16 model with fixed pre-trained weights, y i The original haze frame is represented by σ, which is the kernel coefficient used in the loss function. In summary, the total loss function is: L=aL1+bL c , Where a and b are weighting coefficients.

5. The video dehazing method based on dual constraints of color domain and frequency domain as described in claim 1, characterized in that, In the network fixing phase, the trained network weights are loaded, and the test dataset based on real hazy videos is input into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames. The dehazing effect is evaluated using a no-reference-map quality evaluation index. When the performance on the evaluation index is good, the parameters are fixed to form the final video dehazing network with dual constraints in the color domain and frequency domain. Specifically: Each video in the test dataset, composed of real smog videos, is divided into several reasonable frames. The size of each frame in the same test dataset is consistent, while the size of each frame in different test datasets is not exactly the same. Load the trained network weights and input the test video frame into the video dehazing network with dual constraints in the color domain and frequency domain to obtain the predicted haze-free frame; The dehazing effect of each frame is evaluated using a no-reference image quality evaluation metric. The parameters of the high-performing network are fixed to obtain the weights of the trained network, forming the final video dehazing network with dual constraints in the color domain and frequency domain.

6. A video dehazing system based on dual constraints of color gamut and frequency domain, characterized in that, The system includes: The Synthetic Haze Dataset Unit is used to synthesize a low-light video haze dataset that simulates a real haze scene. A video dehazing network building unit is used to construct a video dehazing network with dual constraints in the color domain and frequency domain. The loss function constraint unit is used to train the video dehazing network with dual constraints in the color domain and frequency domain using the video haze dataset, and uses the similarity comparison loss function L1 and the perceptual contrast loss function L... c This constrains the network's training to update in a more natural, de-fogging direction; The network fixing and application unit is used in the network fixing stage to load the trained network weights, input the test dataset based on real haze videos into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames, evaluate the dehazing effect using a no-reference graph quality evaluation index, and fix the parameters when the performance on the evaluation index is good to form the final video dehazing network with dual constraints in the color domain and frequency domain; in the network application stage, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video.

7. A video dehazing system based on dual constraints of color gamut and frequency domain as described in claim 6, characterized in that, The synthetic haze dataset unit is used to synthesize a low-light video haze dataset that simulates a real haze scene, specifically: Following Occam's razor principle and based on Retinex theory, the original atmospheric light scattering model is improved. The improved model formula is as follows: I(x)≈{m α }(Rg(x)α+1-g(x)α), Where m α <m represents the dark light constraint term, Rg(x)α+1-g(x)α represents the haze constraint term, I(x) is the synthesized haze frame, m is the global atmospheric light value, α is a random value less than 1, R is the reflection component information of the object, x is the pixel value, and g=e -βt The transmittance map is given, where β is a random number in the range [0.1, 0.5], and t is the scene depth. Based on this formula, a low-light video haze dataset simulating a real haze scene is synthesized.

8. A video dehazing system based on dual constraints of color gamut and frequency domain as described in claim 6, characterized in that, The video dehazing network construction unit is used to construct a video dehazing network with dual constraints in both the color domain and the frequency domain, specifically as follows: For the design of the color domain constrained network branch, an encoder-decoder network is first constructed to extract high-quality spatial haze-related features, directly extracting high-level spatial features from the input haze frames. Then, these representation features are input into a bidirectional recurrent long short-term memory network. The construction process is as follows: h (s) =Conv(Relu(Conv(o (s) )),tanh C (s) ), Where f (s) This represents the forget gate information of frame s, μ represents the Sigmoid activation function, and w (f) It is the weight matrix from hidden layer to hidden layer in the forget gate, w (i) It is the weight matrix from hidden layer to hidden layer in the input gate, w (c) It is the weight matrix from hidden layer to hidden layer in the cell, w (o) It is the weight matrix from hidden layer to hidden layer in the output gate. and h represents the spatial features extracted from frame s and frame s-1 by the bidirectional recurrent long short-term memory network, respectively. (s) H represents the restored result of frame s. (s-1) This represents the restored result of frame s-1. This represents the reverse reconstruction result of frame s, v (f) It is the bias vector in the forget gate update process, v (i) It is the bias vector during the input gate update process, v (c) It is the bias vector during the cell update process, v (o) It is the bias vector during the output gate update process, i (s) C represents the input gate information of the s-frame. (s) It is the cell information of the s-frame, C (s-1) It is the cell information of frame s-1. Indicates updated cell information, o (s) This indicates the output gate information of the s-frame. This indicates a more refined dehazing result for frame s, ⊙ represents pixel multiplication, and * represents convolution. and The data is concatenated and input into a bidirectional recurrent long short-term memory network for information updates, and then the input gate is... (s) The information from the updated bidirectional recurrent long short-term memory network is connected and fed into the convolutional layer to obtain the restored result h. (s) Simultaneously, the reverse restoration result is obtained by preserving the order of each frame in reverse. h (s) and Simultaneous input to three convolutional layers yields a more refined dehazing result. In the spatial feature extraction stage, four downsampling modules Block1, Block2, Block3 and Block4 are constructed in the encoder based on ResNet, and four upsampling modules UP1, UP2, UP3 and UP4 are constructed in the decoder using Pixel Shuffle, pixel attention and channel attention. The design of the frequency domain constraint network branch involves swapping the high-frequency and low-frequency components of each hazy video frame and the corresponding clear frame. The low-frequency components in the hazy frame contain more hazy background features and less hazy information. Based on this prior image information, we decompose each frame into low-frequency and high-frequency components using two-dimensional discrete wavelet transform, and thus construct a spatial frequency domain constraint network branch based on two-dimensional discrete wavelet decomposition. Based on the color domain constrained network branch and the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition, a video dehazing network based on dual constraints in the color domain and frequency domain is constructed. Spatial features are extracted using the color domain constrained network branch, and spatial frequency features are extracted using the spatial frequency domain constrained network branch based on two-dimensional discrete wavelet decomposition. To fuse the features extracted from multiple domains, a cross-domain fusion operation is performed, as shown in the following formula: in A learnable factor whose value is determined by the sigmoid operation on the parameter θ. Decision, and F1 and F2 are the output features of the spatial frequency domain constraint network branch and the color domain constraint network branch based on two-dimensional discrete wavelet decomposition, respectively.

9. A video dehazing system based on dual constraints of color gamut and frequency gamut as described in claim 6, characterized in that, The loss function constraint unit is used to train the video dehazing network with dual constraints in the color domain and frequency domain using the video haze dataset, and uses the similarity comparison loss function L1 and the perceptual contrast loss function L... c To constrain the network's training towards a more natural, dehazing-free update process, specifically: First, the L1 loss function is used to measure the similarity between the dehazed image and the clear sample. The formula is as follows: Where y pre ,y gt These represent the predicted fog-free frame output and the true clear label, respectively, where N is the total number of pixels; Meanwhile, a perceptual contrast loss function is used to constrain the haze-free image restored by the network to be closer to the clear sample image. The formula for the target loss function is: Where w is the penalty factor, λ n y is the weighting coefficient. p ,y f Let represent the distances between positive and negative sample pairs in the metric space, respectively, and the specific formulas are as follows: in, For the nth hidden layer feature extracted from the VGG-16 model with fixed pre-trained weights, y i The original haze frame is represented by σ, which is the kernel coefficient used in the loss function. In summary, the total loss function is: L=aL1+bL c , Where a and b are weighting coefficients.

10. A video dehazing system based on dual constraints of color gamut and frequency gamut as described in claim 6, characterized in that, The network fixing and application unit, in the network fixing phase, loads the trained network weights, inputs a test dataset based on real haze videos into the video dehazing network with dual constraints in the color domain and frequency domain to obtain predicted haze-free frames, evaluates the dehazing effect using a no-reference-map quality evaluation index, and fixes the parameters when the performance on the evaluation index is good, forming the final video dehazing network with dual constraints in the color domain and frequency domain; in the network application phase, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video, specifically: Each video in the test dataset, composed of real smog videos, is divided into several reasonable frames. The size of each frame in the same test dataset is consistent, while the size of each frame in different test datasets is not exactly the same. Load the trained network weights and input the test video frame into the video dehazing network with dual constraints in the color domain and frequency domain to obtain the predicted haze-free frame; The dehazing effect of each frame is evaluated using a no-reference-map quality evaluation metric. The parameters of the high-performing network are fixed to obtain the weights of the trained network, forming the final video dehazing network with dual constraints in the color domain and frequency domain. In the network application phase, the user inputs the haze video to be processed into the final video dehazing network with dual constraints in the color domain and frequency domain to obtain the dehazed video.

Images