Video defogging model and method based on memory network fusion of phase features

Abstract

This invention discloses a video dehazing model and method based on memory network fusion of phase features, comprising the following steps: Step 1, extracting color query features and phase query features of the video frame to be dehazed using a color query encoder and a phase query encoder; Step 2, extracting color memory features and phase memory features of the past n frames using a color memory encoder and a phase memory encoder; Step 3, fusing the color query features and phase query features extracted in Step 1 with the color memory features and phase memory features extracted in Step 2 using a cross-modal memory reading module; Step 4, inputting the color query features of the first three layers of the current frame obtained in Step 1 and the fused features obtained in Step 3 into the decoder for result prediction. This invention improves video dehazing performance and obtains high-definition images by fusing frequency domain information and color domain information.

Description

Technical Field

[0001] This invention belongs to the research field of computer vision and image processing, specifically involving a video dehazing model and method based on memory network fusion of phase features. Background Technology

[0002] The purpose of image dehazing is to restore the contrast and true colors of a hazy image, reproducing a clear image taken under ideal weather conditions. Image / video dehazing is of great significance to various tasks in the field of computer vision, such as image understanding, object recognition, object tracking, and intelligent navigation. Existing image dehazing methods include:

[0003] Atmospheric scattering model: Atmospheric scattering model [1] It is an important reference model in the field of defogging research, and its formula is as follows:

[0004] I(x)=J(x)t(x)+A(1-t(x)),

[0005] Where I represents a foggy image, J represents a fog-free image, A represents atmospheric scattered light, and t represents transmittance. Furthermore, t can be expressed as follows:

[0006] t(x)=e -βd(x) ,

[0007] Where β and d represent the atmospheric scattering parameters and the scene depth of the image, respectively. In dehazing research, given a foggy image I, there are two main methods to obtain a clean image J: (1) Solve for A and t respectively, and predict J according to the atmospheric scattering model. (2) Directly construct the mapping relationship between I and J, and obtain J from I.

[0008] The second approach involves using deep learning methods to construct a correspondence between the input foggy video and the output result, thereby completing the video dehazing.

[0009] Memory networks: Memory networks were first used in video object segmentation tasks. [2] It includes a query encoder for extracting features of the current frame and a memory encoder for extracting features of the past n frames, and uses a spatiotemporal memory feature reading module to fuse the features of the query encoder and the memory encoder for target segmentation.

[0010] References

[0011] [1]Shree K Nayar and Srinivasa G Narasimhan.1999.Vision in badweather.In ICCV.820-827.

[0012] [2] SeoungWug Oh, Joon-Young Lee, Ning Xu, and SeonJoo Kim. 2019. VideoObject Segmentation Using Space-Time Memory Networks. In ICCV. Summary of the Invention

[0013] This invention addresses the technical problem that existing image / video dehazing algorithms neglect the contribution of phase information to the dehazing task. It proposes a video dehazing method based on memory networks, which further improves video dehazing performance by fusing frequency domain information (phase) and color domain information (RGB image).

[0014] To address the shortcomings of existing technologies, the present invention adopts the following technical solution:

[0015] A video dehazing model based on phase feature fusion using a memory network, the video dehazing model comprising: a query encoding module, a memory encoding module, a cross-modal memory retrieval module, and a decoder; wherein:

[0016] The query encoding module consists of a color query encoder and a phase query encoder; the query encoding module is used to extract color query features and phase query features from the current video frame to be dehazed to generate first feature information.

[0017] The memory encoding module consists of a color memory encoder and a phase memory encoder; the memory encoding module extracts color memory features and phase memory features from the past n frames of foggy video to generate second feature information;

[0018] The cross-modal memory reading module calculates and generates fused feature information by combining the first feature information generated by the query encoding module and the second feature information generated by the memory encoding module.

[0019] The decoder verifies the color query features and fusion feature information and outputs a dehazed video image.

[0020] Furthermore, the query encoding module is used to extract color query features and phase query features from the current video frame to be dehazed to generate the first feature information;

[0021] 1.1 Input the current video frame It to be dehazed into the color query encoder to obtain the color query features of the current frame;

[0022] 1.2 Apply Fourier transform to the current video frame It to be dehazed to obtain the corresponding phase map Pt, and send it to the phase query encoder to obtain the phase query feature.

[0023] Furthermore, the memory encoding module extracts color memory features and phase memory features from the past n frames of foggy video to generate second feature information:

[0024] 2.1 Input the past n frames of foggy video and the corresponding fog-free video into the color memory encoder to obtain the color memory features of the past n frames;

[0025] 2.2 Apply Fourier transform to the past n frames of foggy video to obtain the corresponding phase map, and send it to the phase memory encoder to obtain phase memory features.

[0026] Furthermore, the cross-modal memory reading module calculates and generates fused feature information by combining the first feature information generated by the query encoding module and the second feature information generated by the memory encoding module:

[0027] 3.1 Obtain the key and value of the color memory feature, as well as the key and value of the color query feature, through the KeyValue operation;

[0028] 3.2. Use the key of the color memory feature and the key of the color query feature as input to the similarity function to calculate the similarity between them;

[0029] 3.3 Obtain the key of the phase memory feature and the key of the phase query feature through the Value operation, and calculate their similarity;

[0030] 3.4. Use the key of the color memory feature and the key of the phase query feature as inputs to the similarity function to calculate the similarity between them;

[0031] 3.5. Use the key of the color query feature and the key of the phase memory feature as input to the similarity function to calculate the similarity between them;

[0032] 3.6. Add up all the similarities and fuse them with the color memory features.

[0033] Furthermore, the decoder verifies the color query features and fusion feature information to output the dehazed video image:

[0034] 4.1 Input the color feature encoding of the first three layers of the current frame and the obtained fused features into the decoder; Apply four convolutional layers to the four inputs of different resolutions respectively, reducing their dimensions from 2c1, c2 and c3 to the minimum dimension c4;

[0035] 4.2 Feature enhancement using a series of Residual Channel Attention Modules (RCABs);

[0036] 4.3. Use different scaling factors to upsample augmented features with different resolutions to the same resolution to prepare for cascading;

[0037] 4.4. Feed the enhanced features into a 1x1 convolutional layer to predict the final result.

[0038] To address the technical problems existing in the prior art, the present invention can also adopt the following technical solution:

[0039] A method for image dehazing using a video dehazing model and memory network fusion of phase features includes the following steps:

[0040] Step 1: The query encoder module extracts the color query features and phase query features of the video frame to be dehazed using the color query encoder and phase query encoder.

[0041] Step 2: The memory encoder module extracts the color memory features and phase memory features of the past n frames through the color memory encoder and phase memory encoder;

[0042] Step 3: The cross-modal memory reading module fuses the color query features and phase query features extracted in Step 1 with the color memory features and phase memory features extracted in Step 2;

[0043] Step 4: Input the color query features of the first three layers of the current frame obtained in Step 1 and the fusion features obtained in Step 3 into the decoder to predict the fog-free image.

[0044] Furthermore, the step one of querying the encoder module includes:

[0045] 1.1. The current video frame to be dehazed (I) t Inputting the color query encoder yields the dimension. Color query features

[0046] 1.2. The current video frame to be dehazed (I) t The corresponding phase diagram P is obtained by applying Fourier transform. t It is then sent to the phase query encoder to obtain a resolution of Phase query features

[0047] Furthermore, the memory encoder module in step two includes:

[0048] 2.1. Take the past n frames of foggy video {I} t-n ,……I t-1} and the corresponding fog-free video frame {G t-n ,……G t-1The input is fed into the color memory encoder to obtain the color memory features of the past n frames. resolution is

[0049] 2.2. Take the past n frames of foggy video {I} t-n ,……I t-1 Applying Fourier transform to obtain the corresponding phase diagram {P} t-n ,……P t-1} and then send it to the phase memory encoder to obtain phase memory features. Its resolution is

[0050] Furthermore, the cross-modal memory retrieval module in step three includes:

[0051] 3.1 Obtaining color memory features through KeyValue operations key and value and color query features key and value The specific process can be represented by the following formula:

[0052]

[0053]

[0054] 3.2 Incorporating color memory characteristics and color query features As input to the similarity function, to calculate and The similarity among all pixels can be expressed by the following formula:

[0055]

[0056] 3.3 Obtaining phase memory features through Value operations key and phase query features key And calculate their similarity, the specific process of which can be expressed by the following formula:

[0057]

[0058]

[0059]

[0060] 3.4. The key of color memory features and phase query feature key As input to the similarity function to calculate the similarity between them, the specific process can be expressed by the following formula:

[0061]

[0062] 3.5. Use the color query feature key and phase memory features key As input to the similarity function to calculate the similarity between them, the specific process can be expressed by the following formula:

[0063]

[0064] 3.6. Add up all the similarities and fuse them with the color memory features. Optimized features It can be represented as:

[0065]

[0066] Furthermore, the decoder in step four includes:

[0067] 4.1. Input the color feature encoding of the first three layers of the current frame obtained in step one and the fused features obtained in step three into the decoder. Apply four convolutional layers to the four inputs at different resolutions respectively, reducing their dimensions from 2c1, c2, and c3 to the minimum dimension c4;

[0068] 4.2 Feature enhancement using a series of Residual Channel Attention Modules (RCABs);

[0069] 4.3. Use different scaling factors to upsample augmented features with different resolutions to the same resolution to prepare for cascading;

[0070] 4.4. Feed the enhanced features into a 1x1 convolutional layer to predict the final result Jt.

[0071] Beneficial effects

[0072] a) This invention experimentally verified that the phase information in the image contains less fog information, therefore it is feasible to use phase information to guide dehazing.

[0073] b) This invention proposes a novel memory network that further improves video dehazing performance by fusing color and phase information from the current frame and consecutive past frames.

[0074] c) Experimental results on benchmark datasets of real dehazed videos and collected synthetic datasets show that the method proposed in this invention is significantly superior to state-of-the-art image and video dehazing methods. Attached Figure Description

[0075] Figure 1 This is a structural diagram of the cross-modal memory read module.

[0076] Figure 2 This is a diagram of the decoder structure.

[0077] Figure 3 This is a demonstration of the results on the real dataset REVIDE.

[0078] Figure 4 For quantitative comparisons on the real dataset REVIDE. Detailed Implementation

[0079] The following will refer to the appendix. Figure 1-4 The features, advantages, and technical effects of exemplary embodiments of the present invention are described below.

[0080] Experimental analysis reveals that the phase information in images contains relatively little fog information. Based on this conclusion, this invention proposes a video dehazing method based on memory network fusion of phase features. This method further improves video dehazing performance by fusing frequency domain information (phase) and color domain information (RGB image). This invention uses a video target segmentation memory network as the backbone network, and adds a phase query encoder and a phase memory encoder for phase feature extraction. Furthermore, this invention employs a cross-modal memory retrieval module to fuse features from different modalities, and then feeds these features into the decoder for feature decoding and prediction of fog-free images.

[0081] This invention provides a video dehazing method based on memory network fusion of phase features, comprising the following steps:

[0082] Step 1: Extract the color query features and phase query features of the video frames to be dehazed using a color query encoder and a phase query encoder; specifically, the query encoder module includes the following methods:

[0083] 1.1. The current video frame to be dehazed (I) t Inputting the color query encoder yields the dimension. Color query features

[0084] 1.2. The current video frame to be dehazed (I) t The corresponding phase diagram P is obtained by applying Fourier transform. t It is then sent to the phase query encoder to obtain a resolution of Phase query features

[0085] Step 2: Extract color memory features and phase memory features from the past n frames using a color memory encoder and a phase memory encoder; a memory encoder module was used, and the specific methods include:

[0086] 2.1. Take the past n frames of foggy video {I} t-n ,……I t-1} and the corresponding fog-free video frame {G t-n ,……G t-1 The input is fed into the color memory encoder to obtain the color memory features of the past n frames. resolution is

[0087] 2.2. Take the past n frames of foggy video {I} t-n ,……I t-1 Applying Fourier transform to obtain the corresponding phase diagram {P} t-n ,……P t-1} and then send it to the phase memory encoder to obtain phase memory features. Its resolution is

[0088] Step 3: The color query features and phase query features extracted in Step 1 are fused with the color memory features and phase memory features extracted in Step 2 using a cross-modal memory reading module; for example... Figure 3 As shown, the cross-modal memory read module is implemented through the following steps:

[0089] 3.1 Obtaining color memory features through KeyValue operations key and value and color query features key and value The specific process can be represented by the following formula:

[0090]

[0091]

[0092] 3.2 Incorporating color memory characteristics and color query features As input to the similarity function, to calculate and The similarity among all pixels can be expressed by the following formula:

[0093]

[0094] 3.3 Obtaining phase memory features through Value operations key and phase query features key And calculate their similarity, the specific process of which can be expressed by the following formula:

[0095]

[0096]

[0097]

[0098] 3.4. The key of color memory features and phase query feature key As input to the similarity function to calculate the similarity between them, the specific process can be expressed by the following formula:

[0099]

[0100] 3.5. Use the color query feature key and phase memory features key As input to the similarity function to calculate the similarity between them, the specific process can be expressed by the following formula:

[0101]

[0102] 3.6. Add up all the similarities and fuse them with the color memory features. Optimized features It can be represented as:

[0103]

[0104] Step 4: Input the color query features of the first three layers of the current frame obtained in Step 1 and the fusion features obtained in Step 3 into the decoder for result prediction. The decoder is implemented through the following steps:

[0105] 4.1 Input the color features of the first three layers of the current frame obtained in step one and the fused features obtained in step three into the decoder. Apply four convolutional layers to the four inputs at different resolutions respectively, reducing their dimensions from 2c1, c2, and c3 to the minimum dimension c4;

[0106] 4.2 Feature enhancement using a series of Residual Channel Attention Modules (RCABs);

[0107] 4.3. Use different scaling factors to upsample augmented features with different resolutions to the same resolution to prepare for cascading;

[0108] 4.4. Feed the enhanced features into a 1x1 convolutional layer to predict the final result Jt.

Claims

1. A video defogging method based on a memory network fusion phase feature, characterized in that, Includes the following steps: Step 1: The query encoder module extracts the color query features and phase query features of the video frame to be dehazed using the color query encoder and phase query encoder. Step 2: The memory encoder module extracts the color memory features and phase memory features of the past n frames through the color memory encoder and phase memory encoder; Step 3: The cross-modal memory reading module fuses the color query features and phase query features extracted in Step 1 with the color memory features and phase memory features extracted in Step 2; Step 4: Input the color query features of the first three layers of the current frame obtained in Step 1 and the fusion features obtained in Step 3 into the decoder for prediction of the haze-free image. The method is based on a video dehazing model, which includes: a query encoding module, a memory encoding module, a cross-modal memory retrieval module, and a decoder. The query encoding module consists of a color query encoder and a phase query encoder; the query encoding module is used to extract color query features and phase query features from the current video frame to be dehazed to generate first feature information; The memory encoding module consists of a color memory encoder and a phase memory encoder; the memory encoding module extracts color memory features and phase memory features from the past n frames of foggy video to generate second feature information; The cross-modal memory reading module calculates and generates fused feature information by combining the first feature information generated by the query encoding module and the second feature information generated by the memory encoding module. The decoder verifies the color query features and fusion feature information and outputs a dehazed video image.

2. The video defogging method based on memory network fusion phase features according to claim 1, characterized in that, The query encoding module is used to extract color query features and phase query features from the current video frame to be dehazed to generate the first feature information. S1.1 inputting a current video frame to be defogged to a color query encoder to obtain color query features of the current frame; S1.2 will select the current video frame to be dehazed. The corresponding phase diagram is obtained by applying Fourier transform. It is then sent to the phase query encoder to obtain the phase query features.

3. The video defogging method based on memory network fusion phase features according to claim 1, characterized in that, The memory encoding module extracts color memory features and phase memory features from the past n frames of foggy video to generate second feature information: S2.1 Input the past n frames of foggy video and the corresponding fog-free video into the color memory encoder to obtain the color memory features of the past n frames; S2.2 applies Fourier transform to the past n frames of foggy video to obtain the corresponding phase map, and sends it to the phase memory encoder to obtain phase memory features.

4. The video defogging method based on memory network fusion phase features according to claim 1, characterized in that, The cross-modal memory reading module calculates and generates fused feature information by combining the first feature information generated by the query encoding module and the second feature information generated by the memory encoding module. S3.1 Obtain the key and value of the color memory feature, as well as the key and value of the color query feature, through the KeyValue operation; S3.2 The key of the color memory feature and the key of the color query feature are used as inputs to the similarity function to calculate the similarity between them; S3.3 Obtain the key of the phase memory feature and the key of the phase query feature through the Value operation, and calculate their similarity; S3.4 The key of the color memory feature and the key of the phase query feature are used as inputs to the similarity function to calculate the similarity between them; S3.5 uses the key of the color query feature and the key of the phase memory feature as inputs to the similarity function to calculate the similarity between them; S3.6 Adds all similarities together and merges them with color memory features.

5. The video dehazing method based on memory network fusion phase features according to claim 1, characterized in that, The process by which the decoder verifies the color query features and fusion feature information and outputs the dehazed video image: S4.1 Input the color feature encoding of the first three layers of the current frame and the obtained fused features into the decoder; apply four convolutional layers to the four inputs of different resolutions respectively, reducing their dimensions from 2c1, c2 and c3 to the minimum dimension c4; S4.2 utilizes a series of residual channel attention modules for feature enhancement; S4.3 uses different scaling factors to upsample augmented features with different resolutions to the same resolution, preparing for cascading; S4.4 feeds the enhanced features into a 1x1 convolutional layer to predict the final result.

Images