Video defogging method and system based on enhanced alignment

Abstract

The invention belongs to the technical field of computer vision and image processing, and particularly discloses a video defogging method and system based on enhanced alignment, and the method comprises the steps: carrying out the color mapping of a plurality of foggy images through a color lookup table, and obtaining a plurality of enhanced images; based on the enhanced image corresponding to the foggy target frame and the enhanced image corresponding to each foggy reference frame, performing space-time optical flow estimation to obtain a three-dimensional optical flow; performing image transformation on the foggy reference frame based on the three-dimensional optical flow to obtain an alignment frame; and based on the alignment frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, performing splicing and feature fusion to obtain a defogged image corresponding to the foggy target frame. According to the method, details of the foggy image are enhanced, a more accurate alignment result is obtained, more inter-frame information is provided through accurate alignment frames, the network can be effectively helped to learn a better defogging capability, the utilization capability of the deep neural network on the inter-frame information can be improved, and a better video defogging effect is achieved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and image processing technology, and more specifically, to a video defogging method and system based on enhanced alignment. Background Art

[0002] Haze is a natural phenomenon caused by the continuous absorption and reflection of light by turbid media in the atmosphere (such as particles and water droplets). Under its influence, the quality of the captured image is reduced, and degradation phenomena such as reduced contrast, color distortion, and unclear details appear, which also makes subsequent image analysis difficult. Therefore, in order to eliminate the haze problem in the image and restore the scene details, image dehazing is an important issue in the field of computer vision.

[0003] In recent years, image dehazing has been widely studied and has achieved good results. However, these methods often have poor results when processing video data because they do not consider the temporal relationship between frames. Therefore, in order to ensure the continuity of video results and improve the effect of video data, the video dehazing task is attracting more and more researchers' attention. Compared with image dehazing, video dehazing can provide more temporal information between frames, which is conducive to improving the dehazing effect.

[0004] With the widespread application of neural networks in the field of computer vision, most of the existing video defogging methods are based on deep neural networks and have made good progress. Compared with traditional methods, deep neural networks can use their powerful feature representation capabilities to learn the characteristics of haze, remove haze of various degrees and restore scene information. Compared with image defogging, a key issue in video defogging is how to effectively utilize inter-frame information. One method is to directly splice the images or features of different frames along the channel dimension and input them into the network. This method does not consider the non-correspondence of spatial position information between different frames. In order to solve this problem, most related methods adopt the idea of ​​alignment-fusion. First, the reference frame and the target frame are aligned in space, and then the alignment result is fused with the target frame using a neural network. Among them, the alignment effect affects the position correspondence between different frames, thereby affecting the network's effective use of the relationship between frames, and then affecting the final defogging effect.

[0005] Existing methods use deformable convolution, spatiotemporal optical flow, or optical flow for image alignment. However, they have the following shortcomings. (1) Haze damages image details, making it difficult to directly align the original image. (2) The depth of the same object in different frames is different, the fog concentration is inconsistent, and the brightness consistency assumption in optical flow estimation is invalid. (3) Although some methods pre-defog the original image, they process each frame independently without considering the consistency between frames, which makes it difficult to estimate the optical flow. Therefore, existing methods have limitations in inter-frame alignment, and a more effective solution needs to be proposed to handle inter-frame alignment. Summary of the invention

[0006] In view of the defects of the prior art, the present application aims to solve the problems of inaccurate frame alignment in the existing video defogging methods and difficulty in the neural network to perceive the relationship between frames, thus affecting the defogging effect.

[0007] To achieve the above objectives, in a first aspect, the present application provides a video defogging method based on enhanced alignment, comprising:

[0008] Color mapping is performed on multiple foggy images through a color lookup table to obtain multiple enhanced images, the multiple foggy images include a foggy target frame and multiple foggy reference frames, the multiple foggy reference frames are continuous frames in time sequence, the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence, and the multiple enhanced images include an enhanced image corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame;

[0009] Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, the spatiotemporal optical flow is estimated to obtain the three-dimensional optical flow;

[0010] Based on the 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame;

[0011] Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame.

[0012] In a possible implementation, the color mapping of the multiple foggy images through a color lookup table to obtain multiple enhanced images includes:

[0013] Perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image;

[0014] Extracting a tag from the enhanced weight, obtaining a tag corresponding to the foggy image, and storing the tag corresponding to the foggy image in a tag memory;

[0015] Input the tiling operation result of the enhanced weight corresponding to the foggy image and the mark in the mark memory to the attention mechanism, and obtain the memory weight output by the attention mechanism;

[0016] Input the enhancement weight and memory weight corresponding to the foggy image to the convolution layer, and obtain the memory enhancement weight output by the convolution layer;

[0017] Through the color lookup table, based on the memory enhancement weight corresponding to the foggy image, the color of the foggy image is enhanced to obtain an enhanced image corresponding to the foggy image.

[0018] In a possible implementation manner, the tag memory stores tags corresponding to each foggy image.

[0019] In a possible implementation, the weight prediction of the foggy image to obtain the enhancement weight corresponding to the foggy image includes:

[0020] Downsample the foggy image to obtain a downsampled foggy image (low-resolution foggy image);

[0021] The weight prediction module is used to predict the enhancement weight corresponding to the downsampled foggy image.

[0022] The enhancement weights corresponding to the downsampled foggy image are upsampled to obtain the enhancement weights corresponding to the foggy image.

[0023] In a possible implementation, there are multiple color lookup tables.

[0024] In a possible implementation, the above-mentioned performing spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain the three-dimensional optical flow includes:

[0025] Performing image transformation on the enhanced image (enhanced reference frame) corresponding to the foggy reference frame using the initial three-dimensional optical flow to obtain an image transformation result;

[0026] Based on the image transformation result, the initial 3D optical flow and the enhanced image corresponding to the foggy target frame (enhanced target frame), splicing is performed along the channel dimension and the spatiotemporal optical flow is estimated to obtain the 3D optical flow.

[0027] In a second aspect, the present application provides a video defogging system based on enhanced alignment, the system comprising: an enhanced alignment network and a fusion network;

[0028] The enhanced alignment network includes a storage lookup table enhancement module and a spatiotemporal optical flow estimator;

[0029] Storage lookup table enhancement module for:

[0030] Color mapping is performed on multiple foggy images through a color lookup table to obtain multiple enhanced images, wherein the multiple foggy images include a foggy target frame and multiple foggy reference frames, wherein the multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence, and the multiple enhanced images include an enhanced image (enhanced frame) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame;

[0031] Spatiotemporal optical flow estimator for:

[0032] Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, the spatiotemporal optical flow is estimated to obtain the three-dimensional optical flow;

[0033] Based on the 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame;

[0034] Converged Network for:

[0035] Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame.

[0036] In a possible implementation, the storage lookup table enhancement module includes: a weight prediction module, a tag generator, an attention mechanism module, a convolutional layer, and a lookup table module;

[0037] The weight prediction module is used to perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image;

[0038] A tag generator is used to extract tags from the enhanced weights to obtain tags corresponding to the foggy image, and store the tags corresponding to the foggy image in a tag storage;

[0039] The attention mechanism module outputs the memory weight corresponding to the foggy image based on the tiling operation result of the enhanced weight corresponding to the foggy image and the mark in the mark memory;

[0040] A convolutional layer is used to output the memory enhancement weight corresponding to the foggy image based on the enhancement weight and memory weight corresponding to the foggy image;

[0041] The lookup table module is used to perform color enhancement on the foggy image through a color lookup table based on the memory enhancement weight corresponding to the foggy image, and obtain an enhanced image corresponding to the foggy image.

[0042] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0044] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0045] (1) The foggy image is enhanced by looking up the table, restoring the image details and enhancing the image contrast, which is beneficial for the neural network to perceive the key information of the image and improve the image alignment effect.

[0046] (2) In order to ensure the color consistency of the enhanced results of different frames, a lookup table-based method is used to enhance the image. The enhancement weight is generated through weight prediction, and the enhanced image is obtained based on the color mapping relationship of the lookup table. By using the lookup table with the same parameters to enhance different frame images, the color consistency between the enhanced frames is ensured, which is conducive to aligning different frame images.

[0047] (3) In order to enhance the continuity of the enhancement results of different frames, a tag memory is set up. The enhancement weight of each frame is marked by a tag generator. The tags of different frames form a tag memory. The attention mechanism is used based on the tag memory and the enhancement weight to generate memory weights. The memory weights are used to optimize the enhancement weights using a convolutional layer to obtain the memory enhancement weights. The tag memory strengthens the association of the enhancement weights used in each frame, improves the continuity of the enhancement results between frames, and is conducive to inter-frame alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a video defogging method based on enhanced alignment provided in an embodiment of the present application;

[0049] Figure 2 Schematic diagram of the architecture of a video defogging system based on enhanced alignment provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of the architecture of the storage lookup table enhancement module provided in an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of the structure of a spatiotemporal optical flow estimator provided in an embodiment of the present application;

[0052] Figure 5 is an image containing fog provided by an embodiment of the present application;

[0053] Figure 6 is a clean image after defogging provided by an embodiment of the present application;

[0054] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0058] First, the technical terms involved in the embodiments of the present application are introduced.

[0059] (1) Weight prediction module: A module composed of a neural network that takes the foggy image as input and outputs the enhancement weight.

[0060] (2) Tag generator: A module composed of a neural network that takes enhancement weights as input and outputs tags. The process of extracting tags from enhancement weights is to filter out features in the enhancement weights that are conducive to maintaining inter-frame consistency, such as the specific weights used by some targets, which represent the inter-frame information in the weight prediction process.

[0061] (3) Flattening operation: An operation performed on an array to expand all elements of a multidimensional array into a row in sequence to obtain a one-dimensional array.

[0062] (4) Color Lookup Table (or LUT): A widely used method for color mapping. The color values ​​of an image are converted through a predefined color mapping relationship. Its basic working principle is: a mapping function is defined for each color channel (usually RGB) of the input image, and this function is stored in a LUT. When processing an image, the color value of each pixel is converted through this LUT to produce the color value of the output image.

[0063] (5) Optical flow predictor: A module composed of a neural network, which is used to predict the 3D optical flow. It takes the enhanced target frame, the initial 3D optical flow, and the result of image transformation as input and splices them along the channel dimension, and outputs the 3D optical flow.

[0064] (6) Image transformation: An operation widely used in the image alignment process. Image transformation usually relies on a transformation matrix that defines how to map each pixel position in the source image to a new position in the target image. Image transformation is the process of remapping the pixels of one image to another through mathematical operations based on the transformation matrix.

[0065] (7) Aligned frame: Use the predicted optical flow to calculate the transformation matrix and transform the reference image.

[0066] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0067] Figure 1 is a flow chart of a video defogging method based on enhanced alignment provided in an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101 to S104.

[0068] Step S101, color mapping is performed on a plurality of foggy images through a color lookup table to obtain a plurality of enhanced images, wherein the plurality of foggy images include a foggy target frame and a plurality of foggy reference frames, wherein the plurality of foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the plurality of foggy reference frames in time sequence, and the plurality of enhanced images include an enhanced image (enhanced frame) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame;

[0069] Step S102, based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, performing spatiotemporal optical flow estimation to obtain a three-dimensional optical flow;

[0070] Step S103, performing image transformation on the foggy reference frame based on the three-dimensional optical flow to obtain an aligned frame;

[0071] Step S104, based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, stitching and feature fusion are performed along the channel dimension to obtain a defogged image corresponding to the foggy target frame.

[0072] It can be understood that enhancing the foggy image according to the lookup table to obtain an enhanced image (restoring the details of the foggy image) can effectively enhance its contrast. Based on the enhanced image, spatiotemporal optical flow estimation can obtain a more accurate three-dimensional optical flow, thereby obtaining a more accurate alignment result. In addition, when enhancing the foggy image based on the lookup table, since the lookup table parameters used in different frames are consistent, the enhancement results of different frames will also maintain a certain consistency, which is conducive to optical flow prediction.

[0073] Therefore, by enhancing the details of foggy images and obtaining more accurate alignment results, accurate aligned frames provide more inter-frame information, which can effectively help the network learn better defogging capabilities and effectively solve the problem of inter-frame alignment of foggy video data in the existing technology. It can enhance the ability of deep neural networks to utilize inter-frame information and achieve better video defogging effects.

[0074] In a possible implementation, the present application provides a video defogging method based on enhanced alignment, comprising the following steps:

[0075] Downsampling is used to obtain a low-resolution foggy image from the foggy target frame as the input of the weight prediction module;

[0076] The weight prediction module is used to predict the low-resolution enhancement weights from the low-resolution foggy image;

[0077] Upsampling is used to obtain enhancement weights from low-resolution enhancement weights as input to the tag generator;

[0078] The tag generator is used to extract tags from the enhanced weights and concatenated with the tag features of other frames to form a tag memory;

[0079] The enhanced weights are tiled and input into the attention mechanism together with the tag memory to obtain the memory weights;

[0080] Concatenate the memory weights and enhancement weights along the channel dimension and input them into the convolution layer to obtain the memory enhancement weights;

[0081] Using memory enhancement weights, according to the lookup table, the foggy image is enhanced to obtain an enhanced image;

[0082] The image is transformed using the enhanced reference frame and the initial 3D optical flow, and the transformed result is concatenated with the enhanced target frame and the initial 3D optical flow and input into the optical flow predictor to obtain the 3D optical flow;

[0083] Using 3D optical flow, the foggy reference frame is transformed to obtain an aligned frame.

[0084] The aligned frame is concatenated with the foggy target frame and the enhanced target frame along the channel dimension and input into the fusion network to obtain the dehazed image.

[0085] It can be understood that in order to further enhance the consistency of enhancement results between different frames, a storage mechanism is adopted; first, the downsampling operation, weight prediction and upsampling operation are used in sequence to obtain the enhancement weight, and then the tag generator is used to obtain the tag, so as to construct a tag memory, and the attention mechanism is used to obtain the memory weight, and the convolution layer is used to obtain the memory enhancement weight, so that the weights used in different frame enhancements remain relevant, which is conducive to maintaining the consistency of the enhancement results and obtaining better inter-frame alignment results.

[0086] Optionally, multiple learnable color lookup tables can be used to perform color mapping on the foggy image using memory-enhanced weights to obtain an enhanced image.

[0087] The video defogging system based on enhanced alignment provided by this application is described below.

[0088] like Figure 2As shown in the figure, the whole architecture consists of two parts: an enhanced alignment network and a fusion network; the enhanced alignment network is a network that performs alignment by predicting optical flow, which mainly includes a storage lookup table enhancement module and a spatiotemporal optical flow estimator. The storage lookup table enhancement module obtains enhanced frames from foggy frames, and the spatiotemporal optical flow estimator estimates the three-dimensional optical flow from the enhanced frames, and obtains aligned frames through image transformation operations; the fusion network is the main network that recovers clean defogged images from foggy images. Figure 2 The subscript t represents the time corresponding to the foggy target frame.

[0089] Specifically, a storage lookup table enhancement module is used to: perform color mapping on multiple foggy images through a color lookup table to obtain multiple enhanced images, the multiple foggy images include a foggy target frame and multiple foggy reference frames, the multiple foggy reference frames are continuous frames in time sequence, the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence, and the multiple enhanced images include enhanced images (enhanced frames) corresponding to the foggy target frames and enhanced images corresponding to each foggy reference frame.

[0090] The spatiotemporal optical flow estimator is used to: perform spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain a three-dimensional optical flow; based on the three-dimensional optical flow, perform image transformation on the foggy reference frame to obtain an aligned frame.

[0091] The fusion network is used to: perform splicing and feature fusion along the channel dimension based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, so as to obtain the defogged image corresponding to the foggy target frame.

[0092] The structure of the storage lookup table enhancement module is as follows Figure 3 As shown in Figure 1, it consists of five parts: weight prediction module, tag generator, attention mechanism, convolutional layer and lookup table. Figure 3 The parameters involved in the description are as follows: I represents the input foggy image, ω represents the enhancement weight, τ represents the tag generated by the tag generator, Q represents the query value used in the attention mechanism, K represents the key value used in the attention mechanism, V represents the true value used in the attention mechanism, ω represents the value of the attention mechanism, M represents the memory weight, ω ME represents the memory enhancement weight, represents the output enhanced image, Represents the scale of each parameter, H represents the height of the image, W represents the length of the image, C represents the number of channels for enhancing the weight, D represents the length of the tag, and N represents the number of tags stored in the tag memory.

[0093] The weight prediction module estimates the low-resolution enhancement weights from the downsampled foggy image, and then uses an upsampling operation to obtain the enhancement weights;

[0094] The tag generator generates corresponding tags from the enhanced weights, and the tags of multiple frames of images form a tag memory;

[0095] The attention mechanism uses the tag memory and the tiled enhanced weights to calculate the attention information and obtain the memory enhanced weights;

[0096] The convolutional layer uses the enhancement weights and memory weights concatenated along the channel dimension as input to obtain the memory enhancement weights;

[0097] The lookup table uses memory enhancement weights to perform color mapping on the foggy image to obtain an enhanced image.

[0098] The structure of the spatiotemporal optical flow estimator is as follows Figure 4 As shown, it mainly includes an optical flow predictor. The spatiotemporal optical flow estimator first uses the initial 3D optical flow to transform the enhanced reference frame, and splices the transformed result with the enhanced target frame and the initial 3D optical flow along the channel dimension. The optical flow predictor uses the splicing result as input to obtain the 3D optical flow.

[0099] At the same time, in order to further verify the effectiveness and robustness of this application, the video defogging model of this application is trained using a publicly synthesized data set, and the trained video defogging model is used to remove haze from foggy images, and the following results are obtained: Figure 5 and Figure 6 The effect diagram shown.

[0100] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0101] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0102] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (CommunicationsInterface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0103] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0104] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0105] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0106] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0107] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0109] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0110] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A video defogging method based on enhanced alignment, characterized in that: include: Color mapping is performed on multiple foggy images through a color lookup table to obtain multiple enhanced images, the multiple foggy images include a foggy target frame and multiple foggy reference frames, the multiple foggy reference frames are continuous frames in time sequence, the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence, and the multiple enhanced images include an enhanced image corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame; Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, the spatiotemporal optical flow is estimated to obtain the three-dimensional optical flow; Based on the 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame; Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame.

2. The video defogging method based on enhanced alignment according to claim 1, characterized in that: The method of performing color mapping on the multiple foggy images through a color lookup table to obtain multiple enhanced images includes: Perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image; Extracting a tag from the enhanced weight, obtaining a tag corresponding to the foggy image, and storing the tag corresponding to the foggy image in a tag memory; Input the tiling operation result of the enhanced weight corresponding to the foggy image and the mark in the mark memory to the attention mechanism, and obtain the memory weight output by the attention mechanism; Input the enhancement weight and memory weight corresponding to the foggy image to the convolution layer, and obtain the memory enhancement weight output by the convolution layer; Through the color lookup table, based on the memory enhancement weight corresponding to the foggy image, the color of the foggy image is enhanced to obtain an enhanced image corresponding to the foggy image.

3. The video defogging method based on enhanced alignment according to claim 2, characterized in that: The tag storage stores tags corresponding to each foggy image.

4. The video defogging method based on enhanced alignment according to claim 2, characterized in that: The step of performing weight prediction on the foggy image to obtain an enhancement weight corresponding to the foggy image includes: Down-sampling the foggy image to obtain a down-sampled foggy image; The weight prediction module is used to predict the enhancement weight corresponding to the downsampled foggy image. The enhancement weights corresponding to the downsampled foggy image are upsampled to obtain the enhancement weights corresponding to the foggy image.

5. The video defogging method based on enhanced alignment according to claim 1, characterized in that: The number of the color lookup tables is multiple.

6. The video defogging method based on enhanced alignment according to any one of claims 1 to 5, characterized in that: The method of performing spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain the three-dimensional optical flow includes: Perform image transformation on the enhanced image corresponding to the foggy reference frame using the initial three-dimensional optical flow to obtain an image transformation result; Based on the image transformation results, the initial 3D optical flow and the enhanced image corresponding to the foggy target frame, they are spliced ​​along the channel dimension and the spatiotemporal optical flow is estimated to obtain the 3D optical flow.

7. A video defogging system based on enhanced alignment, characterized in that: include: Enhanced alignment and fusion networks; The enhanced alignment network includes a storage lookup table enhancement module and a spatiotemporal optical flow estimator; The storage lookup table enhancement module is used to: Color mapping is performed on multiple foggy images through a color lookup table to obtain multiple enhanced images, wherein the multiple foggy images include a foggy target frame and multiple foggy reference frames, wherein the multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence, and the multiple enhanced images include an enhanced image (enhanced frame) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame; The spatiotemporal optical flow estimator is used to: Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, the spatiotemporal optical flow is estimated to obtain the three-dimensional optical flow; Based on the 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame; The fusion network is used for: Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame.

8. The video defogging system based on enhanced alignment according to claim 7, characterized in that: The storage lookup table enhancement module includes: a weight prediction module, a tag generator, an attention mechanism module, a convolutional layer and a lookup table module; The weight prediction module is used to perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image; The tag generator is used to extract tags from the enhanced weights to obtain tags corresponding to the foggy image, and store the tags corresponding to the foggy image in a tag storage; The attention mechanism module is used to output the memory weight corresponding to the foggy image based on the tiling operation result of the enhanced weight corresponding to the foggy image and the mark in the mark memory; The convolution layer is used to output the memory enhancement weight corresponding to the foggy image based on the enhancement weight and memory weight corresponding to the foggy image; The lookup table module is used to perform color enhancement on the foggy image through a color lookup table based on a memory enhancement weight corresponding to the foggy image, and obtain an enhanced image corresponding to the foggy image.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images