A video defogging method and system based on enhanced alignment

Abstract

The application belongs to the technical field of computer vision and image processing, and specifically discloses a video defogging method and system based on enhanced alignment, which comprises the following steps: color mapping is performed on multiple foggy images through a color lookup table to obtain multiple enhanced images; based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, spatiotemporal optical flow estimation is performed to obtain three-dimensional optical flow; based on the three-dimensional optical flow, image transformation is performed on the foggy reference frame to obtain an aligned frame; based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed to obtain a defogged image corresponding to the foggy target frame. The application enhances the details of the foggy image, obtains more accurate alignment results, provides more interframe information with the accurate aligned frame, effectively helps the network to learn better defogging capability, improves the utilization capability of the deep neural network for interframe information, and achieves better video defogging effect.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and image processing technology, and more specifically, relates to a video defogging method and system based on enhanced alignment. Background Art

[0002] Haze is a natural phenomenon caused by the continuous absorption and reflection of light by turbid media in the atmosphere, such as particulate matter and water droplets. This degrades image quality, resulting in reduced contrast, color distortion, and blurred details. This also complicates subsequent image analysis. Therefore, image dehazing is a key issue in computer vision, aiming to eliminate haze and restore scene details.

[0003] Image dehazing has been widely researched in recent years, achieving promising results. However, these methods often perform poorly when processing video data because they fail to consider the temporal relationships between frames. Therefore, to ensure the continuity of video results and improve the performance of video data, video dehazing is attracting increasing research attention. Compared to image dehazing, video dehazing provides more temporal information between frames, which is beneficial for improving dehazing performance.

[0004] With the widespread application of neural networks in computer vision, most existing video dehazing methods are based on deep neural networks and have made significant progress. Compared to traditional methods, deep neural networks can leverage their powerful feature representation capabilities to learn the characteristics of haze, remove various levels of haze, and restore scene information. Compared to image dehazing, a key challenge in video dehazing lies in how to effectively utilize inter-frame information. One approach is to directly concatenate images or features from different frames along the channel dimension and input them into the network. This approach does not consider the spatial positional mismatches between frames. To address this issue, most related methods adopt an alignment-fusion approach. First, the reference frame and the target frame are spatially aligned, and then the alignment result is fused with the target frame using a neural network. However, the alignment effect affects the positional correspondence between frames, which in turn affects the network's ability to effectively utilize inter-frame relationships, and thus the final dehazing performance.

[0005] Existing methods use deformable convolution, spatiotemporal optical flow, or optical flow for image alignment. However, they have the following shortcomings. (1) Haze damages image details, making it difficult to directly align the original image. (2) The depth of the same object in different frames is different, the fog concentration is inconsistent, and the brightness consistency assumption in optical flow estimation fails. (3) Although some methods perform pre-dehazing on the original image, they process each frame independently without considering the consistency between frames, which makes it difficult to estimate optical flow. Therefore, existing methods have limitations in inter-frame alignment, and a more effective solution is needed to handle inter-frame alignment. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, this application aims to solve the problems of inaccurate inter-frame alignment in existing video defogging methods and the difficulty of neural networks in perceiving inter-frame relationships, which affects the defogging effect.

[0007] To achieve the above objectives, in a first aspect, the present application provides a video defogging method based on enhanced alignment, comprising:

[0008] Color mapping is performed on the multiple foggy images using a color lookup table to obtain multiple enhanced images, where the multiple foggy images include a foggy target frame and multiple foggy reference frames. The multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence. The multiple enhanced images include an enhanced image corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame.

[0009] Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, spatiotemporal optical flow estimation is performed to obtain three-dimensional optical flow;

[0010] Based on 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame;

[0011] Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame.

[0012] In one possible implementation, performing color mapping on multiple foggy images using a color lookup table to obtain multiple enhanced images includes:

[0013] Perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image;

[0014] Extracting a tag from the enhanced weight to obtain a tag corresponding to the foggy image, and storing the tag corresponding to the foggy image in a tag memory;

[0015] Input the tiling operation result of the enhanced weight corresponding to the foggy image and the mark in the mark memory to the attention mechanism, and obtain the memory weight output by the attention mechanism;

[0016] Input the enhancement weight and memory weight corresponding to the foggy image to the convolution layer, and obtain the memory enhancement weight output by the convolution layer;

[0017] Through the color lookup table, based on the memory enhancement weight corresponding to the foggy image, the color of the foggy image is enhanced to obtain the enhanced image corresponding to the foggy image.

[0018] In a possible implementation, the tag memory stores tags corresponding to each foggy image.

[0019] In one possible implementation, the weight prediction for the foggy image and obtaining the enhancement weight corresponding to the foggy image include:

[0020] Downsample the foggy image to obtain a downsampled foggy image (low-resolution foggy image);

[0021] The weight prediction module is used to predict the enhancement weight corresponding to the downsampled foggy image.

[0022] The enhancement weight corresponding to the downsampled foggy image is upsampled to obtain the enhancement weight corresponding to the foggy image.

[0023] In a possible implementation, there are multiple color lookup tables.

[0024] In one possible implementation, performing spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain three-dimensional optical flow includes:

[0025] Performing image transformation on the enhanced image (enhanced reference frame) corresponding to the foggy reference frame using the initial three-dimensional optical flow to obtain an image transformation result;

[0026] Based on the image transformation results, the initial 3D optical flow and the enhanced image corresponding to the foggy target frame (enhanced target frame), splicing is performed along the channel dimension and spatiotemporal optical flow estimation is performed to obtain the 3D optical flow.

[0027] In a second aspect, the present application provides a video dehazing system based on enhanced alignment, the system comprising: an enhanced alignment network and a fusion network;

[0028] The enhanced alignment network includes a storage lookup table enhancement module and a spatiotemporal optical flow estimator;

[0029] Storage lookup table enhancement module for:

[0030] Color mapping is performed on the multiple foggy images using a color lookup table to obtain multiple enhanced images, where the multiple foggy images include a foggy target frame and multiple foggy reference frames. The multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence. The multiple enhanced images include an enhanced image (enhanced frame) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame.

[0031] Spatiotemporal optical flow estimator for:

[0032] Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, spatiotemporal optical flow estimation is performed to obtain three-dimensional optical flow;

[0033] Based on 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame;

[0034] Converged Network for:

[0035] Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame.

[0036] In one possible implementation, the storage lookup table enhancement module includes: a weight prediction module, a token generator, an attention mechanism module, a convolutional layer, and a lookup table module;

[0037] The weight prediction module is used to perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image;

[0038] a tag generator, configured to extract tags from the enhanced weights, obtain tags corresponding to the foggy image, and store the tags corresponding to the foggy image in a tag memory;

[0039] The attention mechanism module outputs the memory weight corresponding to the foggy image based on the tiling operation results of the enhanced weight corresponding to the foggy image and the marks in the mark memory;

[0040] The convolution layer is used to output the memory enhancement weight corresponding to the foggy image based on the enhancement weight and memory weight corresponding to the foggy image;

[0041] The lookup table module is used to perform color enhancement on the foggy image based on the memory enhancement weight corresponding to the foggy image through the color lookup table to obtain an enhanced image corresponding to the foggy image.

[0042] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0044] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the existing technologies:

[0045] (1) The foggy image is enhanced by looking up the table, image details are restored, and image contrast is enhanced, which is beneficial for the neural network to perceive the key information of the image and improve the image alignment effect.

[0046] (2) To ensure color consistency across enhanced frames, a lookup table-based approach is used to enhance the image. Enhancement weights are generated through weight prediction, and enhanced images are obtained based on the color mapping relationship in the lookup table. By using the same lookup table parameters to enhance different frames, color consistency between enhanced frames is ensured, which facilitates alignment of different frames.

[0047] (3) To enhance the continuity of enhancement results across different frames, a tag memory is set up. The enhancement weights of each frame are tagged by a tag generator. The tags of different frames form a tag memory. Based on the tag memory and the enhancement weights, an attention mechanism is used to generate memory weights. The memory weights are then used to optimize the enhancement weights using a convolutional layer to obtain the memory enhancement weights. The tag memory strengthens the association between the enhancement weights used in each frame, improves the continuity of the enhancement results across frames, and facilitates inter-frame alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flow chart of a video defogging method based on enhanced alignment provided in an embodiment of the present application;

[0049] Figure 2 Schematic diagram of the architecture of the video defogging system based on enhanced alignment provided in an embodiment of the present application;

[0050] Figure 3 Schematic diagram of the architecture of the storage lookup table enhancement module provided in an embodiment of the present application;

[0051] Figure 4 Schematic diagram of the structure of the spatiotemporal optical flow estimator provided in an embodiment of the present application;

[0052] Figure 5 An image containing fog provided by an embodiment of the present application;

[0053] Figure 6 This is a clean image after defogging provided by an embodiment of the present application;

[0054] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0056] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0058] First, the technical terms involved in the embodiments of this application are introduced.

[0059] (1) Weight prediction module: A module composed of a neural network that takes the foggy image as input and outputs the enhancement weight.

[0060] (2) Tokenizer: A neural network module that takes enhancement weights as input and outputs tokens. The purpose of extracting tokens from enhancement weights is to filter out features in the enhancement weights that are beneficial for maintaining inter-frame consistency, such as the specific weights used by some targets, which represent inter-frame information in the weight prediction process.

[0061] (3) Flattening operation: An operation performed on an array to expand all elements of a multidimensional array into a row in sequence to obtain a one-dimensional array.

[0062] (4) Color Lookup Table (or LUT): A widely used color mapping method. It converts the color values ​​of an image using a predefined color mapping relationship. Its basic working principle is to define a mapping function for each color channel (usually RGB) of the input image, which is stored in a LUT. When processing an image, the color value of each pixel is converted using this LUT to produce the color value of the output image.

[0063] (5) Optical flow predictor: A module composed of a neural network, used to predict 3D optical flow. It takes the enhanced target frame, the initial 3D optical flow, and the image transformation result concatenated along the channel dimension as input and outputs the 3D optical flow.

[0064] (6) Image transformation: An operation widely used in image alignment. Image transformation typically relies on a transformation matrix that defines how each pixel position in the source image is mapped to a new position in the target image. Image transformation is the process of remapping the pixels of one image to another through mathematical operations based on the transformation matrix.

[0065] (7) Aligned frame: Use the predicted optical flow to calculate the transformation matrix and transform the reference image to obtain the result.

[0066] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0067] Figure 1 FIG is a flow chart of a video defogging method based on enhanced alignment provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps S101 to S104.

[0068] Step S101: performing color mapping on multiple foggy images using a color lookup table to obtain multiple enhanced images, where the multiple foggy images include a foggy target frame and multiple foggy reference frames. The multiple foggy reference frames are sequential frames in time sequence, and the foggy target frame is the next frame after the last frame in the multiple foggy reference frames in time sequence. The multiple enhanced images include an enhanced image (enhanced frame) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame.

[0069] Step S102, performing spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain a three-dimensional optical flow;

[0070] Step S103: performing image transformation on the foggy reference frame based on the three-dimensional optical flow to obtain an aligned frame;

[0071] Step S104 , based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, stitching and feature fusion are performed along the channel dimension to obtain a defogged image corresponding to the foggy target frame.

[0072] It can be understood that enhancing the foggy image using the lookup table to obtain an enhanced image (restoring details in the foggy image) effectively enhances its contrast. Estimating spatiotemporal optical flow based on the enhanced image can yield more accurate 3D optical flow, leading to more accurate alignment results. Furthermore, since the lookup table parameters used for foggy image enhancement are consistent across frames, the enhanced results across frames also maintain a certain degree of consistency, which is beneficial for optical flow prediction.

[0073] Therefore, by enhancing the details of foggy images and obtaining more accurate alignment results, accurate alignment frames provide more inter-frame information, which can effectively help the network learn better defogging capabilities, effectively solve the problem of difficult inter-frame alignment of foggy video data in existing technologies, and improve the ability of deep neural networks to utilize inter-frame information, achieving better video defogging effects.

[0074] In a possible implementation, the present application provides a video defogging method based on enhanced alignment, comprising the following steps:

[0075] Downsampling is used to obtain a low-resolution foggy image from the foggy target frame as the input of the weight prediction module;

[0076] A weight prediction module is used to predict low-resolution enhancement weights from low-resolution foggy images;

[0077] Upsampling is used to obtain enhancement weights from low-resolution enhancement weights as input to the token generator;

[0078] A tag generator is used to extract tags from the enhanced weights and concatenate them with the tag features of other frames to form a tag memory;

[0079] The enhanced weights are tiled and input into the attention mechanism together with the tag memory to obtain the memory weights;

[0080] Concatenate the memory weights and enhancement weights along the channel dimension and input them into the convolution layer to obtain the memory enhancement weights.

[0081] Using memory enhancement weights, according to the lookup table, the foggy image is enhanced to obtain an enhanced image;

[0082] The enhanced reference frame and the initial 3D optical flow are used to perform image transformation, and the transformation result is concatenated with the enhanced target frame and the initial 3D optical flow and then input into the optical flow predictor to obtain the 3D optical flow.

[0083] Using 3D optical flow, the foggy reference frame is transformed to obtain an aligned frame;

[0084] The aligned frame is concatenated with the foggy target frame and the enhanced target frame along the channel dimension and input into the fusion network to obtain the dehazed image.

[0085] It can be understood that in order to further enhance the consistency of enhancement results between different frames, a storage mechanism is adopted; first, the downsampling operation, weight prediction and upsampling operation are used in sequence to obtain the enhancement weight, and then the tag generator is used to obtain the tag, so as to construct a tag memory, and the attention mechanism is used to obtain the memory weight, and the convolution layer is used to obtain the memory enhancement weight, so that the weights used in different frame enhancements remain relevant, which is conducive to maintaining the consistency of the enhancement results and obtaining better inter-frame alignment results.

[0086] Optionally, multiple learnable color lookup tables can be used to perform color mapping on the foggy image using memory-enhanced weights to obtain an enhanced image.

[0087] The following describes the video defogging system based on enhanced alignment provided by this application.

[0088] like Figure 2As shown in the figure, the entire architecture consists of two parts: an enhanced alignment network and a fusion network. The enhanced alignment network is a network that performs alignment by predicting optical flow, which mainly includes a storage lookup table enhancement module and a spatiotemporal optical flow estimator. The storage lookup table enhancement module obtains enhanced frames from foggy frames, and the spatiotemporal optical flow estimator estimates the three-dimensional optical flow from the enhanced frames. Through image transformation operations, aligned frames are obtained. The fusion network is the main network that recovers clean defogged images from foggy images. Figure 2 The subscript t represents the time corresponding to the foggy target frame.

[0089] Specifically, a lookup table enhancement module is stored, which is used to: perform color mapping on multiple foggy images through a color lookup table to obtain multiple enhanced images, wherein the multiple foggy images include a foggy target frame and multiple foggy reference frames, the multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence, and the multiple enhanced images include enhanced images (enhanced frames) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame.

[0090] The spatiotemporal optical flow estimator is used to: perform spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain three-dimensional optical flow; and perform image transformation on the foggy reference frame based on the three-dimensional optical flow to obtain an aligned frame.

[0091] The fusion network is used to: perform splicing and feature fusion along the channel dimension based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame to obtain the defogged image corresponding to the foggy target frame.

[0092] The structure of the storage lookup table enhancement module is as follows Figure 3 As shown in Figure 2, it consists of five parts: weight prediction module, token generator, attention mechanism, convolutional layer and lookup table. Figure 3 The parameters involved are explained in

[15] , I represents the input foggy image, ω represents the enhancement weight, τ represents the tag generated by the tag generator, Q represents the query value used in the attention mechanism, K represents the key value used in the attention mechanism, V represents the true value used in the attention mechanism, ω represents the value of the attention mechanism. M represents the memory weight, ω ME represents the memory enhancement weight, represents the output enhanced image, Represents the scale of each parameter, H represents the height of the image, W represents the length of the image, C represents the number of channels for enhancing the weight, D represents the length of the tag, and N represents the number of tags stored in the tag memory.

[0093] The weight prediction module estimates the low-resolution enhancement weight from the downsampled foggy image, and then uses upsampling operation to obtain the enhancement weight;

[0094] The tag generator generates corresponding tags from the enhanced weights, and the tags of multiple frames of images form a tag memory;

[0095] The attention mechanism uses the tag memory and the tiled enhancement weights to calculate the attention information and obtain the memory enhancement weights;

[0096] The convolutional layer uses the enhancement weights and memory weights concatenated along the channel dimension as input to obtain the memory enhancement weights;

[0097] The lookup table uses memory enhancement weights to perform color mapping on the foggy image to obtain an enhanced image.

[0098] The structure of the spatiotemporal optical flow estimator is as follows Figure 4 As shown in Figure 1, it mainly includes an optical flow predictor. The spatiotemporal optical flow estimator first uses the initial 3D optical flow to transform the enhanced reference frame. The transformed result is then concatenated with the enhanced target frame and the initial 3D optical flow along the channel dimension. The optical flow predictor uses the concatenated result as input to generate the 3D optical flow.

[0099] At the same time, in order to further verify the effectiveness and robustness of this application, the video defogging model of this application is trained using a publicly synthesized dataset, and the foggy image is defogged by the trained video defogging model, and the following results are obtained: Figure 5 and Figure 6 The effect diagram shown.

[0100] It is understandable that the detailed functional implementation of each of the above units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0101] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0102] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (CommunicationsInterface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0103] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0104] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0105] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0106] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0107] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0109] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0110] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A video defogging method based on enhanced alignment, characterized in that: include: Color mapping is performed on the multiple foggy images using a color lookup table to obtain multiple enhanced images, where the multiple foggy images include a foggy target frame and multiple foggy reference frames. The multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence. The multiple enhanced images include an enhanced image corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame. Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, spatiotemporal optical flow estimation is performed to obtain three-dimensional optical flow; Based on 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame; Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame; The step of performing color mapping on the multiple foggy images through a color lookup table to obtain multiple enhanced images includes: Perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image; Extracting a tag from the enhanced weight to obtain a tag corresponding to the foggy image, and storing the tag corresponding to the foggy image in a tag memory; Input the tiling operation result of the enhanced weight corresponding to the foggy image and the mark in the mark memory to the attention mechanism, and obtain the memory weight output by the attention mechanism; Input the enhancement weight and memory weight corresponding to the foggy image to the convolution layer, and obtain the memory enhancement weight output by the convolution layer; Through the color lookup table, based on the memory enhancement weight corresponding to the foggy image, the color of the foggy image is enhanced to obtain the enhanced image corresponding to the foggy image.

2. The video defogging method based on enhanced alignment according to claim 1, characterized in that: The tag memory stores tags corresponding to each foggy image.

3. The video defogging method based on enhanced alignment according to claim 1, characterized in that: The weight prediction of the foggy image to obtain the enhancement weight corresponding to the foggy image includes: Downsampling the foggy image to obtain a downsampled foggy image; The weight prediction module is used to predict the enhancement weight corresponding to the downsampled foggy image. The enhancement weight corresponding to the downsampled foggy image is upsampled to obtain the enhancement weight corresponding to the foggy image.

4. The video defogging method based on enhanced alignment according to claim 1, characterized in that: There are multiple color lookup tables.

5. The video defogging method based on enhanced alignment according to any one of claims 1 to 4, characterized in that: The method of performing spatiotemporal optical flow estimation based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame to obtain the three-dimensional optical flow includes: Perform image transformation on the enhanced image corresponding to the foggy reference frame using the initial three-dimensional optical flow to obtain the image transformation result; Based on the image transformation results, the initial 3D optical flow and the enhanced image corresponding to the foggy target frame, they are spliced ​​along the channel dimension and the spatiotemporal optical flow is estimated to obtain the 3D optical flow.

6. A video defogging system based on enhanced alignment, characterized in that: include: Enhanced alignment and fusion networks; The enhanced alignment network includes a storage lookup table enhancement module and a spatiotemporal optical flow estimator; The storage lookup table enhancement module is used to: Color mapping is performed on the multiple foggy images using a color lookup table to obtain multiple enhanced images, where the multiple foggy images include a foggy target frame and multiple foggy reference frames. The multiple foggy reference frames are continuous frames in time sequence, and the foggy target frame is the next frame of the last frame in the multiple foggy reference frames in time sequence. The multiple enhanced images include an enhanced image (enhanced frame) corresponding to the foggy target frame and enhanced images corresponding to each foggy reference frame. The spatiotemporal optical flow estimator is used to: Based on the enhanced image corresponding to the foggy target frame and the enhanced images corresponding to each foggy reference frame, spatiotemporal optical flow estimation is performed to obtain three-dimensional optical flow; Based on 3D optical flow, the foggy reference frame is transformed to obtain the aligned frame; The fusion network is used for: Based on the aligned frame, the foggy target frame and the enhanced image corresponding to the foggy target frame, splicing and feature fusion are performed along the channel dimension to obtain the defogged image corresponding to the foggy target frame; The storage lookup table enhancement module includes: a weight prediction module, a tag generator, an attention mechanism module, a convolutional layer and a lookup table module; The weight prediction module is used to perform weight prediction on the foggy image and obtain the enhancement weight corresponding to the foggy image; The tag generator is used to extract tags from the enhanced weights to obtain tags corresponding to the foggy image, and store the tags corresponding to the foggy image in a tag memory; The attention mechanism module is configured to output a memory weight corresponding to the foggy image based on a tiling operation result of the enhancement weight corresponding to the foggy image and a mark in the mark memory; The convolution layer is used to output the memory enhancement weight corresponding to the foggy image based on the enhancement weight and memory weight corresponding to the foggy image; The lookup table module is used to perform color enhancement on the foggy image based on the memory enhancement weight corresponding to the foggy image through the color lookup table to obtain an enhanced image corresponding to the foggy image.

7. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed on a processor, the processor is caused to execute the method according to any one of claims 1 to 5.

Images