Video restoration method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of video processing, and provides a video restoration method and device, electronic equipment and storage medium, a target defogging model determines prior guide scene features through a prior decoder, which can not only avoid error accumulation caused by component estimation, but also adapt to various different scenes. At the same time, the defogging model can remove the blurring effect caused by haze and retain the details and color authenticity of the video as much as possible. Through space-time alignment of the prior guide scene features by a scene decoder, global features and long-time information can be accurately activated, the defogging difficulty is reduced, the quality, visual effect, robustness and consistency of the video restoration are improved, and a clearer and more accurate video stream is provided for subsequent visual processing tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and in particular to a video restoration method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the increasingly serious environmental problems, haze weather has become a common phenomenon in daily life. Haze greatly reduces the visibility and contrast of outdoor scenes, which adversely affects downstream visual tasks of videos collected outdoors, such as target detection and segmentation in automatic driving and monitoring.

[0003] Existing video restoration methods usually perform defogging based on component estimation of a physical model. However, errors are generated and continuously accumulated in the component estimation process, which will result in poor defogging effect. Moreover, the parameters of the physical model are usually fixed and cannot be adaptively adjusted according to different scenes. In some cases, the haze distribution may be misjudged, and accurate defogging cannot be achieved. In addition, existing video restoration methods aggregate temporal information by using feature stacking or frame-to-frame alignment in a local sliding window. It is difficult to accurately obtain global features and long-range temporal information, which not only increases the difficulty of defogging, but also affects the defogging effect. SUMMARY

[0004] The present application provides a video restoration method, device, electronic device and storage medium to solve the defects in the prior art.

[0005] The present application provides a video restoration method, comprising:

[0006] obtaining a video to be defogged;

[0007] inputting a current video frame in the video to be defogged and one or more historical video frames adjacent to the current video frame into a target defogging model to obtain a defogged video frame of the current video frame;

[0008] The target defogging model comprises an input layer, an encoder of different scales, a prior decoder and a scene decoder, and an output layer. The prior decoder and the scene decoder of the same scale are connected with the encoder. The input layer is connected with the encoder of the initial scale, and the output layer is connected with the input layer and the scene decoder of the initial scale, respectively.

[0009] The input layer is configured to receive the current video frame and the historical video frame.

[0010] The encoder is configured to extract image features of a corresponding scale. The prior decoder is configured to extract initial prior features of a corresponding scale. The scene decoder is configured to extract initial scene features of a corresponding scale.

[0011] The prior decoder is further configured to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of corresponding scales;

[0012] The scene decoder is further configured to perform spatio-temporal alignment on the prior guided scene features of the corresponding scales to obtain target aligned features, and determine a decoding result of a corresponding scale based on the target aligned features;

[0013] The output layer is configured to determine the defogged video frame based on the current video frame and the historical video frame, and the decoding result determined by the scene decoder of the initial scale.

[0014] According to the video restoration method provided in the application, the prior decoder comprises a physical prior guidance module, the scene decoder comprises a pyramid cascaded deformable convolution alignment module and a multi-range time feature aggregation module connected in sequence, and the physical prior guidance module is connected with the pyramid cascaded deformable convolution alignment module.

[0015] The physical prior guidance module is configured to guide and enhance the initial scene features based on the initial prior features to obtain first prior guided scene features of the current video frame and second prior guided scene features of the historical video frame.

[0016] The pyramid cascaded deformable convolution alignment module is configured to perform spatio-temporal alignment on scene feature sets of different time ranges in the first prior guided scene features and the second prior guided scene features to obtain target aligned features corresponding to each of the scene feature sets.

[0017] The multi-range time feature aggregation module is configured to aggregate each of the target aligned features to obtain the decoding result.

[0018] According to the video restoration method provided in the application, the physical prior guidance module comprises a prior compression module, a memory enhancement module and a prior guidance module.

[0019] The prior compression module is configured to classify and normalize the initial prior features, determine a transmission distribution map, and generate a prior compression mark based on the initial prior features and the transmission distribution map.

[0020] The memory enhancement module is configured to store the initial prior features in a memory unit based on the prior compression mark, and determine memory enhanced prior features in the memory unit based on an attention mechanism and the prior compression mark.

[0021] The prior guidance module is configured to generate the first prior guided scene features and the second prior guided scene features based on the memory enhanced prior features and the initial scene features.

[0022] According to the video restoration method provided by the application, the pyramid cascaded deformable convolution alignment module comprises a comprehensive alignment unit and a plurality of alignment units of different levels;

[0023] For any scene feature set, the alignment unit is configured to determine a feature offset of a corresponding level based on a first prior guided scene feature of the corresponding level and the any scene feature set, and a feature offset of a next level, and determine an alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, the any scene feature set and the feature offset, and an alignment feature of the next level;

[0024] The comprehensive alignment unit is configured to determine the target alignment feature based on the first prior guided scene feature of the initial level and the alignment feature.

[0025] According to the video restoration method provided by the application, the alignment unit is specifically configured to:

[0026] The first prior guided scene feature of the corresponding level is concatenated with the any scene feature set and then convolved to obtain an initial offset of the corresponding level, and the initial offset of the corresponding level is concatenated with an up-sampling result of the feature offset of the next level and then multi-layer convolved to obtain the feature offset of the corresponding level.

[0027] The any scene feature set of the corresponding level is concatenated with the feature offset and then deformable convolved to obtain an initial alignment feature of the corresponding level, and the initial alignment feature of the corresponding level is concatenated with an up-sampling result of the alignment feature of the next level and then multi-layer convolved to obtain the alignment feature of the corresponding level.

[0028] According to the video restoration method provided by the application, the comprehensive alignment unit is specifically configured to:

[0029] The first prior guided scene feature of the initial level and the alignment feature are concatenated and then multi-layer convolved to obtain a comprehensive offset.

[0030] The comprehensive offset is deformable convolved to determine the target alignment feature.

[0031] According to the video restoration method provided by the application, the multi-range time feature aggregation module is specifically configured to:

[0032] The target alignment features are concatenated to obtain a concatenated feature.

[0033] After each target alignment feature is sequentially subjected to an average pooling operation and convolution, the target alignment features are concatenated and globally average-pooled to obtain a scaling parameter.

[0034] The concatenation features are sequentially subjected to convolution and global average pooling to obtain a shift parameter;

[0035] Based on the scaling parameter and the shift parameter, the concatenation features are aggregated to obtain an aggregation result, and the aggregation result is subjected to convolution to obtain the decoding result.

[0036] According to the video restoration method provided by the application, the target defogging model is based on the to-be-defogged video sample and the clear video sample corresponding to the to-be-defogged video sample, and the initial defogging model is trained as follows:

[0037] The current video frame sample in the to-be-defogged video sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial defogging model to obtain an initial prior feature sample extracted by an initial prior decoder in the initial defogging model and a defogging video frame sample of the current video frame sample;

[0038] Based on the initial prior feature sample, the transmittance and the atmospheric light value are predicted, and based on the transmittance and the atmospheric light value and the defogging video frame sample, a fog map is reconstructed;

[0039] Based on the fog map and the current video frame sample, a physical model disentanglement loss is calculated, and based on the clear video sample and the defogging video frame sample, a model output loss is calculated, and based on the physical model disentanglement loss and the model output loss, a total loss is determined;

[0040] Based on the total loss, the initial defogging model is iteratively trained to obtain the target defogging model.

[0041] The application further provides a video restoration device, comprising:

[0042] A video acquisition module is configured to acquire a to-be-defogged video.

[0043] A defogging module is configured to input a current video frame in the to-be-defogged video and one or more historical video frames adjacent to the current video frame into a target defogging model to obtain a defogging video frame of the current video frame.

[0044] The target defogging model comprises an input layer, encoders of different scales, a prior decoder, a scene decoder, and an output layer; the prior decoder and the scene decoder of the same scale are connected with the encoders; the input layer is connected with the encoder of the initial scale, and the output layer is connected with the input layer and the scene decoder of the initial scale, respectively;

[0045] The input layer is configured to receive the current video frame and the historical video frame.

[0046] The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale.

[0047] The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale.

[0048] The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features;

[0049] The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video restoration method as described above.

[0051] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video restoration method as described above.

[0052] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video restoration method as described above.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The video restoration method, apparatus, electronic device, and storage medium provided by this invention determine prior-guided scene features through a prior decoder in the target dehazing model. This not only avoids error accumulation caused by component estimation but also adapts to various different scenarios. Simultaneously, it can preserve video details and color realism as much as possible while removing the blurring effect caused by haze. Spatiotemporal alignment of the prior-guided scene features through the scene decoder accurately activates global features and remote temporal information, reducing the difficulty of dehazing and improving the quality, visual effects, robustness, and consistency of video restoration, providing a clearer and more accurate video stream for subsequent visual processing tasks. Attached Figure Description

[0055] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below, based on the drawings, all other drawings that can be obtained by a person of ordinary skill in the art without creative work, shall belong to the protection scope of the present application.

[0056] Figure 1 is a flowchart of the video restoration method provided by the present application;

[0057] Figure 2 is a structural diagram of a target defogging model in the video restoration method provided by the present application;

[0058] Figure 3 is a structural diagram of a prior decoder in the video restoration method provided by the present application;

[0059] Figure 4 is a structural diagram of a physical prior guiding module in the video restoration method provided by the present application;

[0060] Figure 5 is a structural diagram of a pyramid cascaded deformable convolution alignment module in the video restoration method provided by the present application;

[0061] Figure 6 is a structural diagram of a multi-range time feature aggregation module in the video restoration method provided by the present application;

[0062] Figure 7 is a flowchart of a foggy image reconstructed when training the model in the video restoration method provided by the present application;

[0063] Figure 8 is a structural diagram of a video restoration device provided by the present application;

[0064] Figure 9 is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0065] In order to make the objects, technical solutions and advantages of the present application clearer, the following will combine the drawings in the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work shall belong to the protection scope of the present application.

[0066] Figure 1 is a flowchart of the video restoration method provided by the present application, as shown in Figure 1 , the method comprises:

[0067] S1, obtaining a to-be-dehazed video;

[0068] S2, inputting a current video frame in the to-be-dehazed video and one or more historical video frames adjacent to the current video frame into a target dehazing model to obtain a dehazed video frame of the current video frame;

[0069] The target dehazing model comprises an input layer, an encoder of different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and the scene decoder of the same scale are connected with the encoder; the input layer is connected with the encoder of an initial scale, and the output layer is connected with the input layer and the scene decoder of the initial scale respectively;

[0070] The input layer is configured to receive the current video frame and the historical video frames;

[0071] The encoder is configured to extract image features of a corresponding scale; the prior decoder is configured to extract initial prior features of the corresponding scale; and the scene decoder is configured to extract initial scene features of the corresponding scale;

[0072] The prior decoder is further configured to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale;

[0073] The scene decoder is further configured to perform spatio-temporal alignment on the prior guided scene features of the corresponding scale to obtain target aligned features, and determine a decoding result of the corresponding scale based on the target aligned features;

[0074] The output layer is configured to determine the dehazed video frame based on the current video frame and the historical video frames and the decoding result determined by the scene decoder of the initial scale.

[0075] Specifically, the video restoration method provided in the embodiment of the present application has a video restoration device as an execution subject, which can be configured in a computer, which can be a local computer or a cloud computer, and the local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0076] Firstly, step S1 is performed to obtain a to-be-dehazed video. The to-be-dehazed video refers to a video in which each video frame needs to be dehazed, and obtaining the to-be-dehazed video can include multiple video frames.

[0077] Then, step S2 is performed, and the current video frame in the video to be defogged and one or more historical video frames adjacent to the current video frame are input to the target defogging model to obtain a defogged video frame of the current video frame. The number of historical video frames adjacent to the current video frame can be set as needed, for example, it can be set to 3. When the target defogging model is used to defog the current video frame to obtain a defogged video frame, the current video frame and three historical video frames adjacent to the current video frame need to be input.

[0078] As shown in Figure 2 , the target defogging model can include an input layer, encoders of different scales, prior decoders and scene decoders, and an output layer. The prior decoders and the scene decoders of the same scale are connected with the encoders; the input layer is connected with the encoder of the initial scale, and the output layer is connected with the input layer and the scene decoder of the initial scale, respectively. Here, the encoders of each scale can include convolution layers and pooling layers.

[0079] The input layer is used to receive the current video frame and the historical video frames. If the current video frame is i, then the three historical video frames are i-1, i-2 and i-3, respectively.

[0080] After the input layer receives each video frame, each video frame can be downsampled first. The reciprocal s of the downsampling factor is 1, indicating that the size of each video frame remains unchanged. Different scales can be obtained by downsampling operations with different downsampling factors.

[0081] In Figure 2 , the target defogging model can include four encoders of different scales, prior decoders and scene decoders. The four encoders of different scales can be represented as Encoder Layer0, Encoder Layer1, Encoder Layer2 and Encoder Layer3, respectively. The reciprocal s of the sampling factors of the downsampling is 1 / 4, 1 / 8, 1 / 16 and 1 / 32, respectively. When s is the same, it indicates that the scales are the same. The four encoders of different scales are connected in order from large to small s.

[0082] The four prior decoders of different scales can be represented as Prior Decoder Layer0, Prior Decoder Layer1, Prior Decoder Layer2 and Prior Decoder Layer3, respectively. The ratio s of the size of the image features corresponding to the scales to the size of the input video frames is 1 / 4, 1 / 8, 1 / 16 and 1 / 32, respectively.

[0083] The four scene decoders of different scales are sequentially connected in the order of s from small to large, the four scene decoders of different scales are sequentially connected in the order of s from small to large, the encoders of the same s are connected with the prior decoders and the scene decoders respectively to realize skip connection, and the prior decoders and the scene decoders of the same s are connected.

[0084] The functions of the encoders are the same, and the encoders are used for extracting image features of corresponding scales. The functions of the prior decoders are the same, and the prior decoders are used for extracting initial prior features of corresponding scales. The functions of the scene decoders are the same, and the scene decoders are used for extracting initial scene features of corresponding scales.

[0085] The prior decoder is further used for guiding and enhancing the initial scene features by using the initial prior features to obtain prior guided scene features of corresponding scales.

[0086] The scene decoder is further used for performing spatio-temporal alignment on the prior guided scene features of corresponding scales to obtain target alignment features, and determining a decoding result of corresponding scales by using the target alignment features.

[0087] The output layer is used for determining a defogging video frame by using the current video frame and the historical video frame received by the input layer and the decoding result determined by the scene decoder of the initial scale. Figure 2 As shown in the figure, the decoding result determined by the scene decoder of the initial scale can be upsampled to obtain a decoding result with the same size as the current video frame. Thereafter, the current video frame and the historical video frame are superimposed with the upsampled decoding result to obtain a defogging result. Since the input layer receives the current video frame and the historical video frame, the defogging result includes a defogging video frame of the current video frame and defogging video frames of the historical video frames.

[0088] The video restoration method provided in the embodiment of the application can determine prior guided scene features by using the prior decoder, can not only avoid error accumulation caused by component estimation, but also can adapt to various scenes. Meanwhile, the video details and color authenticity can be preserved as much as possible on the basis of removing the blur influence caused by haze. The spatio-temporal alignment of the prior guided scene features by the scene decoder can accurately activate global features and long-time information, reduce the difficulty of defogging, and improve the quality, visual effect, robustness and consistency of video restoration, and provides clearer and more accurate video streams for subsequent visual processing tasks.

[0089] On the basis of the above-mentioned embodiments, as shown in Figure 3As shown, the prior decoder comprises a physical prior guidance module, the scene decoder comprises a pyramid cascaded deformable convolution alignment module and a multi-range time feature aggregation module connected in sequence, and the physical prior guidance module is connected with the pyramid cascaded deformable convolution alignment module.

[0090] The physical prior guidance module is configured to guide and enhance the initial scene feature based on the initial prior feature, to obtain a first prior guided scene feature of the current video frame and a second prior guided scene feature of the historical video frame.

[0091] The pyramid cascaded deformable convolution alignment module is configured to perform spatio-temporal alignment on scene feature sets of different time ranges in the first prior guided scene feature and the second prior guided scene feature, to obtain target alignment features corresponding to the scene feature sets.

[0092] The multi-range time feature aggregation module is configured to aggregate the target alignment features to obtain the defogged video frame.

[0093] Specifically, as shown in the figure, Figure 3 each prior decoder of each scale comprises an initial prior decoding module and a physical prior guidance module (MPG), each scene decoder of each scale comprises an initial scene decoding module, a pyramid cascaded deformable convolution alignment module (PCD) and a multi-range time feature aggregation module (MTA) connected in sequence, and the MPG is connected with the PCD.

[0094] The input of the initial prior decoding module can include the output of the prior decoder of the next scale (namely Input1) and the output of the encoder of the same scale (namely Input2). The initial prior decoding module can first upsample Input1, then perform pixel-by-pixel addition on the upsampled result and Input2, and then perform convolution to obtain the initial prior feature .

[0095] The input of the initial scene decoding module can include the output of the scene decoder of the next scale (namely Input3) and the output of the encoder of the same scale (namely Input4). The initial scene decoding module can first upsample Input3, then perform pixel-by-pixel addition on the upsampled result and Input4, to obtain the initial scene feature .

[0096] The MPG can guide and enhance the initial scene feature based on the initial prior feature to obtain the prior guided scene feature of the corresponding scale The prior guided scene features can include first prior guided scene features of a current video frame and second prior guided scene features of historical video frames.

[0097] The PCD can utilize deformable convolution to spatio-temporally align the first prior guided scene features with scene feature sets of different time ranges in the second prior guided scene features to obtain target aligned features corresponding to the scene feature sets. , i is the current video frame, i-1 is the first historical video frame before the current video frame, i-2 is the second historical video frame before the current video frame, and i-3 is the third historical video frame before the current video frame. Furthermore, the scene feature sets of different time ranges can be represented as .

[0098] The MTA can aggregate the target aligned features to obtain a decoding result at a current scale.

[0099] In the embodiment, the physical prior guidance module can improve the accuracy of the prior guided scene features. In combination with the physical prior guidance module and the pyramid cascaded deformable convolution alignment module, the defects that it is difficult to accurately obtain panoramic features and long-range time information in the prior art can be further avoided, the difficulty of defogging is reduced, and the defogging effect is improved.

[0100] On the basis of the above-mentioned embodiment, the physical prior guidance module comprises a prior compression module, a memory enhancement module and a prior guidance module.

[0101] The prior compression module is configured to classify and normalize the initial prior features, determine a transmission distribution map, and generate a prior compression mark based on the initial prior features and the transmission distribution map.

[0102] The memory enhancement module is configured to store the initial prior features in a memory unit based on the prior compression mark, and determine memory-enhanced prior features in the memory unit based on an attention mechanism and the prior compression mark.

[0103] The prior guidance module is configured to generate the first prior guided scene features and the second prior guided scene features based on the memory-enhanced prior features and the initial scene features.

[0104] Specifically, as shown in Figure 4 The physical prior guidance module comprises a prior compression module a, a memory enhancement module b and a prior guidance module c.

[0105] The prior compression module a is used to compress the initial prior feature The transmission distribution graph is determined by performing classification and normalization by a Softmax function . Wherein, H and W are the height and width of the initial prior feature respectively, C is the channel number of the initial prior feature, and D is the category number of the initial prior feature, and D is less than C. Each element in the transmission distribution graph represents the probability of the pixel point at the corresponding position in the initial prior feature being distributed in the category.

[0106] Thereafter, the initial prior feature and the transmission distribution graph are multiplied by a matrix to generate a prior compression token The prior compression token can be used to count the maximum ratio of the pixel points in each channel in which category. The row of the prior compression token represents the category number, and the column represents the channel number. Each point in the prior compression token records the total score of all pixel points in each channel in the category.

[0107] The memory enhancement module b can use the prior compression token p to store the initial prior feature to the memory unit (Prior Token Memory). The memory unit can store historical haze information at different times. The feature vector in the memory unit is represented as a key K and a value V, and the dimension is . Wherein, N is the number of storage, that is, the number of video frames that have been dehazed in the video to be processed.

[0108] In order to realize the interaction between the current haze information and the historical information, the attention mechanism is used, and the prior compression token p is applied to read the memory enhanced prior feature P in the memory unit. The memory enhanced prior feature P can represent global and local physical information.

[0109] The calculation formula of the attention mechanism is as follows:

[0110] ;

[0111] Wherein, is the query feature, which is obtained by flattening the initial prior feature ; is the key of the historical prior memory in the memory unit; is the value of the historical prior memory in the memory unit; is a scaling factor, which is used to stabilize the numerical value and prevent the gradient from being too large or too small.

[0112] The prior guidance module c can use the memory enhanced prior featureP Features of the initial scene The features are concatenated and processed using multiple convolutional layers to obtain prior guided scene features at the corresponding scale. Due to the initial scene characteristics It contains rich texture, edge, and semantic context information. Prior features are enhanced by memory. P Features of the initial scene By concatenating these layers, physical model guidance information and semantic scene information can be integrated in the feature space. Multiple convolutional layers further learn the complex relationship between the two, extracting important joint features layer by layer—that is, prior guiding scene features—to generate a more accurate dehazed scene representation. This design ensures that the dehazing results conform to the physical laws of atmospheric scattering models while restoring delicate scene details, thus improving the overall quality and consistency of dehazed video frames.

[0113] Based on the above embodiments, the pyramid-cascaded deformable convolution alignment module includes a comprehensive alignment unit and multiple alignment units at different levels;

[0114] For any set of scene features, the alignment unit is used to determine the feature offset of the corresponding level based on the first prior guided scene feature of the corresponding level and any set of scene features, as well as the feature offset of the next level, and to determine the alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, any set of scene features and the feature offset, as well as the alignment feature of the next level.

[0115] The integrated alignment unit is used to determine the target alignment feature based on the first prior guiding scene features and alignment features of the initial level.

[0116] Specifically, such as Figure 5 As shown, PCD aligns features in a coarse-to-fine manner to handle large and complex motions. Using a pyramid structure, lower-scale features are first aligned with a coarse estimate, and then the offset and aligned features are propagated to higher scales to facilitate accurate motion compensation. Furthermore, an additional deformable convolution is cascaded after the pyramid alignment operation to further improve the robustness of the alignment.

[0117] The first prior guided scene feature can be represented as Target J, and the second prior guided scene feature can be represented as Neighbor J. PCD can include a comprehensive alignment unit and multiple alignment units at different levels, and each alignment unit can be represented as L1, L2 and L3 respectively.

[0118] For any scene feature set, the alignment unit can determine the feature offset of the corresponding level by using the first prior guided scene feature of the corresponding level and any scene feature set, and the feature offset of the next level. The first prior guided scene feature of L1 is Target J, and the any scene feature set of L1 is any scene feature set in Neighbor J. The first prior guided scene feature of L2 can be obtained by downsampling Target J using a stride convolution filter, and the any scene feature set of L2 can be obtained by downsampling any scene feature set in Neighbor J using a stride convolution filter. The first prior guided scene feature of L3 can be obtained by downsampling the first prior guided scene feature of L2 using a stride convolution filter, and the any scene feature set of L3 can be obtained by downsampling the any scene feature set of L2 using a stride convolution filter. Here, the downsampling factor is 2, that is, the size of the feature is reduced to one half of the original. Moreover, in order not to increase the amount of calculation, no additional channel number is added in the downsampling.

[0119] The alignment unit of each level can concatenate the first prior guided scene feature of the corresponding level and any scene feature set and then perform convolution to obtain the initial offset of the corresponding level. The initial offset of the corresponding level and the up-sampled result of the feature offset of the next level are concatenated and then multi-layer convolution is performed to obtain the feature offset of the corresponding level.

[0120] In the L3 level, the calculation formula of the feature offset is:

[0121] ;

[0122] wherein, is the feature offset of the L3 level, is a general function composed of multiple convolution layers, which realizes a feature fusion operation. is a concatenation operation. k is the number of historical video frames.

[0123] In the Lth level except the last layer, in order to generate the feature offset of the Lth level, the up-sampled feature offset from the next layer L+1 level is also used to generate the feature offset of the Lth level by the following formula:

[0124] ;

[0125] wherein, is the feature offset of the Lth level, and the value of L includes L1 and L2, is the feature offset of the L+1 level, denotes a double up-sampling, same as down-sampling, without adding extra channels; denotes a set of scene features of k historical video frames.

[0126] An alignment of each level can concatenate any set of scene features of the corresponding level with the feature offset and then perform deformable convolution to obtain an initial alignment of the corresponding level, and concatenate the initial alignment of the corresponding level with an up-sampled result of the alignment of the next level and then perform multi-layer convolution to obtain the alignment of the corresponding level.

[0127] Here, the set of scene features of k historical video frames of the level and the feature offset are concatenated and then passed through deformable convolution (DComv) to generate an up-sampled result of the alignment of the next level, and the process is represented by the following formula:

[0128] ;

[0129] wherein, is the alignment of the L level, is a general function composed of multiple convolution layers, denotes deformable convolution.

[0130] The alignment of the initial level can be represented as:

[0131] .

[0132] wherein, is the alignment of the initial level.

[0133] The comprehensive alignment unit can determine the target alignment feature by concatenating the first prior guided scene feature and the alignment of the initial level and then performing multi-layer convolution to obtain a comprehensive offset, and performing deformable convolution on the comprehensive offset to obtain the target alignment feature.

[0134] The target alignment feature can be represented as:

[0135] ;

[0136] wherein, is the target alignment feature, which includes three, is the target alignment feature corresponding to the first set of scene features, is the target alignment feature corresponding to the second set of scene features, is the target alignment feature corresponding to the third set of scene features.

[0137] On the basis of the above-mentioned embodiments, the multi-range time feature aggregation module is specifically used for:

[0138] The target alignment features are concatenated to obtain concatenated features;

[0139] After each target alignment feature is sequentially subjected to an average pooling operation and convolution, concatenation and global average pooling are performed to obtain a scaling parameter;

[0140] The concatenated features are sequentially subjected to convolution and global average pooling to obtain a shift parameter;

[0141] Based on the scaling parameter and the shift parameter, the concatenated features are aggregated to obtain an aggregation result, and the aggregation result is convolved to obtain the decoding result.

[0142] Specifically, as shown in Figure 6 , the MTA can concatenate , and to obtain concatenated features .

[0143] Then, after each target alignment feature is sequentially subjected to an average pooling operation and convolution, concatenation and global average pooling are performed to obtain a scaling parameter. That is,

[0144] ;

[0145] wherein, is the scaling parameter, is the global average pooling, is a 3x3 convolution kernel, is an average pooling operation for each channel, the number of channels remains unchanged, and C is the number of channels.

[0146] The concatenated features are sequentially subjected to convolution and global average pooling to obtain a shift parameter. That is,

[0147] .

[0148] Finally, the scaling parameter and the shift parameter are used to aggregate the concatenated features to obtain an aggregation result, and the aggregation result is convolved to obtain a decoding result corresponding to a scale. That is,

[0149] ;

[0150] wherein, is the decoding result corresponding to the scale.

[0151] Based on the above embodiments, the target dehazing model is trained by performing the following steps on the initial dehazing model based on the video sample to be dehazed and the corresponding clear video sample:

[0152] The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample.

[0153] Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed.

[0154] Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss.

[0155] Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model.

[0156] Specifically, the target dehazing model can be obtained by training an initial dehazing model based on the video samples to be dehazed and the clear video samples corresponding to the video samples to be dehazed.

[0157] The training steps may include:

[0158] The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature samples extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame samples of the current video frame sample.

[0159] After that, as Figure 7 As shown, transmittance is predicted using the initial prior feature sample Output1. and atmospheric light value and utilize transmittance and atmospheric light value The fog map was reconstructed using the dehazed video frame sample Output2 and an atmospheric scattering model. The initial dehazing model can obtain initial prior feature samples at different scales, which in turn can produce haze maps at different scales. .

[0160] Using the fog map and current video frame samples, calculate the entanglement disentanglement loss of the physical model. That is:

[0161] ;

[0162] in, De-entanglement loss for the physical model, It is an L1 norm. Different scales of current video frame samples can be obtained by downsampling the current video frame samples with different sampling factors.

[0163] Subsequently, using clear video samples and dehazed video frame samples, the model output loss is calculated, namely:

[0164] ;

[0165] in, For dehazing video frame samples, For clear video samples.

[0166] The total loss can be determined by weighted summing of the deentanglement loss of the physical model and the model output loss. That is:

[0167] ;

[0168] in, Weights for the deentanglement loss in the physical model. This represents the total loss.

[0169] Finally, the initial dehazing model is iteratively trained using the total loss until the total loss converges or the preset number of iterations is reached, thus obtaining the target dehazing model. The preset number of iterations can be set as needed and is not specifically limited here.

[0170] like Figure 8 As shown, based on the above embodiments, this embodiment of the invention provides a video restoration device, including:

[0171] Video acquisition module 81 is used to acquire the video to be dehazed;

[0172] The dehazing module 82 is used to input the current video frame in the video to be dehazed and one or more historical video frames adjacent to the current video frame into the target dehazing model to obtain the dehazed video frame of the current video frame.

[0173] The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale.

[0174] The input layer is configured to receive the current video frame and the historical video frame;

[0175] The encoder is configured to extract image features of a corresponding scale; the prior decoder is configured to extract initial prior features of the corresponding scale; and the scene decoder is configured to extract initial scene features of the corresponding scale;

[0176] The prior decoder is further configured to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale;

[0177] The scene decoder is further configured to perform spatio-temporal alignment on the prior guided scene features of the corresponding scale to obtain target alignment features, and determine a decoding result of the corresponding scale based on the target alignment features;

[0178] The output layer is configured to determine the defogged video frame based on the current video frame and the historical video frame, and a decoding result determined by the scene decoder of an initial scale.

[0179] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application comprises a physical prior guidance module, the scene decoder comprises a pyramid cascaded deformable convolution alignment module and a multi-range time feature aggregation module connected in sequence, and the physical prior guidance module is connected with the pyramid cascaded deformable convolution alignment module;

[0180] The physical prior guidance module is configured to guide and enhance the initial scene features based on the initial prior features to obtain first prior guided scene features of the current video frame and second prior guided scene features of the historical video frame;

[0181] The pyramid cascaded deformable convolution alignment module is configured to perform spatio-temporal alignment on scene feature sets of different time ranges in the first prior guided scene features and the second prior guided scene features to obtain target alignment features corresponding to each of the scene feature sets;

[0182] The multi-range time feature aggregation module is configured to aggregate each of the target alignment features to obtain the decoding result.

[0183] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application comprises a physical prior guidance module, the scene decoder comprises a pyramid cascaded deformable convolution alignment module and a multi-range time feature aggregation module connected in sequence, and the physical prior guidance module is connected with the pyramid cascaded deformable convolution alignment module;

[0184] The prior compression module is configured to classify and normalize the initial prior features, determine a transmission distribution map, and generate a prior compression mark based on the initial prior features and the transmission distribution map;

[0185] The memory enhancement module is configured to store the initial prior feature into a memory unit based on the prior compression mark, and determine a memory-enhanced prior feature in the memory unit based on an attention mechanism and the prior compression mark.

[0186] The prior guidance module is configured to generate the first prior guidance scene feature and the second prior guidance scene feature based on the memory-enhanced prior feature and the initial scene feature.

[0187] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application comprises a comprehensive alignment unit and a plurality of alignment units of different levels in the pyramid cascaded deformable convolution alignment module.

[0188] For any scene feature set, the alignment unit is configured to determine a feature offset of a corresponding level based on a first prior guidance scene feature of the corresponding level and any scene feature set, and a feature offset of a next level, and determine an alignment feature of the corresponding level based on the first prior guidance scene feature of the corresponding level, the any scene feature set and the feature offset, and an alignment feature of the next level.

[0189] The comprehensive alignment unit is configured to determine the target alignment feature based on the first prior guidance scene feature of the initial level and the alignment feature.

[0190] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application comprises a comprehensive alignment unit and a plurality of alignment units of different levels in the pyramid cascaded deformable convolution alignment module.

[0191] The first prior guidance scene feature of the corresponding level and any scene feature set are concatenated and then convolved to obtain an initial offset of the corresponding level, and the initial offset of the corresponding level and an up-sampling result of the feature offset of the next level are concatenated and then multi-layer convolved to obtain the feature offset of the corresponding level.

[0192] The any scene feature set of the corresponding level and the feature offset are concatenated and then deformable-convolved to obtain an initial alignment feature of the corresponding level, and the initial alignment feature of the corresponding level and an up-sampling result of the alignment feature of the next level are concatenated and then multi-layer convolved to obtain the alignment feature of the corresponding level.

[0193] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application comprises a comprehensive alignment unit and a plurality of alignment units of different levels in the pyramid cascaded deformable convolution alignment module.

[0194] The first prior guidance scene feature of the initial level and the alignment feature are concatenated and then multi-layer convolved to obtain a comprehensive offset.

[0195] The comprehensive offset is deformable-convolved to determine the target alignment feature.

[0196] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application comprises a multi-range time feature aggregation module, which is specifically used for:

[0197] The target alignment features are concatenated to obtain concatenated features.

[0198] After the target alignment features are sequentially subjected to an average pooling operation and convolution respectively, concatenation and global average pooling are performed to obtain scaling parameters.

[0199] The concatenated features are sequentially subjected to convolution and global average pooling to obtain shift parameters.

[0200] Based on the scaling parameters and the shift parameters, the concatenated features are aggregated to obtain an aggregation result, and the aggregation result is subjected to convolution to obtain the decoding result.

[0201] On the basis of the above-mentioned embodiments, the video restoration device provided in the embodiments of the present application further comprises a training module, which is used for:

[0202] The current video frame sample in the to-be-dehazed video sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain an initial prior feature sample extracted by an initial prior decoder in the initial dehazing model and a dehazed video frame sample of the current video frame sample.

[0203] Based on the initial prior feature sample, transmittance and atmospheric light values are predicted, and a fog map is reconstructed based on the transmittance and the atmospheric light values and the dehazed video frame sample.

[0204] Based on the fog map and the current video frame sample, a physical model disentanglement loss is calculated, and a model output loss is calculated based on the clear video sample and the dehazed video frame sample, and based on the physical model disentanglement loss and the model output loss, a total loss is determined.

[0205] Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model.

[0206] Specifically, the roles of each module in the video restoration device provided in the embodiments of the present application are one-to-one corresponding to the operation processes of each step in the method embodiment, and the effects achieved are consistent. For details, refer to the above-mentioned embodiments, and the embodiments of the present application will not be described here.

[0207] Figure 9 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 9As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete communications with each other through the communications bus 840. The processor 810 can invoke a logical instruction in the memory 830 to execute the video restoration method provided in each of the above embodiments.

[0208] In addition, the logical instruction in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0209] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the video restoration method provided in each of the above embodiments.

[0210] In yet another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the video restoration method provided in each of the above embodiments.

[0211] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0212] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0213] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video restoration method, characterized in that, include: Obtain the video to be dehazed; The current video frame in the video to be dehazed, along with one or more historical video frames adjacent to the current video frame, are input into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, cascaded encoders at different scales, a prior decoder, and a scene decoder, as well as an output layer. The prior decoders and scene decoders at the same scale are connected to the encoder. The input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The prior decoder includes a physical prior guidance module, and the scene decoder includes a pyramid-cascaded deformable convolution alignment module and a multi-range temporal feature aggregation module connected in sequence. The physical prior guidance module is connected to the pyramid-cascaded deformable convolution alignment module. The physical prior guidance module includes a prior compression module, a memory enhancement module, and a prior guidance module. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale; The target dehazing model is trained based on the video samples to be dehazed and the corresponding clear video samples, using the following steps to train the initial dehazing model: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample. Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed. Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss. Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model; The formula for calculating the deentanglement loss of the physical model is as follows: ； in, The entanglement loss is used to resolve the physical model. It is an L1 norm. These are current video frame samples at different scales. The fog map; The formula for calculating the model output loss is as follows: ； in, Output the loss for the model. The dehazed video frame sample, The clear video sample; The formula for calculating the total loss is as follows: ； in, The weights for the deentanglement loss of the physical model. The total loss is mentioned above.

2. The video restoration method according to claim 1, characterized in that, The physical prior guidance module is used to guide and enhance the initial scene features based on the initial prior features, so as to obtain the first prior guided scene features of the current video frame and the second prior guided scene features of the historical video frames. The pyramid-cascaded deformable convolutional alignment module is used to perform spatiotemporal alignment of the first prior guided scene features with the second prior guided scene features in different time ranges to obtain the target alignment features corresponding to each scene feature set. The multi-range temporal feature aggregation module is used to aggregate the alignment features of each target to obtain the decoding result.

3. The video restoration method according to claim 2, characterized in that, The prior compression module is used to classify and normalize the initial prior features, determine the transmission distribution map, and generate prior compression tags based on the initial prior features and the transmission distribution map. The memory enhancement module is used to store the initial prior features into the memory unit based on the prior compression tag, and to determine the memory enhancement prior features in the memory unit by applying the prior compression tag based on the attention mechanism. The prior guidance module is used to generate the first prior guidance scene feature and the second prior guidance scene feature based on the memory-enhanced prior features and the initial scene features.

4. The video restoration method according to claim 3, characterized in that, The pyramid-cascaded deformable convolutional alignment module includes a comprehensive alignment unit and multiple alignment units at different levels; For any set of scene features, the alignment unit is used to determine the feature offset of the corresponding level based on the first prior guided scene feature of the corresponding level and any set of scene features, as well as the feature offset of the next level, and to determine the alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, any set of scene features and the feature offset, as well as the alignment feature of the next level. The integrated alignment unit is used to determine the target alignment feature based on the first prior guiding scene features and alignment features of the initial level.

5. The video restoration method according to claim 4, characterized in that, The alignment unit is specifically used for: The first prior guiding scene feature of the corresponding level is concatenated with any scene feature set and then convolved to obtain the initial offset of the corresponding level. The initial offset of the corresponding level is then concatenated with the upsampled result of the feature offset of the next level and then convolved with multiple layers to obtain the feature offset of the corresponding level. The initial alignment features of the corresponding level are obtained by concatenating any scene feature set with the feature offset and performing deformable convolution. The initial alignment features of the corresponding level are then concatenated with the upsampled results of the alignment features of the next level and performed multi-layer convolution to obtain the alignment features of the corresponding level.

6. The video restoration method according to claim 4, characterized in that, The integrated alignment unit is specifically used for: The first prior guiding scene features and alignment features of the initial layer are concatenated and then subjected to multi-layer convolution to obtain the comprehensive offset. The comprehensive offset is subjected to deformable convolution to determine the target alignment features.

7. The video restoration method according to claim 2, characterized in that, The multi-range time feature aggregation module is specifically used for: The target alignment features are concatenated to obtain concatenated features; After performing average pooling and convolution on each of the target alignment features in sequence, the scale parameters are obtained by concatenation and global average pooling. The concatenated features are sequentially convolved and global average pooled to obtain the shift parameters; Based on the scaling parameter and the shift parameter, the concatenated features are aggregated to obtain the aggregation result, and the aggregation result is convolved to obtain the decoding result.

8. A video restoration device, characterized in that, include: The video acquisition module is used to acquire the video to be dehazed; The dehazing module is used to input the current video frame in the video to be dehazed and one or more historical video frames adjacent to the current video frame into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, cascaded encoders at different scales, a prior decoder, and a scene decoder, as well as an output layer. The prior decoders and scene decoders at the same scale are connected to the encoder. The input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The prior decoder includes a physical prior guidance module, and the scene decoder includes a pyramid-cascaded deformable convolution alignment module and a multi-range temporal feature aggregation module connected in sequence. The physical prior guidance module is connected to the pyramid-cascaded deformable convolution alignment module. The physical prior guidance module includes a prior compression module, a memory enhancement module, and a prior guidance module. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale; The target dehazing model is trained based on the video samples to be dehazed and the corresponding clear video samples, using the following steps to train the initial dehazing model: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample. Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed. Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss. Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model; The formula for calculating the deentanglement loss of the physical model is as follows: ； in, The entanglement loss is used to resolve the physical model. It is an L1 norm. These are current video frame samples at different scales. The fog map; The formula for calculating the model output loss is as follows: ； in, Output the loss for the model. The dehazed video frame sample, The clear video sample; The formula for calculating the total loss is as follows: ； in, The weights for the deentanglement loss of the physical model. The total loss is mentioned above.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the video restoration method as described in any one of claims 1-7.

Images