Video restoration method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of video processing, and provides a video restoration method and apparatus, an electronic device and a storage medium, and a target defogging model determines a priori guidance scene feature through a priori decoder, which not only can avoid error accumulation caused by component estimation, but also can adapt to various different scenes. Meanwhile, details and color reality of the video can be reserved as far as possible on the basis of removing fuzzy influence caused by haze. The scene decoder is used for carrying out space-time alignment on the priori guide scene features, so that global features and remote time information can be accurately moved, the defogging difficulty is reduced, the quality, visual effect, robustness and consistency of video restoration are improved, and clearer and more accurate video streams are provided for subsequent visual processing tasks.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the increasing severity of environmental problems, smog has become a common phenomenon in daily life. Smog greatly reduces the visibility and contrast of outdoor scenes, which has an adverse impact on downstream visual tasks of videos captured outdoors, such as target detection and segmentation in autonomous driving and surveillance.

[0003] Existing video restoration methods typically rely on component estimation from physical models for dehazing. However, errors are generated during component estimation and accumulate, leading to poor dehazing results. Furthermore, the parameters of physical models are usually fixed and cannot be adaptively adjusted for different scenes, potentially causing misjudgments of haze distribution and hindering accurate dehazing. Additionally, existing video restoration methods aggregate temporal information using feature stacking or frame-to-frame alignment within local sliding windows, making it difficult to accurately obtain global features and long-range temporal information. This not only increases the difficulty of dehazing but also negatively impacts the final result. Summary of the Invention

[0004] This invention provides a video restoration method, apparatus, electronic device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a video restoration method, comprising: Obtain the video to be dehazed; The current video frame in the video to be dehazed, along with one or more historical video frames adjacent to the current video frame, are input into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

[0006] According to a video restoration method provided by the present invention, the prior decoder includes a physical prior guidance module, and the scene decoder includes a pyramid cascade deformable convolution alignment module and a multi-range temporal feature aggregation module connected in sequence, wherein the physical prior guidance module is connected to the pyramid cascade deformable convolution alignment module. The physical prior guidance module is used to guide and enhance the initial scene features based on the initial prior features, so as to obtain the first prior guided scene features of the current video frame and the second prior guided scene features of the historical video frames. The pyramid-cascaded deformable convolutional alignment module is used to perform spatiotemporal alignment of the first prior guided scene features with the second prior guided scene features in different time ranges to obtain the target alignment features corresponding to each scene feature set. The multi-range temporal feature aggregation module is used to aggregate the alignment features of each target to obtain the decoding result.

[0007] According to a video restoration method provided by the present invention, the physical prior guidance module includes: a prior compression module, a memory enhancement module, and a prior guidance module; The prior compression module is used to classify and normalize the initial prior features, determine the transmission distribution map, and generate prior compression tags based on the initial prior features and the transmission distribution map. The memory enhancement module is used to store the initial prior features into the memory unit based on the prior compression tag, and to determine the memory enhancement prior features in the memory unit by applying the prior compression tag based on the attention mechanism. The prior guidance module is used to generate the first prior guidance scene feature and the second prior guidance scene feature based on the memory-enhanced prior features and the initial scene features.

[0008] According to a video restoration method provided by the present invention, the pyramid cascaded deformable convolutional alignment module includes a comprehensive alignment unit and multiple alignment units at different levels; For any set of scene features, the alignment unit is used to determine the feature offset of the corresponding level based on the first prior guided scene feature of the corresponding level and any set of scene features, as well as the feature offset of the next level, and to determine the alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, any set of scene features and the feature offset, as well as the alignment feature of the next level. The integrated alignment unit is used to determine the target alignment feature based on the first prior guiding scene features and alignment features of the initial level.

[0009] According to a video restoration method provided by the present invention, the alignment unit is specifically used for: The first prior guiding scene feature of the corresponding level is concatenated with any scene feature set and then convolved to obtain the initial offset of the corresponding level. The initial offset of the corresponding level is then concatenated with the upsampled result of the feature offset of the next level and then convolved with multiple layers to obtain the feature offset of the corresponding level. The initial alignment features of the corresponding level are obtained by concatenating any scene feature set with the feature offset and performing deformable convolution. The initial alignment features of the corresponding level are then concatenated with the upsampled results of the alignment features of the next level and performed multi-layer convolution to obtain the alignment features of the corresponding level.

[0010] According to a video restoration method provided by the present invention, the comprehensive alignment unit is specifically used for: The first prior guiding scene features and alignment features of the initial layer are concatenated and then subjected to multi-layer convolution to obtain the comprehensive offset. The comprehensive offset is subjected to deformable convolution to determine the target alignment features.

[0011] According to a video restoration method provided by the present invention, the multi-range temporal feature aggregation module is specifically used for: The target alignment features are concatenated to obtain concatenated features; After performing average pooling and convolution on each of the target alignment features in sequence, the scale parameters are obtained by concatenation and global average pooling. The concatenated features are sequentially convolved and global average pooled to obtain the shift parameters; Based on the scaling parameter and the shift parameter, the concatenated features are aggregated to obtain the aggregation result, and the aggregation result is convolved to obtain the decoding result.

[0012] According to a video restoration method provided by the present invention, the target dehazing model is trained based on the video sample to be dehazed and the corresponding clear video sample, by performing the following steps on an initial dehazing model: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample. Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed. Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss. Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model.

[0013] The present invention also provides a video restoration model, comprising: The video acquisition module is used to acquire the video to be dehazed; The dehazing module is used to input the current video frame in the video to be dehazed and one or more historical video frames adjacent to the current video frame into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video restoration method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video restoration method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video restoration method as described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The video restoration method, apparatus, electronic device, and storage medium provided by this invention determine prior-guided scene features through a prior decoder in the target dehazing model. This not only avoids error accumulation caused by component estimation but also adapts to various different scenarios. Simultaneously, it can preserve video details and color realism as much as possible while removing the blurring effect caused by haze. Spatiotemporal alignment of the prior-guided scene features through the scene decoder accurately activates global features and remote temporal information, reducing the difficulty of dehazing and improving the quality, visual effects, robustness, and consistency of video restoration, providing a clearer and more accurate video stream for subsequent visual processing tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0019] Figure 1 This is a flowchart illustrating the video restoration method provided by the present invention; Figure 2 This is a schematic diagram of the target dehazing model in the video restoration method provided by the present invention; Figure 3 This is a schematic diagram of the prior decoder structure in the video restoration method provided by the present invention; Figure 4 This is a schematic diagram of the physical prior guidance module in the video restoration method provided by the present invention; Figure 5 This is a schematic diagram of the structure of the pyramid-cascaded deformable convolution alignment module in the video restoration method provided by this invention; Figure 6This is a schematic diagram of the structure of the multi-range temporal feature aggregation module in the video restoration method provided by the present invention; Figure 7 This is a schematic diagram of the process of reconstructing fog maps during model training in the video restoration method provided by this invention; Figure 8 This is a schematic diagram of the video restoration device provided by the present invention; Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Figure 1 This is a flowchart illustrating the video restoration method provided in this embodiment of the invention, such as... Figure 1 As shown, the method includes: S1, Obtain the video to be dehazed; S2, input the current video frame in the video to be dehazed and one or more historical video frames adjacent to the current video frame into the target dehazing model to obtain the dehazed video frame of the current video frame; The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

[0022] Specifically, the video restoration method provided in this embodiment of the invention is executed by a video restoration device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0023] First, step S1 is executed to obtain the video to be dehazed. This means obtaining a video from which dehazing needs to be performed on each video frame; the video to be dehazed may include multiple video frames.

[0024] Then, step S2 is executed, where the current video frame and one or more adjacent historical video frames from the video to be dehazed are input into the target dehazing model to obtain the dehazed video frame of the current video frame. The number of adjacent historical video frames can be set as needed, for example, it can be set to 3. When using the target dehazing model to dehaze the current video frame to obtain the dehazed video frame, the current video frame and 3 adjacent historical video frames need to be input.

[0025] like Figure 2 As shown, the target dehazing model can include an input layer, encoders at different scales, a prior decoder and a scene decoder, and an output layer. The prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. Here, the encoder at each scale can include convolutional layers and pooling layers.

[0026] The input layer is used to receive the current video frame and historical video frames. If the current video frame is i, then the three historical video frames are i-1, i-2, and i-3, respectively.

[0027] After receiving each video frame at the input layer, each video frame can be downsampled. The reciprocal of the downsampling factor, s, is 1, indicating that the size of each video frame remains unchanged. Different scales can be obtained through downsampling operations with different downsampling factors.

[0028] exist Figure 2In this model, the target dehazing model can include four encoders at different scales, a prior decoder, and a scene decoder. The four encoders at different scales can be represented as Encoder Layer0, Encoder Layer1, Encoder Layer2, and Encoder Layer3, respectively. The reciprocals 's' of the downsampling factors 's' are 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. If 's' is the same, it indicates that the scale is the same. The four encoders at different scales are connected sequentially in descending order of 's'.

[0029] The four prior decoders at different scales can be represented as Prior Decoder Layer0, Prior Decoder Layer1, Prior Decoder Layer2 and Prior Decoder Layer3, respectively. The ratios s of the size of the image features at the corresponding scales to the size of each input video frame are 1 / 4, 1 / 8, 1 / 16 and 1 / 32, respectively.

[0030] Four scene decoders of different scales are connected sequentially in ascending order of 's'. Encoders with the same 's' are connected to the prior decoder and the scene decoder respectively, implementing a skip connection. The prior decoder with the same 's' is connected to the scene decoder. It can be understood that the initial scale refers to the size obtained by downsampling the current video frame for the first time.

[0031] Each encoder serves the same purpose: to extract image features at its corresponding scale. Each prior decoder serves the same purpose: to extract initial prior features at its corresponding scale. Each scene decoder serves the same purpose: to extract initial scene features at its corresponding scale.

[0032] The prior decoder is also used to enhance the initial scene features by utilizing the initial prior features, thereby obtaining prior guided scene features at the corresponding scale.

[0033] The scene decoder is also used to spatiotemporally align prior guided scene features of the corresponding scale to obtain target alignment features, and to determine the decoding result of the corresponding scale using the target alignment features.

[0034] The output layer is used to determine the dehazed video frames using the current and historical video frames received from the input layer, as well as the decoding results determined by the scene decoder at the initial scale. For example... Figure 2As shown, the decoding result determined by the scene decoder at the initial scale can be upsampled first to obtain a decoding result with the same size as the current video frame. Then, the current video frame and historical video frames are superimposed with the upsampled decoding result to obtain the dehazing result. Since the input layer receives the current video frame and historical video frames, the dehazing result includes dehazed video frames from both the current and historical video frames.

[0035] The video restoration method provided in this embodiment of the invention uses a priori decoder to determine prior-guided scene features in the target dehazing model. This not only avoids error accumulation caused by component estimation but also adapts to various different scenarios. Simultaneously, it can preserve video details and color realism as much as possible while removing the blurring effect caused by haze. Spatiotemporal alignment of the prior-guided scene features using a scene decoder accurately activates global features and remote temporal information, reducing the difficulty of dehazing and improving the quality, visual effects, robustness, and consistency of video restoration. This provides a clearer and more accurate video stream for subsequent visual processing tasks.

[0036] Based on the above embodiments, such as Figure 3 As shown, the prior decoder includes a physical prior guidance module, and the scene decoder includes a pyramid cascaded deformable convolution alignment module and a multi-range temporal feature aggregation module connected in sequence. The physical prior guidance module is connected to the pyramid cascaded deformable convolution alignment module. The physical prior guidance module is used to guide and enhance the initial scene features based on the initial prior features, so as to obtain the first prior guided scene features of the current video frame and the second prior guided scene features of the historical video frames. The pyramid-cascaded deformable convolutional alignment module is used to perform spatiotemporal alignment of the first prior guided scene features with the second prior guided scene features in different time ranges to obtain the target alignment features corresponding to each scene feature set. The multi-range temporal feature aggregation module is used to aggregate the alignment features of each target to obtain the dehazed video frame.

[0037] Specifically, such as Figure 3 As shown, each scale's Prior Decoder includes an Initial Prior Decoder module and a Physical Prior Guidance module (MPG), and each scale's Scene Decoder includes an Initial Scene Decoder module, a Pyramid Cascaded Deformable Convolution Alignment module (PCD), and a Multi-Range Temporal Feature Aggregation module (MTA) connected in sequence. MPG and PCD are connected.

[0038] The input to the initial prior decoding module can include the output of the next-scale prior decoder (i.e., Input1) and the output of the encoder at the same scale (i.e., Input2). The initial prior decoding module can first upsample Input1, then add the upsampled result pixel-by-pixel to Input2 and perform convolution to obtain the initial prior features. .

[0039] The input to the initial scene decoding module can include the output of the next-scale scene decoder (i.e., Input3) and the output of the encoder at the same scale (i.e., Input4). The initial scene decoding module can first upsample Input3, and then add the upsampled result to Input4 pixel by pixel to obtain the initial scene features. .

[0040] MPG can utilize initial prior features For the initial scene features Enhanced guidance is performed to obtain prior guidance scene features at the corresponding scale. Here, the prior guiding scene features can include the first prior guiding scene features of the current video frame and the second prior guiding scene features of historical video frames.

[0041] PCD can utilize deformable convolution to spatiotemporally align the first prior guided scene features with scene feature sets from different time ranges within the second prior guided scene features, obtaining the target aligned features corresponding to each scene feature set. Here, PCD can first divide the second prior guided scene features into scene feature sets from different time ranges. For example, if the historical video frames include 3, the second prior guided scene features can be represented as follows: Let i be the current video frame, i-1 be the first historical video frame before the current video frame, i-2 be the second historical video frame before the current video frame, and i-3 be the third historical video frame before the current video frame. Furthermore, the set of scene features for different time ranges can be represented as... .

[0042] MTA can perform weighted aggregation of the alignment features of each target to obtain the decoding result at the current scale.

[0043] In this embodiment of the invention, the accuracy of prior-guided scene features can be improved through a physical prior guidance module. Combining the physical prior guidance module with the pyramid-cascaded deformable convolution alignment module further avoids the shortcomings of existing technologies in accurately obtaining panoramic features and remote temporal information, reducing the difficulty of dehazing and improving the dehazing effect.

[0044] Based on the above embodiments, the physical prior guidance module includes: a prior compression module, a memory enhancement module, and a prior guidance module; The prior compression module is used to classify and normalize the initial prior features, determine the transmission distribution map, and generate prior compression tags based on the initial prior features and the transmission distribution map. The memory enhancement module is used to store the initial prior features into the memory unit based on the prior compression tag, and to determine the memory enhancement prior features in the memory unit by applying the prior compression tag based on the attention mechanism. The prior guidance module is used to generate the first prior guidance scene feature and the second prior guidance scene feature based on the memory-enhanced prior features and the initial scene features.

[0045] Specifically, such as Figure 4 As shown, the physics prior guidance module includes: prior compression module a, memory enhancement module b, and prior guidance module c; Prior compression module a is used to compress initial prior features The transmission distribution map is determined by classifying the data and normalizing it using the Softmax function. Where H and W are the height and width of the initial prior feature, respectively, C is the number of channels in the initial prior feature, and D is the number of categories in the initial prior feature, with D being less than C. Each element in the transmission distribution map represents the probability of the pixel at the corresponding position in the initial prior feature being distributed among the categories.

[0046] Subsequently, the initial prior features and the transmission distribution map are multiplied by matrix to generate prior compressed labels. Prior compression tags can be used to determine which category has the highest percentage of pixels in each channel. The rows of the prior compression tag represent the number of categories, and the columns represent the number of channels. Each point in the prior compression tag records the total score of all pixels in that category for each channel.

[0047] Memory enhancement module b can utilize prior compression tags p , initial prior features The data is stored in a Prior Token Memory. This memory can store historical smog information from different times. The feature vector in the memory is represented by a key K and a value V, with dimensions [missing information]. Where N is the number of times the data is stored, i.e., the number of dehazed video frames in the video to be processed.

[0048] To enable the interaction between current and historical haze information, an attention mechanism is used, applying prior compression labeling.p Read memory-enhancing prior features from memory units P This memory-enhancing prior feature P It can represent global and local physical information.

[0049] The formula for calculating the attention mechanism is as follows: ; in, To query features, it is to use the initial prior features. Obtained by flattening; It serves as the key for prior historical memories within a memory unit; The value of the prior historical memory in the memory unit; This is a scaling factor used to stabilize values ​​and prevent the gradient from being too large or too small.

[0050] Prior guidance module c can enhance prior features through memory. P Features of the initial scene The features are concatenated and processed using multiple convolutional layers to obtain prior guided scene features at the corresponding scale. Due to the initial scene characteristics It contains rich texture, edge, and semantic context information. Prior features are enhanced by memory. P Features of the initial scene By concatenating these layers, physical model guidance information and semantic scene information can be integrated in the feature space. Multiple convolutional layers further learn the complex relationship between the two, extracting important joint features layer by layer—that is, prior guiding scene features—to generate a more accurate dehazed scene representation. This design ensures that the dehazing results conform to the physical laws of atmospheric scattering models while restoring delicate scene details, thus improving the overall quality and consistency of dehazed video frames.

[0051] Based on the above embodiments, the pyramid-cascaded deformable convolution alignment module includes a comprehensive alignment unit and multiple alignment units at different levels; For any set of scene features, the alignment unit is used to determine the feature offset of the corresponding level based on the first prior guided scene feature of the corresponding level and any set of scene features, as well as the feature offset of the next level, and to determine the alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, any set of scene features and the feature offset, as well as the alignment feature of the next level. The integrated alignment unit is used to determine the target alignment feature based on the first prior guiding scene features and alignment features of the initial level.

[0052] Specifically, such as Figure 5As shown, PCD aligns features in a coarse-to-fine manner to handle large and complex motions. Using a pyramid structure, lower-scale features are first aligned with a coarse estimate, and then the offset and aligned features are propagated to higher scales to facilitate accurate motion compensation. Furthermore, an additional deformable convolution is cascaded after the pyramid alignment operation to further improve the robustness of the alignment.

[0053] The first prior guided scene feature can be represented as Target J, and the second prior guided scene feature can be represented as Neighbor J. PCD can include a comprehensive alignment unit and multiple alignment units at different levels, and each alignment unit can be represented as L1, L2 and L3 respectively.

[0054] For any set of scene features, the alignment unit can determine the feature offset of the corresponding level using the first prior guiding scene feature of the corresponding level, any set of scene features, and the feature offset of the next level. The first prior guiding scene feature of L1 is Target J, and any set of scene features in L1 is any set of scene features in Neighbor J. The first prior guiding scene feature of L2 can be obtained by downsampling Target J using a strided convolutional filter, and any set of scene features in L2 can be obtained by downsampling any set of scene features in Neighbor J using a strided convolutional filter. The first prior guiding scene feature of L3 can be obtained by downsampling the first prior guiding scene feature of L2 using a strided convolutional filter, and any set of scene features in L3 can be obtained by downsampling any set of scene features in L2 using a strided convolutional filter. Here, the downsampling factor is 2, which reduces the feature size to half its original size. Furthermore, to avoid increasing computational cost, no additional channels are added during downsampling.

[0055] Each alignment unit at each level concatenates the first prior guiding scene feature of the corresponding level with any set of scene features and then performs a convolution to obtain the initial offset of the corresponding level. The initial offset of the corresponding level is then concatenated with the upsampled feature offset of the next level and subjected to multiple convolutions to obtain the feature offset (Offset) of the corresponding level.

[0056] At the L3 level, the formula for calculating the feature offset is: ; in, This represents the feature offset at the L3 level. This is a general function consisting of multiple convolutional layers, which implements a feature fusion operation. This is a concatenation operation. k is the number of historical video frames.

[0057] In the Lth level (excluding the last level), to generate the feature offset for the Lth level, the upsampled feature offset from the next (L+1)th level is also used. The generation process of the feature offset for the Lth level is represented by the following formula: ; in, Here, L represents the feature offset at level L, where L can take values ​​of L1 and L2. The feature offset for level L+1. This indicates double upsampling, which is the same as downsampling and does not add any additional channels. This represents the set of scene features for k historical video frames.

[0058] The alignment feature of each level can be obtained by concatenating any scene feature set of the corresponding level with the feature offset and then performing deformable convolution to obtain the initial alignment feature of the corresponding level. The initial alignment feature of the corresponding level is then concatenated with the upsampled result of the alignment feature of the next level and then performed multi-layer convolution to obtain the alignment feature of the corresponding level.

[0059] Here, the scene feature set and feature offset of the k historical video frames at this level are concatenated, then processed by deformable convolution (DComv), and combined with the upsampling result of the alignment features of the next level to generate the following process: ; in, For L-level alignment features, It is a general function composed of multiple convolutional layers. This represents deformable convolution.

[0060] The alignment features of the initial layer can be represented as: .

[0061] in, Alignment features for the initial level.

[0062] The comprehensive alignment unit can determine the target alignment feature by utilizing the first prior guiding scene feature and alignment feature of the initial layer. That is, the target alignment feature is obtained by concatenating the first prior guiding scene feature and alignment feature of the initial layer and performing multi-layer convolution to obtain the comprehensive offset, and then performing deformable convolution on the comprehensive offset.

[0063] The target alignment feature can be represented as: ; in, The target alignment features include three in total. The target alignment features are the features corresponding to the first set of scene features. The target alignment features are the features corresponding to the second scene feature set. This refers to the target alignment feature corresponding to the third scene feature set.

[0064] Based on the above embodiments, the multi-range time feature aggregation module is specifically used for: The target alignment features are concatenated to obtain concatenated features; After performing average pooling and convolution on each of the target alignment features in sequence, the scale parameters are obtained by concatenation and global average pooling. The concatenated features are sequentially convolved and global average pooled to obtain the shift parameters; Based on the scaling parameter and the shift parameter, the concatenated features are aggregated to obtain the aggregation result, and the aggregation result is convolved to obtain the decoding result.

[0065] Specifically, such as Figure 6 As shown, the MTA can , and By concatenating the series connections, we can obtain the series characteristics. .

[0066] Then, each target alignment feature is sequentially subjected to average pooling and convolution, followed by concatenation and global average pooling to obtain the scaling parameters. That is: ; in, For scaling parameters, For global average pooling, The convolution kernel is 3×3. The average pooling operation is performed for each channel, keeping the number of channels constant, where C is the number of channels.

[0067] The concatenated features are sequentially convolved and then subjected to global average pooling to obtain the shift parameters. That is: .

[0068] Finally, the concatenated features are aggregated using scaling and shift parameters to obtain the aggregated result. This aggregated result is then convolved to obtain the decoding result at the corresponding scale. That is: ; in, This is the decoding result for the corresponding scale.

[0069] Based on the above embodiments, the target dehazing model is trained by performing the following steps on the initial dehazing model based on the video sample to be dehazed and the corresponding clear video sample: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample. Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed. Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss. Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model.

[0070] Specifically, the target dehazing model can be obtained by training an initial dehazing model based on the video samples to be dehazed and the clear video samples corresponding to the video samples to be dehazed.

[0071] The training steps may include: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature samples extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame samples of the current video frame sample.

[0072] After that, as Figure 7 As shown, transmittance is predicted using the initial prior feature sample Output1. and atmospheric light value and utilize transmittance and atmospheric light value The fog map was reconstructed using the dehazed video frame sample Output2 and an atmospheric scattering model. The initial dehazing model can obtain initial prior feature samples at different scales, which in turn can produce haze maps at different scales. .

[0073] Using the fog map and current video frame samples, calculate the entanglement disentanglement loss of the physical model. That is: ; in, De-entanglement loss for the physical model, It is an L1 norm. Different scales of current video frame samples can be obtained by downsampling the current video frame samples with different sampling factors.

[0074] Subsequently, using clear video samples and dehazed video frame samples, the model output loss is calculated, namely: ; in, For dehazing video frame samples, For clear video samples.

[0075] The total loss can be determined by weighted summing of the deentanglement loss of the physical model and the model output loss. That is: ; in, Weights for the deentanglement loss in the physical model. This represents the total loss.

[0076] Finally, the initial dehazing model is iteratively trained using the total loss until the total loss converges or the preset number of iterations is reached, thus obtaining the target dehazing model. The preset number of iterations can be set as needed and is not specifically limited here.

[0077] like Figure 8 As shown, based on the above embodiments, this embodiment of the invention provides a video restoration model, including: Video acquisition module 81 is used to acquire the video to be dehazed; The dehazing module 82 is used to input the current video frame in the video to be dehazed and one or more historical video frames adjacent to the current video frame into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

[0078] Based on the above embodiments, the video restoration model provided in this embodiment of the invention includes a physical prior guide module in the prior decoder and a scene decoder including a pyramid cascade deformable convolution alignment module and a multi-range temporal feature aggregation module connected in sequence. The physical prior guide module is connected to the pyramid cascade deformable convolution alignment module. The physical prior guidance module is used to guide and enhance the initial scene features based on the initial prior features, so as to obtain the first prior guided scene features of the current video frame and the second prior guided scene features of the historical video frames. The pyramid-cascaded deformable convolutional alignment module is used to perform spatiotemporal alignment of the first prior guided scene features with the second prior guided scene features in different time ranges to obtain the target alignment features corresponding to each scene feature set. The multi-range temporal feature aggregation module is used to aggregate the alignment features of each target to obtain the decoding result.

[0079] Based on the above embodiments, the video restoration model provided in this embodiment of the invention includes a physical prior guidance module comprising: a prior compression module, a memory enhancement module, and a prior guidance module; The prior compression module is used to classify and normalize the initial prior features, determine the transmission distribution map, and generate prior compression tags based on the initial prior features and the transmission distribution map. The memory enhancement module is used to store the initial prior features into the memory unit based on the prior compression tag, and to determine the memory enhancement prior features in the memory unit by applying the prior compression tag based on the attention mechanism. The prior guidance module is used to generate the first prior guidance scene feature and the second prior guidance scene feature based on the memory-enhanced prior features and the initial scene features.

[0080] Based on the above embodiments, the video restoration model provided in this embodiment of the invention includes a pyramid cascaded deformable convolution alignment module comprising a comprehensive alignment unit and multiple alignment units at different levels. For any set of scene features, the alignment unit is used to determine the feature offset of the corresponding level based on the first prior guided scene feature of the corresponding level and any set of scene features, as well as the feature offset of the next level, and to determine the alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, any set of scene features and the feature offset, as well as the alignment feature of the next level. The integrated alignment unit is used to determine the target alignment feature based on the first prior guiding scene features and alignment features of the initial level.

[0081] Based on the above embodiments, the video restoration model provided in this embodiment of the invention, wherein the alignment unit is specifically used for: The first prior guiding scene feature of the corresponding level is concatenated with any scene feature set and then convolved to obtain the initial offset of the corresponding level. The initial offset of the corresponding level is then concatenated with the upsampled result of the feature offset of the next level and then convolved with multiple layers to obtain the feature offset of the corresponding level. The initial alignment features of the corresponding level are obtained by concatenating any scene feature set with the feature offset and performing deformable convolution. The initial alignment features of the corresponding level are then concatenated with the upsampled results of the alignment features of the next level and performed multi-layer convolution to obtain the alignment features of the corresponding level.

[0082] Based on the above embodiments, the video restoration model provided in this embodiment of the invention, wherein the comprehensive alignment unit is specifically used for: The first prior guiding scene features and alignment features of the initial layer are concatenated and then subjected to multi-layer convolution to obtain the comprehensive offset. The comprehensive offset is subjected to deformable convolution to determine the target alignment features.

[0083] Based on the above embodiments, the video restoration model provided in this embodiment of the invention, wherein the multi-range temporal feature aggregation module is specifically used for: The target alignment features are concatenated to obtain concatenated features; After performing average pooling and convolution on each of the target alignment features in sequence, the scale parameters are obtained by concatenation and global average pooling. The concatenated features are sequentially convolved and global average pooled to obtain the shift parameters; Based on the scaling parameter and the shift parameter, the concatenated features are aggregated to obtain the aggregation result, and the aggregation result is convolved to obtain the decoding result.

[0084] Based on the above embodiments, the video restoration model provided in this embodiment of the invention further includes a training module, used for: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample. Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed. Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss. Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model.

[0085] Specifically, the functions of each module in the video restoration device provided in this embodiment correspond one-to-one with the operation flow of each step in the above method embodiment, and the achieved effect is also the same. Please refer to the above embodiments for details, and this will not be repeated in this embodiment.

[0086] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the video restoration methods provided in the above embodiments.

[0087] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the video restoration methods provided in the above embodiments.

[0089] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the video restoration methods provided in the above embodiments.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video restoration method, characterized in that, include: Obtain the video to be dehazed; The current video frame in the video to be dehazed, along with one or more historical video frames adjacent to the current video frame, are input into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

2. The video restoration method according to claim 1, characterized in that, The prior decoder includes a physical prior guidance module, and the scene decoder includes a pyramid cascaded deformable convolution alignment module and a multi-range temporal feature aggregation module connected in sequence. The physical prior guidance module is connected to the pyramid cascaded deformable convolution alignment module. The physical prior guidance module is used to guide and enhance the initial scene features based on the initial prior features, so as to obtain the first prior guided scene features of the current video frame and the second prior guided scene features of the historical video frames. The pyramid-cascaded deformable convolutional alignment module is used to perform spatiotemporal alignment of the first prior guided scene features with the second prior guided scene features in different time ranges to obtain the target alignment features corresponding to each scene feature set. The multi-range temporal feature aggregation module is used to aggregate the alignment features of each target to obtain the decoding result.

3. The video restoration method according to claim 2, characterized in that, The physical prior guidance module includes: a prior compression module, a memory enhancement module, and a prior guidance module; The prior compression module is used to classify and normalize the initial prior features, determine the transmission distribution map, and generate prior compression tags based on the initial prior features and the transmission distribution map. The memory enhancement module is used to store the initial prior features into the memory unit based on the prior compression tag, and to determine the memory enhancement prior features in the memory unit by applying the prior compression tag based on the attention mechanism. The prior guidance module is used to generate the first prior guidance scene feature and the second prior guidance scene feature based on the memory-enhanced prior features and the initial scene features.

4. The video restoration method according to claim 3, characterized in that, The pyramid-cascaded deformable convolutional alignment module includes a comprehensive alignment unit and multiple alignment units at different levels; For any set of scene features, the alignment unit is used to determine the feature offset of the corresponding level based on the first prior guided scene feature of the corresponding level and any set of scene features, as well as the feature offset of the next level, and to determine the alignment feature of the corresponding level based on the first prior guided scene feature of the corresponding level, any set of scene features and the feature offset, as well as the alignment feature of the next level. The integrated alignment unit is used to determine the target alignment feature based on the first prior guiding scene features and alignment features of the initial level.

5. The video restoration method according to claim 4, characterized in that, The alignment unit is specifically used for: The first prior guiding scene feature of the corresponding level is concatenated with any scene feature set and then convolved to obtain the initial offset of the corresponding level. The initial offset of the corresponding level is then concatenated with the upsampled result of the feature offset of the next level and then convolved with multiple layers to obtain the feature offset of the corresponding level. The initial alignment features of the corresponding level are obtained by concatenating any scene feature set with the feature offset and performing deformable convolution. The initial alignment features of the corresponding level are then concatenated with the upsampled results of the alignment features of the next level and performed multi-layer convolution to obtain the alignment features of the corresponding level.

6. The video restoration method according to claim 4, characterized in that, The integrated alignment unit is specifically used for: The first prior guiding scene features and alignment features of the initial layer are concatenated and then subjected to multi-layer convolution to obtain the comprehensive offset. The comprehensive offset is subjected to deformable convolution to determine the target alignment features.

7. The video restoration method according to claim 2, characterized in that, The multi-range time feature aggregation module is specifically used for: The target alignment features are concatenated to obtain concatenated features; After performing average pooling and convolution on each of the target alignment features in sequence, the scale parameters are obtained by concatenation and global average pooling. The concatenated features are sequentially convolved and global average pooled to obtain the shift parameters; Based on the scaling parameter and the shift parameter, the concatenated features are aggregated to obtain the aggregation result, and the aggregation result is convolved to obtain the decoding result.

8. The video restoration method according to any one of claims 1-7, characterized in that, The target dehazing model is trained based on the video samples to be dehazed and the corresponding clear video samples, using the following steps to train the initial dehazing model: The current video frame sample and one or more historical video frame samples adjacent to the current video frame sample are input into the initial dehazing model to obtain the initial prior feature sample extracted by the initial prior decoder in the initial dehazing model and the dehazed video frame sample of the current video frame sample. Based on the initial prior feature samples, the transmittance and atmospheric light value are predicted, and based on the transmittance and atmospheric light value, as well as the dehazed video frame samples, the fog map is reconstructed. Based on the fog map and the current video frame sample, calculate the physical model de-entanglement loss, and based on the clear video sample and the de-fog video frame sample, calculate the model output loss. Based on the physical model de-entanglement loss and the model output loss, determine the total loss. Based on the total loss, the initial dehazing model is iteratively trained to obtain the target dehazing model.

9. A video restoration model, characterized in that, include: The video acquisition module is used to acquire the video to be dehazed; The dehazing module is used to input the current video frame in the video to be dehazed and one or more historical video frames adjacent to the current video frame into the target dehazing model to obtain the dehazed video frame of the current video frame. The target dehazing model includes an input layer, encoders cascaded at different scales, a prior decoder and a scene decoder, and an output layer; the prior decoder and scene decoder at the same scale are both connected to the encoder; the input layer is connected to the encoder at the initial scale, and the output layer is connected to both the input layer and the scene decoder at the initial scale. The input layer is used to receive the current video frame and the historical video frames; The encoder is used to extract image features at the corresponding scale; the prior decoder is used to extract initial prior features at the corresponding scale; the scene decoder is used to extract initial scene features at the corresponding scale. The prior decoder is also used to guide and enhance the initial scene features based on the initial prior features to obtain prior guided scene features of the corresponding scale. The scene decoder is also used to perform spatiotemporal alignment of the prior guided scene features at the corresponding scale to obtain target alignment features, and to determine the decoding result at the corresponding scale based on the target alignment features; The output layer is used to determine the dehazed video frame based on the current video frame, the historical video frames, and the decoding result determined by the scene decoder at the initial scale.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the video restoration method as described in any one of claims 1-8.

Images