A temporally stable blind degraded video restoration method

Abstract

This invention discloses a temporally stable blindly degraded video restoration method. To address the temporal instability problem of blindly degraded video restoration, the method first removes blindly degraded noise based on the degradation level of the blindly degraded video to obtain smoothed frame content. Next, the smoothed frame content is analyzed from three perspectives: the target object, instance semantics, and overall description, providing comprehensive prior knowledge for video restoration. Simultaneously, the spatiotemporal relationships of the latent space features of the smoothed frame content are analyzed to obtain spatiotemporally aware features. This comprehensive prior knowledge is then integrated into the spatiotemporally aware features, and a high-quality feature map is predicted in multiple steps. Finally, the high-quality feature map is mapped to the image space to restore high-quality video. This invention achieves modeling of real-world blindly degraded videos and restores high-definition and temporally coherent video content.

Description

Technical Field

[0001] The present invention relates to the technical field of blind degradation data processing, and in particular to a temporally stable blind degradation video restoration method. Background Art

[0002] In modern society, video has become an essential means of information exchange and recording. However, in various application scenarios, such as old videos on the Internet and blurry videos in security systems, video quality often suffers from varying degrees of degradation. This degradation can be caused by compression, transmission loss, low-resolution camera equipment, or harsh environmental conditions. Video can be affected by multiple degradation types simultaneously, such as blur, noise, compression artifacts, and color distortion, and these types of degradation can interact with each other, further complicating the recovery process. When data is simultaneously subject to multiple degradation modes without knowing the specific degradation patterns, this is called "blind degradation."

[0003] Existing blind degradation restoration methods are designed for images. They fail to consider the inter-frame correlation in videos and therefore cannot be directly applied to the restoration of blindly degraded videos. On the other hand, existing video restoration methods typically target specific degradation patterns, such as super-resolution and deblurring. While these methods perform well on their specific degraded data, they often rely on known degradation models and struggle to cope with various combinations of unknown degradations, making them difficult to directly apply to real-world scenarios. For example, super-resolution focuses solely on improving video resolution but offers limited improvement in blur or noise degradation. Deblurring, on the other hand, is effective only for motion blur and defocus blur but is less effective in the face of compression losses or low resolution. Therefore, developing a blindly degraded video restoration technique that can handle a variety of unknown degradation types is crucial for improving video quality and enhancing information acquisition. Such a technique must not only be effective under a wide range of degradation conditions but also be robust and versatile to meet the needs of diverse application scenarios. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies and shortcomings of the existing technology and provide a temporally stable blind degraded video restoration method to achieve modeling of real-world blind degraded videos and restore high-definition and temporally coherent video content.

[0005] To achieve the above object, the present invention provides a technical solution: a method for restoring time-stable blindly degraded video, comprising the following steps:

[0006] 1) Obtain the blindly degraded video to be restored and evenly extract blindly degraded frames. Perform the following operations on each blindly degraded frame:

[0007] 1.1) Extract latent space features from the blindly degraded frames and decompose them into different frequency bands using discrete wavelet transform. Based on the degree of degradation of the blindly degraded frames, filter the content of each frequency band to remove the included blindly degraded noise. Finally, use inverse discrete wavelet transform to recombine the filtered content of each frequency band to obtain the smoothed frame content.

[0008] 1.2) Analyze the smoothed frame content from three perspectives: target object, instance semantics, and overall description. This identifies the object's location and semantic region, and obtains a textual description of the smoothed frame content. By fusing information related to the object's location, semantic region, and the smoothed frame's content description, a comprehensive content understanding representation is generated, providing comprehensive understanding prior knowledge for subsequent recovery.

[0009] 1.3) extracting latent space features from the smooth frame content obtained in step 1.1), and analyzing the spatiotemporal relationship of the extracted latent space features through a self-attention mechanism to obtain features with spatiotemporal perception; then, integrating the comprehensive understanding prior knowledge obtained in step 1.2) into the acquired spatiotemporal perception features through a cross-attention mechanism; then, predicting a high-quality feature map in multiple steps, i.e., determining which high-quality feature vectors in a pre-constructed high-quality feature pool the smooth frame content corresponds to, and using these high-quality feature vectors to characterize the smooth frame content, wherein the high-quality feature pool contains multiple high-quality feature vectors that can express high-quality video semantic content, and each high-quality feature vector has the same dimension; in each prediction step, first selecting a high-confidence prediction result, and extracting the corresponding high-quality feature vector from the high-quality feature pool to replace the latent space features of the smooth frame content, and then continuing to iteratively predict the remaining latent space features that have not been replaced until all latent space features of the smooth frame content are replaced by high-quality feature vectors in the high-quality feature pool, thereby obtaining a complete high-quality feature map, and finally mapping the complete high-quality feature map to the image space to obtain a restored high-definition frame;

[0010] 2) Evenly extracting blind degraded frames from the blindly degraded video to be restored, splicing the restored high-definition frames with the extracted blindly degraded frames in the temporal dimension, and using attention weighting to enhance the attention of the blindly degraded frames to the content of the restored high-definition frames during the restoration process, thereby ensuring consistency between the restored result and the content of the high-definition frames. The blindly degraded frames are restored by executing steps 1.1) to 1.3) above.

[0011] 3) Repeat step 2) until all blindly degraded frames in the blindly degraded video to be restored are restored, and finally generate coherent and high-definition video content.

[0012] Furthermore, in step 1.1), a video encoder is used to extract latent space features from the blind degraded frame. The video encoder is composed of a stack of multiple 3D convolutional layers, which decomposes the latent space features of the blind degraded frame into content in different frequency bands:

[0013] {F1,F2,...F f ,...,F B =DWT(Z lq )

[0014] Where Z lq represents the latent space features of the blind degraded frame, and DWT represents discrete wavelet transform, which is used to decompose the input content into content of different frequency bands {F1, F2, ... F f ,...,F B}, B represents the total number of decomposed frequency bands, and the symbol f is used to represent the frequency band number. The larger the number, the higher the frequency band, and the smaller the number, the lower the frequency band. That is, F1 represents the lowest frequency band content, and F B Indicates the content of the highest frequency band; in each frequency band, the cross-attention mechanism is used to analyze the degree of content degradation of the frequency band based on the content of the current frequency band and the reference content lower than the current frequency band. The degree of degradation analysis is expressed as follows:

[0015]

[0016] Where, F f and F f-1 represents the content of the fth and f-1th frequency bands in the latent space feature, Q f , K f-1 、V f-1 Represents the query features, key features, and value features of the corresponding frequency band in the cross-attention mechanism, represents the learnable parameters of the corresponding frequency band, u represents the dimension of the key feature of the cross-attention mechanism in the degradation degree analysis, δ represents the function used to analyze the degree of content degradation, which outputs the filter coefficient t for the frequency band f The greater the degree of degradation, the smaller the filter coefficient, the minimum value is 0, the smaller the degree of degradation, the larger the filter coefficient, the maximum value is 1, the filter coefficient of the frequency band is multiplied by the content of the frequency band to obtain the filtered content F of the frequency band f ', expressed as F f '=t f F f When analyzing the degradation of the lowest frequency band content F1, the self-attention mechanism is used without referring to the content of other frequency bands. Finally, the filtered content of each frequency band is recombined through the inverse discrete wavelet transform:

[0017] Z'=IDWT(F1',F2',...F f ',...,FB '),

[0018] Where IDWT represents the inverse discrete wavelet transform, which is used to transform the filtered content F1', F2', ... F in each frequency band f ',...,F B 'Recombine to obtain the filtered latent space feature Z'; finally, use the video decoder to map the latent space feature Z' to the image space to obtain smooth frame content, and the video decoder is composed of multiple stacked 3D convolutional layers.

[0019] Furthermore, in step 1.2), the BLIP-2 network is used to obtain text describing the content of the smooth frame, the YOLOv8 detection network is used to detect the target object in the smooth frame content, and the SAM semantic segmentation network is used to segment the instance semantics in the smooth frame content; the T5 network is used to obtain text features from the text describing the content of the smooth frame, and the deep features of target detection and semantic segmentation are extracted from the YOLOv8 detection network and the SAM semantic segmentation network respectively. Then, based on the graph network, the text features, target detection and semantic segmentation deep features are fused to obtain a comprehensive content understanding representation, which provides comprehensive understanding prior knowledge for subsequent recovery.

[0020] Furthermore, we integrate the deep features of text features, object detection, and semantic segmentation based on the graph network, as follows:

[0021] Construct an undirected graph in Represents a node set of an undirected graph, which includes n local local nodes and a global node v global , v1 represents the first local node, and so on, v nlocal Indicates the nth local local nodes, Represents the edge set of an undirected graph; the local node is mapped by the deep features corresponding to the detection box generated by the YOLOv8 detection network and the deep feature maps corresponding to the segmentation area generated by the SAM semantic segmentation network, and the global node v global The text feature representation extracted by the T5 network is connected to all local nodes as the central node. The nodes corresponding to the detection boxes of the same detection category are connected through the graph network to obtain a connection edge. The detection confidence value of the YOLOv8 detection network is used to define the weight of the connection edge. The nodes corresponding to the segmentation areas of the same segmentation category are connected to obtain a connection edge. The segmentation confidence value of the SAM segmentation network is used to define the weight of the connection edge. The information of all node features is transferred through graph convolution. The graph convolution process is defined as:

[0022]

[0023] Where, ω ij Represents the i-th local node v i With the jth local node v j The fixed connection weight, h j Represents the jth local node v j Characteristics, h i represents the i-th local node v i The features of , Λ represents the learnable weight matrix in graph convolution, Represents the i-th local node v i The set of adjacent nodes, σ represents the activation function; through multi-layer graph convolution operations, the deep features of text features, target detection and semantic segmentation are gradually integrated into the global node, and finally from the global node v global Get a comprehensive understanding of prior knowledge Z prior =Φ(v global ), where φ represents a linear mapping function that transforms v global Projection to Z prior .

[0024] Furthermore, in step 1.3), the latent space feature Z of the smoothed frame content is given, and the feature Z with spatiotemporal perception is obtained based on the self-attention mechanism. space ; Then, based on the cross attention mechanism, the feature Z with spatiotemporal perception space and comprehensive understanding of prior knowledge Z prior The fusion is performed as follows:

[0025]

[0026] Where Q, K, and V are the query features, key features, and value features in the cross-attention mechanism, respectively, and W Q 、W K 、W V is a learnable parameter, d is the dimension of feature K; based on the fused feature Z fuse , predict a high-quality feature map in multiple steps The high-quality feature map With Z fuse With the same dimension as Z, the prediction formula is as follows:

[0027]

[0028] Where, represents the prediction network, Represents the confidence of predicting the mth high-quality feature vector in the high-quality feature pool, N represents the size of the high-quality feature pool, Represents feature Z fuseThe position in the latent space feature Z is the value of the f-th frame, h-th row, and w-th column. According to the above prediction formula, the f-th frame, h-th row, and w-th column position in the latent space feature Z corresponds to the l-th high-quality feature vector in the high-quality feature pool. Therefore, the l-th high-quality feature vector is taken from the high-quality feature pool to replace the value of the corresponding position in Z. When the values ​​of all positions in Z are replaced, a high-quality feature map can be obtained.

[0029] Furthermore, the high-quality feature vectors in the high-quality feature pool are learned based on the reconstruction of high-quality HD videos and images by a 3D VQGAN network. The 3D VQGAN network includes a 3D VQGAN encoder and a 3D VQGAN decoder. The high-quality feature vectors in the high-quality feature pool are divided into high-quality video feature vectors and high-quality image feature vectors. When an image is input, only the high-quality image feature vector part is updated. When a video is input, only the high-quality video feature vector part is updated. After the high-quality feature pool completes learning, a high-quality feature map is obtained through step 1.3). Then, under the action of the 3DVQGAN decoder, Mapping to image space to obtain the restored high-definition frame

[0030] Further, in step 2), the degraded frame V to be restored is given LQ And the HD frame recovered from step 1) Extract the corresponding latent space features and Spliced ​​together in the time dimension, calculated based on the self-attention mechanism and After the concatenation, the attention matrix A is then added with a weighted term ΔA to enhance right of attention, namely:

[0031] A'=A+ΔA

[0032] In the subsequent calculation of the self-attention mechanism, the weighted attention matrix A' is used to enhance the restoration process for the restored high-definition frame. The content of the HD frame is focused on, thus ensuring that the restored results of other blindly degraded frames can be aligned with the content of the restored HD frame.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. Existing blind degradation methods are primarily designed for images and cannot be directly applied to restore blindly degraded videos. In contrast, the method presented in this paper is specifically designed for video restoration. This method improves the temporal coherence of the video by removing blindly degraded noise and extracting smooth frame content. High-quality feature maps are predicted based on spatiotemporal-aware features, ensuring spatiotemporal correlation in the prediction results. The restored video exhibits greater temporal stability.

[0035] 2. Existing video restoration methods are usually designed for specific degradation patterns, such as super-resolution technology and deblurring technology. Although these methods perform well in dealing with video data with specific degradation patterns, they rely on known degradation patterns and have difficulty dealing with unknown blind degradation problems, so they have limitations in practical applications. The method of the present invention first obtains smoother video content by eliminating blind degradation noise; then, the smoothed frame content is analyzed in combination with the target object, instance semantics and overall description to generate a comprehensive content understanding representation, providing prior knowledge for the subsequent prediction of high-quality feature maps, thereby achieving high-quality video restoration. This process does not rely on known degradation patterns, can effectively deal with blind degradation problems, and is more suitable for degradation scenarios in the real world.

[0036] 3. The method of the present invention adopts a multi-step prediction method for high-quality feature maps, aiming to accurately predict high-quality feature vectors through multiple iterations. Unlike the traditional method of predicting all content in one step, the method of the present invention can more effectively mine the effective information in the video data, ensuring that each frame and each area can be restored in the optimal state, thereby enhancing the robustness of the restoration and significantly improving the overall quality of the final restored content.

[0037] 4. When processing long videos, current video restoration technologies usually rely on sliding window technology or optical flow methods to ensure the continuity and smoothness of video content. However, the limitation of the sliding window method is that it can only process local segments of long videos at a time, which often leads to incoherence in the splicing between different segments in video restoration. On the other hand, although the optical flow method has certain advantages in maintaining video continuity, it becomes very difficult to accurately estimate the optical flow in the degraded video when facing blind video degradation. If there is a deviation in the optical flow estimation, the restored video content will often have serious incoherence problems. To address this challenge, the method of the present invention splices the restored high-definition frame with the blind degraded frame to be restored in the time dimension, and adopts an attention weighting mechanism to enhance the attention of the blind degraded frame to the restored high-definition frame content during the restoration process, thereby ensuring the consistency of the restoration result with the high-definition frame content, effectively solving the incoherence problem in long video restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of the present invention.

[0039] Figure 2 Schematic diagram of noise filtering for blind degraded frames.

[0040] Figure 3 Schematic diagram of building a high-quality feature pool.

[0041] Figure 4 Schematic diagram for obtaining comprehensive understanding of prior knowledge and predicting high-quality feature maps.

[0042] Figure 5 Schematic diagram of predicting high-quality feature maps in multiple steps. DETAILED DESCRIPTION

[0043] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] like Figures 1 to 5 As shown, this embodiment discloses a method for restoring time-stable blindly degraded video, the details of which are as follows:

[0045] 1) Obtain the blindly degraded video to be restored and evenly extract blindly degraded frames. Perform the following operations on each blindly degraded frame:

[0046] 1.1) A video encoder is used to extract latent space features from blindly degraded frames. The video encoder is composed of multiple stacked 3D convolutional layers. The extracted latent space features are then decomposed into content in different frequency bands using a discrete wavelet transform. Based on the degree of degradation of the blindly degraded frames, the content in each frequency band is filtered to remove the included blindly degraded noise. Finally, the filtered content in each frequency band is recombined using an inverse discrete wavelet transform to obtain smoothed frame content. The details are as follows:

[0047] Decompose the latent space features of the blind degraded frame into contents of different frequency bands:

[0048] {F1,F2,...F f ,...,F B =DWT(Z lq )

[0049] Where Z lq represents the latent space features of the blind degraded frame, and DWT represents discrete wavelet transform, which is used to decompose the input content into content of different frequency bands {F1, F2, ... F f ,...,F B}, B represents the total number of decomposed frequency bands, and the symbol f is used to represent the frequency band number. The larger the number, the higher the frequency band, and the smaller the number, the lower the frequency band. That is, F1 represents the lowest frequency band content, and F BIndicates the content of the highest frequency band; in each frequency band, the cross-attention mechanism is used to analyze the degree of content degradation of the frequency band based on the content of the current frequency band and the reference content lower than the current frequency band. The degree of degradation analysis is expressed as follows:

[0050]

[0051] Where, F f and F f-1 represents the content of the fth and f-1th frequency bands in the latent space feature, Q f , K f-1 、V f-1 Represents the query features, key features, and value features of the corresponding frequency band in the cross-attention mechanism, represents the learnable parameters of the corresponding frequency band, u represents the dimension of the key feature of the cross-attention mechanism in the degradation degree analysis, δ represents the function used to analyze the degree of content degradation, which outputs the filter coefficient t for the frequency band f The greater the degree of degradation, the smaller the filter coefficient, the minimum value is 0, the smaller the degree of degradation, the larger the filter coefficient, the maximum value is 1, the filter coefficient of the frequency band is multiplied by the content of the frequency band to obtain the filtered content F of the frequency band f ', expressed as F f '=t f F f When analyzing the degradation of the lowest frequency band content F1, the self-attention mechanism is used without referring to the content of other frequency bands. Finally, the filtered content of each frequency band is recombined through the inverse discrete wavelet transform:

[0052] Z'=IDWT(F1',F2',...F f ',...,F B '),

[0053] Where IDWT represents the inverse discrete wavelet transform, which is used to transform the filtered content F1', F2', ... F in each frequency band f ',...,F B 'Recombining to obtain the filtered latent space feature Z'; finally, using a video decoder to map the latent space feature Z' to the image space to obtain a smooth frame content, the video decoder consists of multiple 3D convolutional layers.

[0054] 1.2) Analyze the smoothed frame content from three perspectives: target object, instance semantics, and overall description. This allows identification of object locations and semantic regions, as well as the acquisition of smoothed frame content description text. By integrating information related to object locations, semantic regions, and smoothed frame content description text, a comprehensive content understanding representation is generated, providing comprehensive understanding prior knowledge for subsequent recovery. The details are as follows:

[0055] The BLIP-2 network is used to obtain text describing the content of the smoothed frame. The YOLOv8 detection network is used to detect the target object in the smoothed frame content. The SAM semantic segmentation network is used to segment the instance semantics in the smoothed frame content. The T5 network is used to obtain text features from the text describing the content of the smoothed frame. The deep features for target detection and semantic segmentation are extracted from the YOLOv8 detection network and the SAM semantic segmentation network respectively. Then, the text features, target detection, and semantic segmentation deep features are fused based on a graph network to obtain a comprehensive content understanding representation, which provides comprehensive understanding prior knowledge for subsequent recovery.

[0056] The deep features of text features, object detection, and semantic segmentation are integrated based on the graph network, as follows:

[0057] Construct an undirected graph Represents a node set of an undirected graph, which includes n local local nodes and a global node v global , v1 represents the first local node, and so on, v nlocal Indicates the nth local local nodes, Represents the edge set of an undirected graph; the local node is mapped by the deep features corresponding to the detection box generated by the YOLOv8 detection network and the deep feature maps corresponding to the segmentation area generated by the SAM semantic segmentation network, and the global node v global The text feature representation extracted by the T5 network is connected to all local nodes as the central node. The nodes corresponding to the detection boxes of the same detection category are connected through the graph network to obtain a connection edge. The detection confidence value of the YOLOv8 detection network is used to define the weight of the connection edge. The nodes corresponding to the segmentation areas of the same segmentation category are connected to obtain a connection edge. The segmentation confidence value of the SAM segmentation network is used to define the weight of the connection edge. The information of all node features is transferred through graph convolution. The graph convolution process is defined as:

[0058]

[0059] Where, ω ij Represents the i-th local node v i With the jth local node v j The fixed connection weight, h j Represents the jth local node v j Characteristics, h i represents the i-th local node v i The features of , Λ represents the learnable weight matrix in graph convolution, Represents the i-th local node v iThe set of adjacent nodes, σ represents the activation function; through multi-layer graph convolution operations, the deep features of text features, target detection and semantic segmentation are gradually integrated into the global node, and finally from the global node v global Get a comprehensive understanding of prior knowledge Z prior =Φ(v global ), where Φ represents a linear mapping function, which transforms v global Projection to Z prior .

[0060] 1.3) extracting latent space features from the smooth frame content obtained in step 1.1), and analyzing the spatiotemporal relationship of the extracted latent space features through a self-attention mechanism to obtain features with spatiotemporal awareness; then, integrating the comprehensive understanding prior knowledge obtained in step 1.2) into the acquired spatiotemporal awareness features through a cross-attention mechanism; then, predicting a high-quality feature map in multiple steps, i.e., determining which high-quality feature vectors in a pre-constructed high-quality feature pool the smooth frame content corresponds to, and using these high-quality feature vectors to characterize the smooth frame content, wherein the high-quality feature pool contains multiple high-quality feature vectors that can express high-quality video semantic content, and each high-quality feature vector has the same dimension; in each prediction step, first selecting a high-confidence prediction result, and extracting the corresponding high-quality feature vector from the high-quality feature pool to replace the latent space features of the smooth frame content, and then continuing to iteratively predict the remaining latent space features that have not been replaced until all latent space features of the smooth frame content are replaced by high-quality feature vectors in the high-quality feature pool, thereby obtaining a complete high-quality feature map, and finally mapping the complete high-quality feature map to the image space to obtain a restored high-definition frame; specifically as follows:

[0061] Given the latent space feature Z of the smooth frame content, the feature Z with spatiotemporal perception is obtained based on the self-attention mechanism space ; Then, based on the cross attention mechanism, the feature Z with spatiotemporal perception space and comprehensive understanding of prior knowledge Z prior The fusion is performed as follows:

[0062]

[0063] Where Q, K, and V are the query features, key features, and value features in the cross-attention mechanism, respectively, and W Q 、W K 、W V is a learnable parameter, d is the dimension of feature K; based on the fused feature Z fuse , predict a high-quality feature map in multiple steps The high-quality feature map With Z fuseWith the same dimension as Z, the prediction formula is as follows:

[0064]

[0065] Where, represents the prediction network, Represents the confidence of predicting the mth high-quality feature vector in the high-quality feature pool, N represents the size of the high-quality feature pool, Represents feature Z fuse The position in the latent space feature Z is the value of the f-th frame, h-th row, and w-th column. According to the above prediction formula, the f-th frame, h-th row, and w-th column position in the latent space feature Z corresponds to the l-th high-quality feature vector in the high-quality feature pool. Therefore, the l-th high-quality feature vector is taken from the high-quality feature pool to replace the value of the corresponding position in Z. When the values ​​of all positions in Z are replaced, a high-quality feature map can be obtained.

[0066] The high-quality feature vectors in the high-quality feature pool are learned based on the reconstruction of high-quality HD videos and images by the 3D VQGAN network. The 3D VQGAN network includes a 3D VQGAN encoder and a 3D VQGAN decoder. The high-quality feature vectors in the high-quality feature pool are divided into high-quality video feature vectors and high-quality image feature vectors. When an image is input, only the high-quality image feature vector part is updated. When a video is input, only the high-quality video feature vector part is updated. After the high-quality feature pool completes learning, a high-quality feature map is obtained through step 1.3). Then, under the action of the 3D VQGAN decoder, Mapping to image space to obtain the restored high-definition frame

[0067] 2) Continue to uniformly extract blind degraded frames from the blindly degraded video to be restored, splice the restored high-definition frames with the extracted blindly degraded frames in the temporal dimension, and use attention weighting to enhance the attention of the blindly degraded frames to the content of the restored high-definition frames during the restoration process, thereby ensuring the consistency of the restored results with the content of the high-definition frames. The blindly degraded frames are restored by executing steps 1.1) to 1.3) above; specifically, as follows:

[0068] Given the degraded frame V to be restored LQ And the HD frame recovered from step 1) Extract the corresponding latent space features and Spliced ​​together in the time dimension, calculated based on the self-attention mechanism and After the concatenation, the attention matrix A is then added with a weighted term ΔA to enhance right of attention, namely:

[0069] A'=A+ΔA

[0070] In the subsequent calculation of the self-attention mechanism, the weighted attention matrix A' is used to enhance the restoration process for the restored high-definition frame. The content of the HD frame is focused on, thus ensuring that the restored results of other blindly degraded frames can be aligned with the content of the restored HD frame.

[0071] 3) Repeat step 2) until all blindly degraded frames in the blindly degraded video to be restored are restored, and finally generate coherent and high-definition video content.

[0072] Furthermore, for the learnable parameters involved in step 1.3), the present invention designs two training stages. In the first stage, the present invention fixes the 3D VQGAN decoder and only trains the remaining learnable parameters, relying on the cross-entropy loss function to achieve the prediction of high-quality feature maps. After the first stage of training, although it can process most blind degraded videos, the restored video may appear slightly blurred due to the incomplete matching problem between the 3D VQGAN decoder and the predicted high-quality feature maps. In order to further optimize the adaptability of the 3D VQGAN decoder and the predicted features, the present invention enters the second stage of training. In this stage, all learnable parameters except the 3D VQGAN decoder are fixed, and the focus is on training the parameters of the 3D VQGAN decoder part. The training of this stage introduces the L1 loss function, the perceptual loss function and the adversarial loss function to improve the recovery ability of the 3D VQGAN decoder from multiple aspects. When the second stage of training is completed, the method of the present invention can restore richer and more delicate video details, significantly improving the overall quality of the video.

[0073] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for restoring time-stable blindly degraded video, characterized in that: The following steps are involved: 1) Obtain the blindly degraded video to be restored and evenly extract blindly degraded frames. Perform the following operations on each blindly degraded frame: 1.1) Extract latent space features from the blindly degraded frames and decompose them into different frequency bands using discrete wavelet transform. Based on the degree of degradation of the blindly degraded frames, filter the content of each frequency band to remove the included blindly degraded noise. Finally, use inverse discrete wavelet transform to recombine the filtered content of each frequency band to obtain the smoothed frame content. 1.2) Analyze the smooth frame content from three perspectives: target object, instance semantics, and overall description, to identify the location and semantic region of the object and obtain the smooth frame content description text; By fusing information related to object location, semantic region, and smooth frame content description text, a comprehensive content understanding representation is generated, providing comprehensive understanding prior knowledge for subsequent recovery; 1.3) extracting latent space features from the smooth frame content obtained in step 1.1), and analyzing the spatiotemporal relationship of the extracted latent space features through a self-attention mechanism to obtain features with spatiotemporal perception; then, integrating the comprehensive understanding prior knowledge obtained in step 1.2) into the acquired spatiotemporal perception features through a cross-attention mechanism; then, predicting a high-quality feature map in multiple steps, i.e., determining which high-quality feature vectors in a pre-constructed high-quality feature pool the smooth frame content corresponds to, and using these high-quality feature vectors to characterize the smooth frame content, wherein the high-quality feature pool contains multiple high-quality feature vectors that can express high-quality video semantic content, and each high-quality feature vector has the same dimension; in each prediction step, first selecting a high-confidence prediction result, and extracting the corresponding high-quality feature vector from the high-quality feature pool to replace the latent space features of the smooth frame content, and then continuing to iteratively predict the remaining latent space features that have not been replaced until all latent space features of the smooth frame content are replaced by high-quality feature vectors in the high-quality feature pool, thereby obtaining a complete high-quality feature map, and finally mapping the complete high-quality feature map to the image space to obtain a restored high-definition frame; 2) Evenly extracting blind degraded frames from the blindly degraded video to be restored, splicing the restored high-definition frames with the extracted blindly degraded frames in the temporal dimension, and using attention weighting to enhance the attention of the blindly degraded frames to the content of the restored high-definition frames during the restoration process, thereby ensuring consistency between the restored result and the content of the high-definition frames. The blindly degraded frames are restored by executing steps 1.1) to 1.3) above. 3) Repeat step 2) until all blindly degraded frames in the blindly degraded video to be restored are restored, and finally generate coherent and high-definition video content.

2. The method for restoring a temporally stable blindly degraded video according to claim 1, wherein: In step 1.1), a video encoder is used to extract latent space features from the blind degraded frame. The video encoder is composed of multiple stacked 3D convolutional layers, which decomposes the latent space features of the blind degraded frame into content in different frequency bands: {F1,F2,...F f ,...,F B }=DWT(Z lq ) Where Z lq represents the latent space features of the blind degraded frame, and DWT represents discrete wavelet transform, which is used to decompose the input content into content of different frequency bands {F1, F2, ... F f ,...,F B }, B represents the total number of decomposed frequency bands, and the symbol f is used to represent the frequency band number. The larger the number, the higher the frequency band, and the smaller the number, the lower the frequency band. That is, F1 represents the lowest frequency band content, and F B Indicates the content of the highest frequency band; in each frequency band, the cross-attention mechanism is used to analyze the degree of content degradation of the frequency band based on the content of the current frequency band and the reference content lower than the current frequency band. The degree of degradation analysis is expressed as follows: Where, F f and F f-1 represents the content of the fth and f-1th frequency bands in the latent space feature, Q f , K f-1 、V f-1 Represents the query features, key features, and value features of the corresponding frequency band in the cross-attention mechanism, represents the learnable parameters of the corresponding frequency band, u represents the dimension of the key feature of the cross-attention mechanism in the degradation degree analysis, δ represents the function used to analyze the degree of content degradation, which outputs the filter coefficient t for the frequency band f The greater the degree of degradation, the smaller the filter coefficient, the minimum value is 0, the smaller the degree of degradation, the larger the filter coefficient, the maximum value is 1, the filter coefficient of the frequency band is multiplied by the content of the frequency band to obtain the filtered content F' of the frequency band f , denoted as F′ f =t f F f When analyzing the degradation of the lowest frequency band content F1, the self-attention mechanism is used without referring to the content of other frequency bands. Finally, the filtered content of each frequency band is recombined through the inverse discrete wavelet transform: Z'=IDWT(F1',F′2,...F′ f ,...,F′ B ), Where IDWT represents the inverse discrete wavelet transform, which is used to transform the filtered content F1', F'2, ... F' in each frequency band f ,...,F′ B Recombine to obtain the filtered latent space feature Z'; Finally, a video decoder is used to map the latent space features Z' to the image space to obtain smooth frame content. The video decoder consists of multiple stacked 3D convolutional layers.

3. The method for restoring a temporally stable blindly degraded video according to claim 2, wherein: In step 1.2), the BLIP-2 network is used to obtain text describing the content of the smooth frame, the YOLOv8 detection network is used to detect the target object in the smooth frame content, and the SAM semantic segmentation network is used to segment the instance semantics in the smooth frame content; the T5 network is used to obtain text features from the text describing the content of the smooth frame, and the deep features of target detection and semantic segmentation are extracted from the YOLOv8 detection network and the SAM semantic segmentation network respectively. Then, based on the graph network, the text features, target detection and semantic segmentation deep features are fused to obtain a comprehensive content understanding representation, which provides comprehensive understanding prior knowledge for subsequent recovery.

4. The method for restoring a temporally stable blindly degraded video according to claim 3, wherein: The deep features of text features, object detection, and semantic segmentation are integrated based on the graph network, as follows: Construct an undirected graph in Represents a node set of an undirected graph, which includes n local local nodes and a global node v global , v1 represents the first local node, and so on, Indicates the nth local local nodes, Represents the edge set of an undirected graph; the local node is mapped by the deep features corresponding to the detection box generated by the YOLOv8 detection network and the deep feature maps corresponding to the segmentation area generated by the SAM semantic segmentation network, and the global node v global The text feature representation extracted by the T5 network is connected to all local nodes as the central node. The nodes corresponding to the detection boxes of the same detection category are connected through the graph network to obtain a connection edge. The detection confidence value of the YOLOv8 detection network is used to define the weight of the connection edge. The nodes corresponding to the segmentation areas of the same segmentation category are connected to obtain a connection edge. The segmentation confidence value of the SAM segmentation network is used to define the weight of the connection edge. The information of all node features is transferred through graph convolution. The graph convolution process is defined as: Where, ω ij Represents the i-th local node v i With the jth local node v j The fixed connection weight, h j Represents the jth local node v j Characteristics, h i represents the i-th local node v i The features of , Λ represents the learnable weight matrix in graph convolution, Represents the i-th local node v i The set of adjacent nodes, σ represents the activation function; through multi-layer graph convolution operations, the deep features of text features, target detection and semantic segmentation are gradually integrated into the global node, and finally from the global node v global Get a comprehensive understanding of prior knowledge Z prior =φ(v global ), where φ represents a linear mapping function that transforms v global Projection to Z prior .

5. The method for restoring a temporally stable blindly degraded video according to claim 4, wherein: In step 1.3), the latent space feature Z of the smoothed frame content is given, and the feature Z with spatiotemporal perception is obtained based on the self-attention mechanism space ; Then, based on the cross attention mechanism, the feature Z with spatiotemporal perception space and comprehensive understanding of prior knowledge Z prior The fusion is performed as follows: Q=W Q Z space K=W K WITH prior V=W V Z prior Where Q, K, and V are the query features, key features, and value features in the cross-attention mechanism, respectively, and W Q 、W K 、W V is a learnable parameter, d is the dimension of feature K; based on the fused feature Z fuse , predict a high-quality feature map in multiple steps The high-quality feature map With Z fuse With the same dimension as Z, the prediction formula is as follows: Where, represents the prediction network, Represents the confidence of predicting the mth high-quality feature vector in the high-quality feature pool, N represents the size of the high-quality feature pool, Represents feature Z fuse The position in the latent space feature Z is the value of the f-th frame, h-th row, and w-th column. According to the above prediction formula, the f-th frame, h-th row, and w-th column position in the latent space feature Z corresponds to the l-th high-quality feature vector in the high-quality feature pool. Therefore, the l-th high-quality feature vector is taken from the high-quality feature pool to replace the value of the corresponding position in Z. When the values ​​of all positions in Z are replaced, a high-quality feature map can be obtained.

6. The method for restoring a temporally stable blindly degraded video according to claim 5, wherein: The high-quality feature vectors in the high-quality feature pool are learned based on the reconstruction of high-quality HD videos and images by the 3D VQGAN network. The 3D VQGAN network includes a 3D VQGAN encoder and a 3D VQGAN decoder. The high-quality feature vectors in the high-quality feature pool are divided into high-quality video feature vectors and high-quality image feature vectors. When an image is input, only the high-quality image feature vector part is updated. When a video is input, only the high-quality video feature vector part is updated. After the high-quality feature pool completes learning, a high-quality feature map is obtained through step 1.3). Then, under the action of the 3D VQGAN decoder, Mapping to image space to obtain the restored high-definition frame 7. The method for restoring a temporally stable blindly degraded video according to claim 6, wherein: In step 2), the degraded frame V to be restored is given LQ And the HD frame recovered from step 1) Extract the corresponding latent space features and Spliced ​​together in the time dimension, calculated based on the self-attention mechanism and After the concatenation, the attention matrix A is then added with a weighted term ΔA to enhance right of attention, namely: A'=A+ΔA In the subsequent calculation of the self-attention mechanism, the weighted attention matrix A' is used to enhance the restoration process for the restored high-definition frame. The content of the HD frame is focused on, thus ensuring that the restored results of other blindly degraded frames can be aligned with the content of the restored HD frame.

Images