Method for recovering blind degraded video with stable time sequence

Abstract

The invention discloses a method for recovering a blind degraded video with a stable time sequence, which aims at solving the problem of unstable time sequence of blind degraded video recovery, and comprises the following steps of: firstly, removing blind degraded noise according to the degradation degree of the blind degraded video to obtain smooth frame content; thirdly, analyzing the content of the smooth frame from three angles of a target object, instance semantics and overall description, and providing comprehensive understanding priori knowledge for video recovery; meanwhile, analyzing the space-time relationship of the hidden space features of the smooth frame content to obtain features with space-time perception; and then, integrating comprehensive understanding priori knowledge into features with space-time perception, predicting a high-quality feature map in multiple steps, and finally mapping the high-quality feature map to an image space to recover a high-quality video. The real world blind degraded video modeling is realized, and the high-definition video content with coherent time sequence is recovered.

Description

Technical Field

[0001] The invention relates to the technical field of blind degradation data processing, and in particular to a temporally stable blind degradation video restoration method. Background Art

[0002] In modern society, video has become an important means of information exchange and recording. However, in many application scenarios, such as old videos on the Internet and blurry videos in security systems, video quality is often degraded to varying degrees. These degradations may be caused by compression, transmission loss, low-resolution shooting equipment or harsh environmental conditions. Since videos may be affected by multiple degradation types such as blur, noise, compression distortion, color distortion, etc., and there may be interactions between various degradation types, the recovery process is more complicated. When data is subjected to multiple degradations at the same time and the specific degradation mode is unknown, we call it "blind degradation".

[0003] Existing blind degradation restoration methods are designed for images. Such methods do not take into account the correlation between frames in the video, so they cannot be directly applied to the restoration of blind degraded videos. On the other hand, existing video restoration methods usually target specific degradation patterns, such as super-resolution and deblurring techniques. Although these methods perform well on their specific degradation data, most of them rely on known degradation models and have difficulty dealing with a combination of multiple unknown degradations, so they are difficult to apply directly to real-world scenarios. For example, super-resolution only focuses on improving the resolution of the video, but has limited improvements on blur or noise degradation. Deblurring is only effective for motion blur or defocus blur, but is not effective when faced with compression loss or low resolution. Therefore, developing a blind degradation video restoration technology that can handle various unknown degradation types is of great significance for improving video quality and enhancing information acquisition capabilities. This technology not only needs to be effective under a variety of degradation conditions, but also needs to be robust and versatile to meet the needs of different application scenarios. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies and shortcomings of the prior art and provide a temporally stable blind degraded video restoration method to achieve modeling of real-world blind degraded videos and restore high-definition and temporally coherent video content.

[0005] To achieve the above object, the technical solution provided by the present invention is: a method for restoring a time-series stable blind degraded video, comprising the following steps:

[0006] 1) Obtain the blind degraded video to be restored, and evenly extract the blind degraded frames, and perform the following operations on each blind degraded frame:

[0007] 1.1) Extract latent space features from the blind degraded frame, and use discrete wavelet transform to decompose the extracted latent space features into contents of different frequency bands; according to the degradation degree of the blind degraded frame, perform noise filtering on the contents of each frequency band to remove the blind degradation noise contained; finally, reassemble the filtered contents of each frequency band through inverse discrete wavelet transform to obtain smooth frame content;

[0008] 1.2) Analyze the content of the smooth frame from three perspectives: target object, instance semantics, and overall description, to identify the location and semantic region of the object and obtain the description text of the smooth frame content; generate a comprehensive content understanding representation by fusing information related to the object location, semantic region, and the description text of the smooth frame content, and provide comprehensive understanding prior knowledge for subsequent recovery;

[0009] 1.3) extracting latent space features from the smooth frame content obtained in step 1.1), and analyzing the spatiotemporal relationship of the extracted latent space features through a self-attention mechanism to obtain features with spatiotemporal perception; then, integrating the comprehensive understanding prior knowledge obtained in step 1.2) into the acquired features with spatiotemporal perception through a cross-attention mechanism; then, predicting a high-quality feature map in multiple steps, that is, determining which high-quality feature vectors in a pre-constructed high-quality feature pool the smooth frame content corresponds to, and using these high-quality feature vectors to characterize the smooth frame content, wherein the high-quality feature pool contains multiple high-quality feature vectors that can express high-quality video semantic content, and each high-quality feature vector has the same dimension; in each prediction step, firstly selecting a high-confidence prediction result, and extracting a corresponding high-quality feature vector from the high-quality feature pool to replace the latent space features of the smooth frame content, and then continuing to iteratively predict the remaining latent space features that have not been replaced, until all latent space features of the smooth frame content are replaced by high-quality feature vectors in the high-quality feature pool, and obtaining a complete high-quality feature map, and finally mapping the complete high-quality feature map to the image space to obtain a restored high-definition frame;

[0010] 2) Continue to uniformly extract blind degraded frames from the blind degraded video to be restored, splice the restored high-definition frames with the extracted blind degraded frames in the time dimension, and enhance the attention of the blind degraded frames to the restored high-definition frames during the restoration process through attention weighting, thereby ensuring the consistency of the restored results with the content of the high-definition frames, wherein the blind degraded frames are restored by executing the above steps 1.1) to 1.3);

[0011] 3) Repeat step 2) until all blindly degraded frames in the blindly degraded video to be restored are restored, and finally a coherent and high-definition video content is generated.

[0012] Further, in step 1.1), a video encoder is used to extract latent space features from the blind degraded frame, wherein the video encoder is composed of a plurality of stacked 3D convolutional layers, and the latent space features of the blind degraded frame are decomposed into contents of different frequency bands:

[0013] {F1,F2,...F f ,...,F B = DWT(Z lq )

[0014] In the formula, Z lq represents the latent space features of the blind degraded frame, and DWT represents discrete wavelet transform, which is used to decompose the input content into content of different frequency bands {F1, F2, ... F f ,...,F B}, B represents the total number of decomposed frequency bands, and the symbol f is used to represent the serial number of the frequency band. The larger the serial number, the higher the frequency band, and the smaller the serial number, the lower the frequency band. That is, F1 represents the lowest frequency band content, and F B Indicates the content of the highest frequency band; in each frequency band, the cross-attention mechanism is used to analyze the degree of content degradation of the frequency band based on the content of the current frequency band and the reference content lower than the current frequency band. The method of analyzing the degree of degradation is expressed as follows:

[0015]

[0016] In the formula, F f and F f-1 represents the content of the fth and f-1th frequency bands in the latent space feature, Q f , K f-1 、V f-1 Represents the query features, key features, and value features of the corresponding frequency bands in the cross-attention mechanism, represents the learnable parameters of the corresponding frequency band, u represents the dimension of the key features of the cross-attention mechanism in the degradation degree analysis, δ represents the function used to analyze the degree of content degradation, which outputs the filter coefficient t for the frequency band f The greater the degree of degradation, the smaller the filter coefficient, the minimum value is 0, the smaller the degree of degradation, the larger the filter coefficient, the maximum value is 1, the filter coefficient of the frequency band is multiplied by the content of the frequency band to obtain the filtered content F of the frequency band f ', expressed as F f '=t f F f ; When analyzing the degradation degree of the lowest frequency band content F1, the self-attention mechanism is used without referring to the content of other frequency bands; finally, the filtered content of each frequency band is recombined through the inverse discrete wavelet transform:

[0017] Z'=IDWT(F1',F2',...F f ',...,FB '),

[0018] Where, IDWT represents the inverse discrete wavelet transform, which is used to transform the filtered content F1', F2', ... F f ',...,F B 'Recombine to obtain the filtered latent space feature Z'; finally, use a video decoder to map the latent space feature Z' to the image space to obtain a smooth frame content, and the video decoder is composed of a plurality of stacked 3D convolutional layers.

[0019] Furthermore, in step 1.2), the BLIP-2 network is used to obtain text describing the content of the smooth frame, the YOLOv8 detection network is used to detect the target object in the content of the smooth frame, and the SAM semantic segmentation network is used to segment the instance semantics in the content of the smooth frame; the T5 network is used to obtain text features from the text describing the content of the smooth frame, and the deep features of target detection and semantic segmentation are extracted from the YOLOv8 detection network and the SAM semantic segmentation network respectively, and then the text features, target detection and semantic segmentation deep features are fused based on the graph network to obtain a comprehensive content understanding representation, which provides comprehensive understanding prior knowledge for subsequent recovery.

[0020] Furthermore, the deep features of text features, object detection and semantic segmentation are fused based on the graph network, as follows:

[0021] Constructing an undirected graph in Represents a node set of an undirected graph, which includes n local local nodes and a global node v global , v1 represents the first local node, and so on, v nlocal Indicates the nth local local nodes, Represents the edge set of an undirected graph; the local node is mapped by the deep features corresponding to the detection box generated by the YOLOv8 detection network and the deep feature mapping corresponding to the segmentation area generated by the SAM semantic segmentation network, and the global node v global The text feature representation extracted by the T5 network is connected to all local nodes as the central node; the nodes corresponding to the detection boxes of the same detection category are connected through the graph network to obtain the connection edge, and the detection confidence value of the YOLOv8 detection network is used to define the weight of the connection edge; the nodes corresponding to the segmentation areas of the same segmentation category are connected to obtain the connection edge, and the segmentation confidence of the SAM segmentation network is used to define the weight of the connection edge; all node features are transmitted through graph convolution, and the graph convolution process is defined as:

[0022]

[0023] In the formula, ω ij represents the i-th local node v i With the jth local node v j The fixed connection weight, h j represents the jth local node v j The characteristics of h i The i-th local node v i The features of , Λ represents the learnable weight matrix in graph convolution, represents the i-th local node v i The adjacent node set of v represents the activation function; through multi-layer graph convolution operations, the deep features of text features, object detection and semantic segmentation are gradually integrated into the global node, and finally from the global node v global Get a comprehensive understanding of prior knowledge Z prior =Φ(v global ), where φ represents a linear mapping function that transforms v global Projection to Z prior .

[0024] Furthermore, in step 1.3), the latent space feature Z of the smoothed frame content is given, and the feature Z with spatiotemporal perception is obtained based on the self-attention mechanism space ; Then, based on the cross attention mechanism, the feature Z with spatiotemporal perception space and comprehensive understanding of prior knowledge Z prior The fusion is performed, and the fusion method is as follows:

[0025]

[0026] Where Q, K, and V are the query features, key features, and value features in the cross-attention mechanism, respectively, and W Q , W K , W V is a learnable parameter, d is the dimension of feature K; based on the fused feature Z fuse , predict a high-quality feature map in multiple steps Among them, high-quality feature maps With Z fuse With the same dimension as Z, the prediction formula is as follows:

[0027]

[0028] In the formula, represents the prediction network, Represents the confidence of predicting the mth high-quality feature vector in the high-quality feature pool, N represents the size of the high-quality feature pool, Represents feature Z fuseThe position in the latent space feature Z is the value of the f-th frame, h-th row, and w-th column. According to the above prediction formula, the f-th frame, h-th row, and w-th column position in the latent space feature Z corresponds to the l-th high-quality feature vector in the high-quality feature pool. Therefore, the l-th high-quality feature vector is taken from the high-quality feature pool to replace the value of the corresponding position in Z. When the values ​​of all positions in Z are replaced, a high-quality feature map can be obtained.

[0029] Furthermore, the high-quality feature vectors in the high-quality feature pool are learned based on reconstructing high-quality high-definition videos and images through a 3D VQGAN network. The 3D VQGAN network includes a 3D VQGAN encoder and a 3D VQGAN decoder. The high-quality feature vectors in the high-quality feature pool are divided into high-quality video feature vectors and high-quality image feature vectors. When an image is input, only the high-quality image feature vector part is updated. When a video is input, only the high-quality video feature vector part is updated. When the high-quality feature pool completes learning, a high-quality feature map is obtained through step 1.3). Then, under the action of the 3DVQGAN decoder, Mapping to image space to obtain the restored high-definition frame

[0030] Further, in step 2), the degraded frame V to be restored is given LQ And the HD frame recovered from step 1) Extract the corresponding latent space features and Spliced ​​together in the time dimension, calculated based on the self-attention mechanism and After the concatenation, the attention matrix A is then added with a weighted term ΔA to enhance right of concern, namely:

[0031] A'=A+ΔA

[0032] In the subsequent calculation of the self-attention mechanism, the weighted attention matrix A' is used to enhance the restoration process for the restored HD frame. The content of the HD frame is focused on, thereby ensuring that the restored results of other blind degraded frames can be aligned with the content of the restored HD frame.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. Existing blind degradation methods are mainly designed for images and cannot be directly applied to the restoration of blind degraded videos. In contrast, the method of the present invention is designed specifically for video restoration. In this method, the blind degradation noise in the video is removed and the smooth frame content is extracted, thereby improving the temporal coherence of the video; based on the features with spatiotemporal perception, high-quality feature maps are predicted, so that the prediction results have spatiotemporal correlation, and the finally restored video shows stronger temporal stability.

[0035] 2. Existing video restoration methods are usually designed for specific degradation patterns, such as super-resolution technology and deblurring technology. Although these methods perform well in dealing with video data with specific degradation patterns, they rely on known degradation patterns and have difficulty dealing with unknown blind degradation problems, so they have limitations in practical applications. The method of the present invention first obtains smoother video content by eliminating blind degradation noise; then, the smooth frame content is analyzed in combination with the target object, instance semantics and overall description to generate a comprehensive content understanding representation, which provides prior knowledge for the subsequent prediction of high-quality feature maps, thereby achieving high-quality video restoration. This process does not rely on known degradation patterns, can effectively deal with blind degradation problems, and is more suitable for degradation scenarios in the real world.

[0036] 3. The method of the present invention adopts multi-step prediction of high-quality feature maps, aiming to accurately predict high-quality feature vectors through multiple iterations. Different from the traditional method of predicting all content in one step, the method of the present invention can more effectively mine the effective information in the video data, ensuring that each frame and each area can be restored in the optimal state, thereby enhancing the robustness of the restoration and significantly improving the overall quality of the final restored content.

[0037] 4. When processing long videos, current video restoration technology usually relies on sliding window technology or optical flow method to ensure the continuity and smoothness of video content. However, the limitation of the sliding window method is that it can only process local fragments of long videos at a time, which often leads to incoherence in the splicing between different fragments in video restoration. On the other hand, although the optical flow method has certain advantages in maintaining video coherence, it becomes very difficult to accurately estimate the optical flow in the degraded video when facing blind video degradation. If there is a deviation in the optical flow estimation, the restored video content often has serious incoherence problems. In response to this challenge, the method of the present invention splices the restored high-definition frame with the blind degraded frame to be restored in the time dimension, and adopts an attention weighting mechanism to enhance the attention of the blind degraded frame to the restored high-definition frame content during the restoration process, thereby ensuring the consistency of the restoration result with the high-definition frame content, and effectively solving the incoherence problem in long video restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention.

[0039] Figure 2 Schematic diagram of noise filtering for blind degraded frames.

[0040] Figure 3 Schematic diagram for constructing a high-quality feature pool.

[0041] Figure 4 Schematic diagram for obtaining comprehensive understanding of prior knowledge and predicting high-quality feature maps.

[0042] Figure 5 Schematic diagram of predicting high-quality feature maps in multiple steps. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] like Figures 1 to 5 As shown, this embodiment discloses a method for restoring a time-series stable blind degraded video, the details of which are as follows:

[0045] 1) Obtain the blind degraded video to be restored, and evenly extract the blind degraded frames, and perform the following operations on each blind degraded frame:

[0046] 1.1) A video encoder is used to extract latent space features from the blind degraded frame, wherein the video encoder is composed of a plurality of stacked 3D convolutional layers, and then a discrete wavelet transform is used to decompose the extracted latent space features into contents of different frequency bands; according to the degree of degradation of the blind degraded frame, the contents of each frequency band are subjected to noise filtering to remove the included blind degradation noise; finally, the filtered contents of each frequency band are recombined through an inverse discrete wavelet transform to obtain a smooth frame content; specifically as follows:

[0047] Decompose the latent space features of the blind degraded frame into contents of different frequency bands:

[0048] {F1,F2,...F f ,...,F B = DWT(Z lq )

[0049] In the formula, Z lq represents the latent space features of the blind degraded frame, and DWT represents discrete wavelet transform, which is used to decompose the input content into content of different frequency bands {F1, F2, ... F f ,...,F B}, B represents the total number of decomposed frequency bands, and the symbol f is used to represent the serial number of the frequency band. The larger the serial number, the higher the frequency band, and the smaller the serial number, the lower the frequency band. That is, F1 represents the lowest frequency band content, and F BIndicates the content of the highest frequency band; in each frequency band, the cross-attention mechanism is used to analyze the degree of content degradation of the frequency band based on the content of the current frequency band and the reference content lower than the current frequency band. The method of analyzing the degree of degradation is expressed as follows:

[0050]

[0051] In the formula, F f and F f-1 represents the content of the fth and f-1th frequency bands in the latent space feature, Q f , K f-1 、V f-1 Represents the query features, key features, and value features of the corresponding frequency bands in the cross-attention mechanism, represents the learnable parameters of the corresponding frequency band, u represents the dimension of the key features of the cross-attention mechanism in the degradation degree analysis, δ represents the function used to analyze the degree of content degradation, which outputs the filter coefficient t for the frequency band f The greater the degree of degradation, the smaller the filter coefficient, the minimum value is 0, the smaller the degree of degradation, the larger the filter coefficient, the maximum value is 1, the filter coefficient of the frequency band is multiplied by the content of the frequency band to obtain the filtered content F of the frequency band f ', expressed as F f '=t f F f ; When analyzing the degradation degree of the lowest frequency band content F1, the self-attention mechanism is used without referring to the content of other frequency bands; finally, the filtered content of each frequency band is recombined through the inverse discrete wavelet transform:

[0052] Z'=IDWT(F1',F2',...F f ',...,F B '),

[0053] Where, IDWT represents the inverse discrete wavelet transform, which is used to transform the filtered content F1', F2', ... F f ',...,F B 'Recombine to obtain the filtered latent space feature Z'; finally, use a video decoder to map the latent space feature Z' to the image space to obtain a smooth frame content, and the video decoder is composed of multiple 3D convolutional layers.

[0054] 1.2) Analyze the smooth frame content from three perspectives: target object, instance semantics, and overall description, to identify the location and semantic region of the object and obtain the smooth frame content description text; generate a comprehensive content understanding representation by fusing information related to the object location, semantic region, and smooth frame content description text, and provide comprehensive understanding prior knowledge for subsequent recovery; the details are as follows:

[0055] Use the BLIP-2 network to obtain text describing the content of the smooth frame, use the YOLOv8 detection network to detect the target object in the content of the smooth frame, and use the SAM semantic segmentation network to segment the instance semantics in the content of the smooth frame; use the T5 network to obtain text features from the text describing the content of the smooth frame, extract the deep features of target detection and semantic segmentation from the YOLOv8 detection network and the SAM semantic segmentation network respectively, and then fuse the text features, target detection and semantic segmentation deep features based on the graph network to obtain a comprehensive content understanding representation, which provides comprehensive understanding prior knowledge for subsequent recovery;

[0056] Based on the graph network, the deep features of text features, object detection and semantic segmentation are integrated as follows:

[0057] Constructing an undirected graph Represents a node set of an undirected graph, which includes n local local nodes and a global node v global , v1 represents the first local node, and so on, v nlocal Indicates the nth local local nodes, Represents the edge set of an undirected graph; the local node is mapped by the deep features corresponding to the detection box generated by the YOLOv8 detection network and the deep feature mapping corresponding to the segmentation area generated by the SAM semantic segmentation network, and the global node v global The text feature representation extracted by the T5 network is connected to all local nodes as the central node; the nodes corresponding to the detection boxes of the same detection category are connected through the graph network to obtain the connection edge, and the detection confidence value of the YOLOv8 detection network is used to define the weight of the connection edge; the nodes corresponding to the segmentation areas of the same segmentation category are connected to obtain the connection edge, and the segmentation confidence of the SAM segmentation network is used to define the weight of the connection edge; all node features are transmitted through graph convolution, and the graph convolution process is defined as:

[0058]

[0059] In the formula, ω ij represents the i-th local node v i With the jth local node v j The fixed connection weight, h j represents the jth local node v j The characteristics of h i The i-th local node v i The features of , Λ represents the learnable weight matrix in graph convolution, represents the i-th local node v iThe adjacent node set of v represents the activation function; through multi-layer graph convolution operations, the deep features of text features, object detection and semantic segmentation are gradually integrated into the global node, and finally from the global node v global Get a comprehensive understanding of prior knowledge Z prior =Φ(v global ), where Φ represents a linear mapping function that transforms v global Projection to Z prior .

[0060] 1.3) extracting latent space features from the smooth frame content obtained in step 1.1), and analyzing the spatiotemporal relationship of the extracted latent space features through a self-attention mechanism to obtain features with spatiotemporal perception; then, through a cross-attention mechanism, integrating the comprehensive understanding prior knowledge obtained in step 1.2) into the acquired features with spatiotemporal perception; then, predicting a high-quality feature map in multiple steps, that is, determining which high-quality feature vectors in a pre-constructed high-quality feature pool the smooth frame content corresponds to, and using these high-quality feature vectors to characterize the smooth frame content, wherein the high-quality feature pool contains multiple high-quality feature vectors that can express high-quality video semantic content, and each high-quality feature vector has the same dimension; in each prediction step, firstly selecting a high-confidence prediction result, and extracting a corresponding high-quality feature vector from the high-quality feature pool to replace the latent space features of the smooth frame content, and then continuing to iteratively predict the remaining latent space features that have not been replaced, until all latent space features of the smooth frame content are replaced by high-quality feature vectors in the high-quality feature pool, and obtaining a complete high-quality feature map, and finally mapping the complete high-quality feature map to the image space to obtain a restored high-definition frame; specifically as follows:

[0061] Given the latent space feature Z of the smooth frame content, the feature Z with spatiotemporal perception is obtained based on the self-attention mechanism space ; Then, based on the cross attention mechanism, the feature Z with spatiotemporal perception space and comprehensive understanding of prior knowledge Z prior The fusion is performed, and the fusion method is as follows:

[0062]

[0063] Where Q, K, and V are the query features, key features, and value features in the cross-attention mechanism, respectively, and W Q , W K , W V is a learnable parameter, d is the dimension of feature K; based on the fused feature Z fuse , predict a high-quality feature map in multiple steps Among them, high-quality feature maps With Z fuseWith the same dimension as Z, the prediction formula is as follows:

[0064]

[0065] In the formula, represents the prediction network, Represents the confidence of predicting the mth high-quality feature vector in the high-quality feature pool, N represents the size of the high-quality feature pool, Represents feature Z fuse The position in the latent space feature Z is the value of the f-th frame, h-th row, and w-th column. According to the above prediction formula, the f-th frame, h-th row, and w-th column position in the latent space feature Z corresponds to the l-th high-quality feature vector in the high-quality feature pool. Therefore, the l-th high-quality feature vector is taken from the high-quality feature pool to replace the value of the corresponding position in Z. When the values ​​of all positions in Z are replaced, a high-quality feature map can be obtained.

[0066] The high-quality feature vectors in the high-quality feature pool are learned based on the reconstruction of high-quality high-definition videos and images by a 3D VQGAN network. The 3D VQGAN network includes a 3D VQGAN encoder and a 3D VQGAN decoder. The high-quality feature vectors in the high-quality feature pool are divided into high-quality video feature vectors and high-quality image feature vectors. When an image is input, only the high-quality image feature vector part is updated. When a video is input, only the high-quality video feature vector part is updated. When the high-quality feature pool completes learning, a high-quality feature map is obtained through step 1.3). Then, under the action of the 3D VQGAN decoder, Mapping to image space to obtain the restored high-definition frame

[0067] 2) Continue to uniformly extract blind degraded frames from the blind degraded video to be restored, splice the restored high-definition frames with the extracted blind degraded frames in the time dimension, and enhance the attention of the blind degraded frames to the restored high-definition frames during the restoration process through attention weighting, thereby ensuring the consistency of the restored results with the content of the high-definition frames, wherein the blind degraded frames are restored by executing the above steps 1.1) to 1.3); specifically as follows:

[0068] Given the degraded frame V to be restored LQ And the HD frame recovered from step 1) Extract the corresponding latent space features and Spliced ​​together in the time dimension, calculated based on the self-attention mechanism and After the concatenation, the attention matrix A is then added with a weighted term ΔA to enhance right of concern, namely:

[0069] A'=A+ΔA

[0070] In the subsequent calculation of the self-attention mechanism, the weighted attention matrix A' is used to enhance the restoration process for the restored HD frame. The content of the HD frame is focused on, thereby ensuring that the restored results of other blind degraded frames can be aligned with the content of the restored HD frame.

[0071] 3) Repeat step 2) until all blindly degraded frames in the blindly degraded video to be restored are restored, and finally a coherent and high-definition video content is generated.

[0072] Further, for the learnable parameters involved in step 1.3), the present invention designs two training stages. In the first stage, the present invention fixes the 3D VQGAN decoder and only trains the remaining learnable parameters, relying on the cross entropy loss function to achieve the prediction of high-quality feature maps. After the first stage of training, although most blind degraded videos can be processed, the restored video may be slightly blurred due to the incomplete matching problem between the 3D VQGAN decoder and the predicted high-quality feature map. In order to further optimize the adaptability of the 3D VQGAN decoder and the predicted features, the present invention enters the second stage of training. In this stage, all learnable parameters except the 3D VQGAN decoder are fixed, and the parameters of the 3D VQGAN decoder part are focused on. The training of this stage introduces the L1 loss function, the perceptual loss function and the adversarial loss function, which improves the recovery ability of the 3D VQGAN decoder from multiple aspects. When the second stage of training is completed, the method of the present invention can restore richer and more delicate video details, significantly improving the overall quality of the video.

[0073] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for restoring time-stable blindly degraded video, characterized in that: The following steps are involved: 1) Obtain the blind degraded video to be restored, and evenly extract the blind degraded frames, and perform the following operations on each blind degraded frame: 1.1) Extract latent space features from the blind degraded frame, and use discrete wavelet transform to decompose the extracted latent space features into contents of different frequency bands; according to the degradation degree of the blind degraded frame, perform noise filtering on the contents of each frequency band to remove the blind degradation noise contained; finally, reassemble the filtered contents of each frequency band through inverse discrete wavelet transform to obtain smooth frame content; 1.2) Analyze the smooth frame content from three perspectives: target object, instance semantics, and overall description, to identify the location and semantic area of ​​the object and obtain the smooth frame content description text; By fusing information related to object location, semantic region, and text describing the content of smoothed frames, a comprehensive content understanding representation is generated to provide comprehensive understanding prior knowledge for subsequent recovery; 1.3) extracting latent space features from the smooth frame content obtained in step 1.1), and analyzing the spatiotemporal relationship of the extracted latent space features through a self-attention mechanism to obtain features with spatiotemporal perception; then, integrating the comprehensive understanding prior knowledge obtained in step 1.2) into the acquired features with spatiotemporal perception through a cross-attention mechanism; then, predicting a high-quality feature map in multiple steps, that is, determining which high-quality feature vectors in a pre-constructed high-quality feature pool the smooth frame content corresponds to, and using these high-quality feature vectors to characterize the smooth frame content, wherein the high-quality feature pool contains multiple high-quality feature vectors that can express high-quality video semantic content, and each high-quality feature vector has the same dimension; in each prediction step, firstly selecting a high-confidence prediction result, and extracting a corresponding high-quality feature vector from the high-quality feature pool to replace the latent space features of the smooth frame content, and then continuing to iteratively predict the remaining latent space features that have not been replaced, until all latent space features of the smooth frame content are replaced by high-quality feature vectors in the high-quality feature pool, and obtaining a complete high-quality feature map, and finally mapping the complete high-quality feature map to the image space to obtain a restored high-definition frame; 2) Continue to uniformly extract blind degraded frames from the blind degraded video to be restored, splice the restored high-definition frames with the extracted blind degraded frames in the time dimension, and enhance the attention of the blind degraded frames to the restored high-definition frames during the restoration process through attention weighting, thereby ensuring the consistency of the restored results with the content of the high-definition frames, wherein the blind degraded frames are restored by executing the above steps 1.1) to 1.3); 3) Repeat step 2) until all blindly degraded frames in the blindly degraded video to be restored are restored, and finally a coherent and high-definition video content is generated.

2. A method for restoring a time-stable blindly degraded video according to claim 1, characterized in that: In step 1.1), a video encoder is used to extract latent space features from the blind degraded frame. The video encoder is composed of a stack of multiple 3D convolutional layers, which decomposes the latent space features of the blind degraded frame into contents of different frequency bands: {F1,F2,...F f ,...,F B }=DWT(Z lq ) In the formula, Z lq represents the latent space features of the blind degraded frame, and DWT represents discrete wavelet transform, which is used to decompose the input content into content of different frequency bands {F1, F2, ... F f ,...,F B }, B represents the total number of decomposed frequency bands, and the symbol f is used to represent the serial number of the frequency band. The larger the serial number, the higher the frequency band, and the smaller the serial number, the lower the frequency band. That is, F1 represents the lowest frequency band content, and F B Indicates the content of the highest frequency band; in each frequency band, the cross-attention mechanism is used to analyze the degree of content degradation of the frequency band based on the content of the current frequency band and the reference content lower than the current frequency band. The method of analyzing the degree of degradation is expressed as follows: In the formula, F f and F f-1 represents the content of the fth and f-1th frequency bands in the latent space feature, Q f , K f-1 、V f-1 Represents the query features, key features, and value features of the corresponding frequency bands in the cross-attention mechanism, represents the learnable parameters of the corresponding frequency band, u represents the dimension of the key features of the cross-attention mechanism in the degradation degree analysis, δ represents the function used to analyze the degree of content degradation, which outputs the filter coefficient t for the frequency band f The greater the degree of degradation, the smaller the filter coefficient, the minimum value is 0, the smaller the degree of degradation, the larger the filter coefficient, the maximum value is 1, the filter coefficient of the frequency band is multiplied by the content of the frequency band to obtain the filtered content F′ of the frequency band f , denoted as F′ f =t f F f ; When analyzing the degradation degree of the lowest frequency band content F1, the self-attention mechanism is used without referring to the content of other frequency bands; finally, the filtered content of each frequency band is recombined through the inverse discrete wavelet transform: Z'=IDWT(F1',F′2,...F′ f ,...,F′ B ), Where IDWT stands for inverse discrete wavelet transform, which is used to transform the filtered content F1', F'2, ... F' in each frequency band f ,...,F′ B Recombine to obtain the filtered latent space feature Z'; Finally, a video decoder is used to map the latent space features Z' to the image space to obtain smooth frame content, and the video decoder consists of multiple stacked 3D convolutional layers.

3. A method for restoring time-stable blindly degraded video according to claim 2, characterized in that: In step 1.2), the BLIP-2 network is used to obtain the text describing the smooth frame content, the YOLOv8 detection network is used to detect the target object in the smooth frame content, and the SAM semantic segmentation network is used to segment the instance semantics in the smooth frame content; the T5 network is used to obtain text features from the text describing the smooth frame content, and the deep features of target detection and semantic segmentation are extracted from the YOLOv8 detection network and the SAM semantic segmentation network respectively. Then, based on the graph network, the text features, the deep features of target detection and semantic segmentation are fused to obtain a comprehensive content understanding representation, which provides comprehensive understanding prior knowledge for subsequent recovery.

4. The method for restoring a time-stable blindly degraded video according to claim 3, characterized in that: Based on the graph network, the deep features of text features, object detection and semantic segmentation are integrated as follows: Constructing an undirected graph in Represents a node set of an undirected graph, which includes n local local nodes and a global node v global , v1 represents the first local node, and so on, Indicates the nth local local nodes, Represents the edge set of an undirected graph; the local node is mapped by the deep features corresponding to the detection box generated by the YOLOv8 detection network and the deep feature mapping corresponding to the segmentation area generated by the SAM semantic segmentation network, and the global node v global The text feature representation extracted by the T5 network is connected to all local nodes as the central node; the nodes corresponding to the detection boxes of the same detection category are connected through the graph network to obtain the connection edge, and the detection confidence value of the YOLOv8 detection network is used to define the weight of the connection edge; the nodes corresponding to the segmentation areas of the same segmentation category are connected to obtain the connection edge, and the segmentation confidence of the SAM segmentation network is used to define the weight of the connection edge; all node features are transmitted through graph convolution, and the graph convolution process is defined as: In the formula, ω ij represents the i-th local node v i With the jth local node v j The fixed connection weight, h j represents the jth local node v j The characteristics of h i The i-th local node v i The features of , Λ represents the learnable weight matrix in graph convolution, represents the i-th local node v i The adjacent node set of v represents the activation function; through multi-layer graph convolution operations, the deep features of text features, object detection and semantic segmentation are gradually integrated into the global node, and finally from the global node v global Get a comprehensive understanding of prior knowledge Z prior =φ(v global ), where φ represents a linear mapping function that transforms v global Projection to Z prior .

5. A method for restoring time-stable blindly degraded video according to claim 4, characterized in that: In step 1.3), the latent space feature Z of the smooth frame content is given, and the feature Z with spatiotemporal perception is obtained based on the self-attention mechanism space ; Then, based on the cross attention mechanism, the feature Z with spatiotemporal perception space and comprehensive understanding of prior knowledge Z prior The fusion is performed, and the fusion method is as follows: Q=W Q Z space W=W K WITH prior V=W V Z prior Where Q, K, and V are the query features, key features, and value features in the cross-attention mechanism, respectively, and W Q , W K , W V is a learnable parameter, d is the dimension of feature K; based on the fused feature Z fuse , predict a high-quality feature map in multiple steps Among them, high-quality feature maps With Z fuse With the same dimension as Z, the prediction formula is as follows: In the formula, represents the prediction network, Represents the confidence of predicting the mth high-quality feature vector in the high-quality feature pool, N represents the size of the high-quality feature pool, Represents feature Z fuse The position in the latent space feature Z is the value of the f-th frame, h-th row, and w-th column. According to the above prediction formula, the f-th frame, h-th row, and w-th column position in the latent space feature Z corresponds to the l-th high-quality feature vector in the high-quality feature pool. Therefore, the l-th high-quality feature vector is taken from the high-quality feature pool to replace the value of the corresponding position in Z. When the values ​​of all positions in Z are replaced, a high-quality feature map can be obtained.

6. A method for restoring time-stable blindly degraded video according to claim 5, characterized in that: The high-quality feature vectors in the high-quality feature pool are learned based on the reconstruction of high-quality high-definition videos and images by a 3D VQGAN network. The 3D VQGAN network includes a 3D VQGAN encoder and a 3D VQGAN decoder. The high-quality feature vectors in the high-quality feature pool are divided into high-quality video feature vectors and high-quality image feature vectors. When an image is input, only the high-quality image feature vector part is updated. When a video is input, only the high-quality video feature vector part is updated. When the high-quality feature pool completes learning, a high-quality feature map is obtained through step 1.3). Then, under the action of the 3D VQGAN decoder, Mapping to image space to obtain the restored high-definition frame 7. A method for restoring time-stable blindly degraded video according to claim 6, characterized in that: In step 2), the degraded frame V to be restored is given LQ And the HD frame recovered from step 1) Extract the corresponding latent space features and Spliced ​​together in the time dimension, calculated based on the self-attention mechanism and After the concatenation, the attention matrix A is then added with a weighted term ΔA to enhance right of concern, namely: A'=A+ΔA In the subsequent calculation of the self-attention mechanism, the weighted attention matrix A' is used to enhance the restoration process for the restored HD frame. The content of the HD frame is focused on, thereby ensuring that the restored results of other blind degraded frames can be aligned with the content of the restored HD frame.

Images