A method, system, terminal and medium for implementing video error concealment

Abstract

The application provides a method, system, terminal and medium for realizing video error concealment, wherein an original video is compressed by coding and decoding to form a test set and a training set; frame extraction processing is performed on the test set and the training set, a mask block operation is performed, and a network training set is obtained; a network model is constructed, the network model is trained, and a video error concealment model is obtained; the video frame test set is input into the video error concealment model, and a reconstructed frame is output; a generator of the model uses a Video Swin Transformer network to perform video frame multi-layer feature extraction on the input video frame test set, then different levels of features are fused for reconstruction of a video loss or damaged area, and the reconstructed frame is output. The application solves the problem that the loss or damaged area of a VVC standard compressed video cannot be effectively recovered, and overcomes the disadvantage that a traditional convolutional neural network cannot simultaneously consider parallel execution and long-range time dependency modeling capability.

Description

Technical Field

[0001] This invention relates to the field of video communication technology, and more specifically, to a method, system, terminal, and medium for hiding video errors using the Transformer model. Background Technology

[0002] With the rapid development of multimedia, communication, and network technologies, video applications are becoming increasingly widespread. To meet the increasingly complex application scenarios and demands, video requires higher spatiotemporal resolution, wider viewing angles, and larger amounts of data, necessitating compression encoding for transmission over channels. However, given limited channel resources, efficient video signal compression is essential, presenting several challenges for video communication. First, the underlying transmission network is not always reliable, and data loss often occurs during transmission. Second, despite source or channel coding techniques for protection, erroneous reception or loss of compressed data is unavoidable. Furthermore, efficient video compression encoding removes significant spatial and temporal dependencies from video images, resulting in minimal redundancy in the encoded output bitstream. Consequently, the video stream is highly sensitive to errors during transmission. Therefore, methods for handling lost or erroneous video data are needed to improve the perceived video quality for end-users and meet their demands for high-quality video.

[0003] Video error concealment is a technique that operates solely at the decoding end, consuming no network bandwidth and generating only a small amount of latency to effectively improve video communication quality. It is only invoked when an error occurs, minimizing resource consumption and making it a crucial method for addressing bit error rates in video communication. Traditional video error concealment techniques utilize the spatial and temporal correlations of video data at the decoding end to recover lost or damaged areas without requiring additional information from the encoder. Based on the selection of reference information, it can be broadly categorized into spatial error concealment and temporal error concealment. Spatial error concealment leverages the correlation between spatially adjacent regions, i.e., the redundancy within a single video frame, to recover lost areas. Temporal error concealment utilizes temporal correlation, i.e., inter-frame redundancy information, to recover lost content.

[0004] Typical spatial error concealment methods include bilinear interpolation (BI) and direction interpolation (DI). Both utilize the boundary pixel information of the damaged macroblock for interpolation, the difference being the choice of interpolation direction and pixel calculation method. Lee YH, Lin CH, Chen CC, et al. The Video Spatial Error Concealment Algorithm Using Separately-Directional Interpolation Technique[J]. Journal of Signal Processing Systems, 2017, 88(1):13-27. This paper proposes an independent direction interpolation method that combines bilinear interpolation and direction interpolation. The method determines which interpolation method to use for recovery based on texture complexity information. Bilinear interpolation is very effective for uniform textures, while direction interpolation is suitable for complex textures dominated by edges. The independent direction interpolation method combines the advantages of both methods, thus improving the performance of error concealment. When the loss area is relatively large, spatially adjacent usable pixels are far from the lost pixels, and except in static and uniform scenes, the central part of the loss region has little correlation with spatially adjacent blocks. Therefore, simple spatial error hiding methods are not very effective at restoring the lost pixels.

[0005] Among the more classic temporal error concealment methods are the Outer Boundary Matching Algorithm (OBMA) and the Motion Vector Extrapolation (MVE). THAIPANICHT, WU Ping-hao, KUO C J. Low-Complexity Video Error Concealment for Mobile Applications Using OBMA[J].IEEE Trans on Consumer Electronics,2008,54(2):753-761. This paper proposes to use a linear translation model to hide the lost macroblocks. In the matching process, all adjacent macroblocks in one outer boundary layer are used, and the macroblock with the smallest distortion is selected as the best candidate macroblock. Q. Peng, T. Yang, C. Zhu, Block-based temporal error concealment for video packet using motion vector extrapolation. IEEE International Conference on Communications, Circuits and Systems and West Sino Expositions, 10–14, 2002. This paper proposes projecting the motion vector of the previous frame block into the current frame block and selecting the motion vector pointing to the block with the largest overlap area with the current block as the recovered motion vector for that block. This method preserves the object's motion well. To improve the concealment effect, some later improved algorithms changed the selection range of the reference frame and the calculation method of overlapping pixels, while others extended it to the pixel level. However, for static regions near moving objects, the motion vector extrapolation method sometimes fails to achieve good recovery results.

[0006] With the continuous development of deep learning, different types of network models have been developed and proposed, and have been widely applied in various fields. Currently, they have also proven to have achieved great success in the field of computer vision. Sankisa A, Punjabi A, Katsaggelos A K. Video Error Concealment Using Deep Neural Networks[C] / / 2018 25th IEEE International Conference on Image Processing (ICIP).IEEE, 2018. This paper proposes a network that combines convolutional long short-term memory (Conv LSTM) layers and simple convolutional layers to predict optical flow in lost regions. The predicted optical flow is combined with the damaged frame to predict the error / missing part of the frame, thereby achieving error concealment of the damaged frame. Xiang C, Xu J, Yan C, et al. Generative Adversarial Networks Based Error Concealment for Low Resolution Video[C] / / ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019. This paper proposes an error concealment method based on Generative Adversarial Networks (GANs). The generator network is a network architecture similar to U-Net, which extracts features and outputs a repaired image. Simultaneously, a global discriminator and a local discriminator are employed. This dual-discriminator structure significantly improves the speed and quality of image repair, ensuring a good visual experience for the concealed image in both global and local dimensions. For example, Chinese invention patent CN113256521A discloses a method and apparatus for concealing data-missing errors, which is based on the Generative Adversarial Network (GAN) network model PV-GAN and applied to error concealment technology for panoramic videos. The generator network of PV-GAN is a network similar to U-Net.

[0007] Most of the network models mentioned above for video error hiding use traditional convolutional neural networks to obtain spatiotemporal information of the video, and perform feature extraction and prediction through adjacent frames. They do not take into account the ability to perform parallel execution and model long-term time dependencies, which limits the feature extraction ability of the model. Summary of the Invention

[0008] In view of the deficiencies in the prior art, the purpose of this invention is to provide a method, system, terminal and medium for hiding video errors.

[0009] According to one aspect of the present invention, a method for implementing video error hiding is provided, comprising:

[0010] The original video is encoded, decoded, and compressed to form a test set and a training set. The test set consists of lossy videos, and the training set consists of lossless videos.

[0011] Frame extraction is performed on the test set and the training set to obtain a video frame test set and a video frame training set. A mask block is added to the video frame training set to obtain the network training set.

[0012] Construct a network model and train the network model using the network training set to obtain a video error concealment model;

[0013] The video frame test set is input into the video error hiding model, and the reconstructed frames are output to complete the video error hiding;

[0014] The video error concealment model uses a generative adversarial network (GAN) as its basic framework, comprising a generator and a discriminator. The generator employs a Video Swing Transformer (SSW) network to extract multi-layer features from the input video frame test set, and then fuses the features from different levels for the reconstruction of lost or damaged areas of the video, outputting reconstructed frames. The discriminator calculates an adversarial loss based on the reconstructed frames output by the generator to constrain the training of the video error concealment model and optimize its performance.

[0015] According to a second aspect of the present invention, a system for implementing video error hiding is provided, comprising:

[0016] Dataset preprocessing module: The original video is encoded and decoded to form a test set and a training set. The test set is a lossy video and the training set is a lossless video. Frame extraction is performed on the test set and the training set to obtain a video frame test set and a video frame training set. A mask block is added to the video frame training set to obtain the network training set.

[0017] Network model construction and training module: Construct a video error hiding model, and train the video error hiding model using the network training set obtained from the dataset preprocessing module to obtain the trained video error hiding model;

[0018] Reconstruction module: Inputs the video frame test set obtained from the dataset preprocessing module into the trained video error hiding model, outputs reconstructed frames, and completes video error hiding;

[0019] In the network model construction and training module, the video error hiding model uses a generative adversarial network as its basic framework, including a generator and a discriminator. The generator uses a Video Swin Transformer network to extract multi-layer features from the input video frame test set, and then fuses the features at different levels for the reconstruction of lost or damaged areas of the video, outputting reconstructed frames. The discriminator calculates adversarial loss based on the reconstructed frames output by the generator to constrain the training of the video error hiding model and optimize model performance.

[0020] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method for implementing video error hiding.

[0021] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, is used to perform the aforementioned method for implementing video error hiding.

[0022] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0023] The method and system for video error hiding provided in this invention overcome the limitations of feature extraction in traditional methods and solve the problem of not being able to effectively recover lost or damaged areas of the latest generation video coding standard VVC (Versatile Video Coding) compressed video. At the same time, it overcomes the drawbacks of traditional convolutional neural networks that cannot take into account both parallel execution and long-term time dependency modeling capabilities. It creatively integrates the Video Swin Transformer module into the generative adversarial network framework model, and uses its powerful feature extraction capabilities to make fuller use of the effective spatiotemporal information of video frames to complete the effective error hiding of damaged video frames.

[0024] The method and system for video error hiding provided in this invention use the Video SwinTransformer network, which incorporates hierarchy, locality, and translation invariance into the Transformer network structure design. This enables multi-scale feature extraction while significantly reducing computational complexity. As a result, it can fully utilize the spatiotemporal information between video frames to complete the effective video error hiding task, improve the video quality perceived by the end user, and meet people's needs for video quality. Attached Figure Description

[0025] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0026] Figure 1 This is a flowchart illustrating a method for hiding video errors according to an embodiment of the present invention.

[0027] Figure 2 This is a flowchart of a method for hiding video errors in a preferred embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of a network model in a preferred embodiment of the present invention;

[0029] Figure 4 This is a flowchart of the dataset preprocessing process in a preferred embodiment of the present invention;

[0030] Figure 5 This is a diagram of the internal structure of the Video Swin Transformer Block in a preferred embodiment of the present invention;

[0031] Figure 6 This is a detailed diagram of the model network structure in a preferred embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0033] Existing video error concealment techniques cannot effectively recover lost or damaged areas of VVC standard compressed videos due to limitations in feature extraction. Furthermore, the use of traditional convolutional neural networks cannot simultaneously address the drawbacks of parallel execution and long-term time-dependent modeling capabilities.

[0034] As a powerful tool for feature extraction, Transformer offers significant advantages in long-range feature capture, comprehensive feature extraction, and parallel computing capabilities. The Swing Transformer, in particular, incorporates hierarchy, locality, and translation invariance into its network structure design, enabling multi-scale feature extraction while significantly reducing computational complexity, achieving remarkable results in computer vision tasks. Therefore, this invention aims to overcome the shortcomings of traditional video error hiding methods by fully utilizing the spatiotemporal correlation between video frames. It employs the Video Swing Transformer to extract features from video frames, making fuller use of effective spatiotemporal information. Furthermore, by combining this with the basic framework of generative adversarial networks, a method and system for video error hiding using the Transformer model are proposed.

[0035] Reference Figure 1 The diagram shown is a flowchart of a method for hiding video errors in one embodiment of the present invention. Specifically, this embodiment includes the following steps:

[0036] S100 compresses and encodes the original video to form a test set and a training set;

[0037] In this step, the test set consists of lossy videos, and the training set consists of lossless videos. Specifically, the training set can be compressed lossless videos obtained through lossless encoding and decoding, while the test set can be compressed lossy videos obtained through lossy encoding and decoding.

[0038] S200: Perform frame extraction on the test set and training set to obtain the video frame test set and video frame training set. Add the mask block to the video frame training set to obtain the network training set.

[0039] In this step, frame skipping can be performed at the set frequency.

[0040] S300, Construct a network model, train the network model using the network training set, and obtain a video error hiding model; the video error hiding model uses a Transformer network;

[0041] Specifically, the video error hiding model uses a generative adversarial network as its basic framework, which includes a generator and a discriminator. The generator uses the Video Swin Transformer network to extract multi-layer features from the input video frame test set, and then fuses the features at different levels for the reconstruction of lost or damaged areas of the video, outputting reconstructed frames. The discriminator calculates adversarial loss based on the reconstructed frames output by the generator to constrain the training of the video error hiding model and optimize the model performance.

[0042] The S400 takes a set of video frame test sets as input to the video error hiding model and outputs reconstructed frames to complete the video error hiding.

[0043] In this embodiment of the invention, a generative adversarial network (GAN) is used as the basic framework. The Video Swin Transformer network serves as the foundational network for the generator part, enabling feature extraction from video frames. Features from different levels are then fused for reconstructing lost or damaged regions. For the network input, the original video undergoes frame extraction, and the video frames are superimposed with masks to simulate video frame corruption, which is then used as the overall network input. This invention integrates the Video Swin Transformer network into the GAN framework, and through training on a dataset, obtains the final video error concealment model.

[0044] In this embodiment of the invention, the Video Swin Transformer network is used as the base network of the generator of the video error hiding model. The generator first uses the encoder composed of the Video Swin Transformer network to extract multi-layer features from the damaged video frames of the input video frame test set and saves the feature output. Then, the decoder is used to reconstruct the frame, and the output of each decoding layer is combined with the encoder output features of the same size in a skip link manner as the input of the next decoding layer, so as to make full use of the feature information of different levels to recover the lost or damaged areas of the video frame.

[0045] In the above embodiments, by selecting the Video Swin Transformer network, the hierarchy, locality, and translation invariance are introduced into the Transformer network structure design, which can realize multi-scale feature extraction and greatly reduce the computational complexity. This allows for full utilization of the spatiotemporal information between video frames to complete an effective video error hiding task.

[0046] To better achieve multi-scale feature extraction, in some embodiments, the generator of the video error concealment model includes an encoder with n stages, implementing feature extraction and downsampling at n levels, and storing the feature output of each stage; n is a natural number greater than or equal to 2. Correspondingly, the decoder's decoding layer consists of a concatenated convolutional layer and an upsampling module to reconstruct the pixel structure of the video frame. The decoder needs to perform upsampling, typically a 2x upsampling, until the image is restored to the same size as the input video frame.

[0047] In a specific embodiment, n can be selected according to actual needs. For example, in one specific embodiment:

[0048] The encoder consists of four stages, each containing 2, 2, 6, and 2 Video Swin Transformer Blocks respectively. The first three stages also include a Patch Merging structure. The Patch Merging structure functions similarly to the pooling layer in a convolutional neural network, merging features from spatially adjacent tokens and applying a linear layer to reduce the dimensionality of the merged features.

[0049] Accordingly, the decoder can consist of four network layers (decoding layers), where:

[0050] The first three layers consist of two-dimensional convolution and bilinear upsampling modules. The outputs of these three layers are concatenated with the outputs of the corresponding coding layers in the channel dimension as the input of the next network layer.

[0051] The final layer consists of a 3D convolution and a trilinear upsampling module, which ultimately outputs a reconstructed video frame of the same size as the original video frame.

[0052] In the overall network model Figure 3 The diagram shows the splicing relationship between the outputs of corresponding layers in the encoder and decoder. The output of block partitioning and linear embedding (Patch Embed3D structure in the diagram) is spliced ​​with the output of the third decoding layer; the output of the first stage is spliced ​​with the output of the second decoding layer; and the output of the second stage is spliced ​​with the output of the first decoding layer. This splicing method achieves feature fusion to reduce feature loss. The corresponding dimensions mentioned above refer to the fact that the output dimensions of the corresponding layers are the same. In this embodiment, the combination of interpolation and convolutional layers can achieve better pixel restoration results. The first three layers of bilinear interpolation and two-dimensional convolutional layers gradually restore the video frame size, while the final trilinear interpolation and three-dimensional convolution additionally restore the temporal dimension of the video frame.

[0053] In some embodiments of this invention, to better achieve video error hiding by the video error hiding model, the input damaged video frame (network training set) is first processed by a 3D convolutional layer for shallow feature extraction to obtain an output feature map. The output feature map is then processed by block partitioning and linear embedding, and the result is used as the input to the Video Swin Transformer network. The Video Swin Transformer network is then used to perform multi-layer feature extraction on the input shallow feature map. In this preferred embodiment, the shallow feature extraction by the 3D convolutional layer can obtain texture information to help with texture reconstruction, thereby solving the problem of blurred texture in the reconstructed frame; block partitioning and linear embedding (Patch Embed3D structure in the figure) are used to obtain the input to the Video Swin Transformer network.

[0054] To enable the network model to focus more on image details during training, some embodiments use pixel-level reconstruction loss and adversarial loss as loss functions to constrain the video error hiding model. Since damaged and undamaged regions may have different importance, the reconstruction losses for both are calculated separately in the embodiments. Training on damaged frames is achieved by minimizing the pixel differences between the reconstructed frames and the original video frames. The discriminator attempts to distinguish the reconstructed frames from the original frames, and the adversarial loss calculated from its output allows training to obtain reconstruction results that more closely resemble the original video frames.

[0055] In the above embodiments of the present invention, random mask blocks are superimposed on the original video frames to simulate real video frame damage and are used as input to the network. Pixel-level reconstruction loss and adversarial loss are used as loss functions to constrain the model. The Adam optimizer is used for training, and the learning rate adjustment strategy, skipping link method, upsampling method, etc., are adjusted according to the results to determine the final model hyperparameters and detailed structure. The resulting video error hiding model breaks through the limitations of feature extraction in traditional methods, effectively recovering lost or damaged areas of VVC standard compressed videos. It also overcomes the shortcomings of traditional convolutional neural networks in simultaneously handling parallel execution and long-term time-dependent modeling capabilities. Utilizing the powerful feature extraction capabilities of the Video Swin Transformer, it more fully leverages the effective spatiotemporal information of video frames to complete effective video error hiding tasks, while significantly reducing computational complexity, improving the video quality perceived by the end user, and meeting people's demands for video quality.

[0056] Based on the same technical concept, another embodiment of the present invention also provides a system for hiding video errors. Specifically, the system in this embodiment includes:

[0057] Dataset preprocessing module: The original video is encoded and decoded to form a test set and a training set. The test set is a lossy video and the training set is a lossless video. Frame extraction is performed on the test set and the training set to obtain a video frame test set and a video frame training set. Mask blocks are added to the video frame training set to obtain the network training set.

[0058] Network model building and training module: Builds the network model and trains it using the network training set obtained from the dataset preprocessing module to obtain the video error hiding model;

[0059] Reconstruction module: Inputs the video frame test set obtained from the dataset preprocessing module into the video error hiding model, outputs reconstructed frames, and completes video error hiding;

[0060] In the network model construction and training module, the video error hiding model uses a generative adversarial network as its basic framework, including a generator and a discriminator. The generator uses the Video Swin Transformer network to extract multi-layer features from the input video frame test set, and then fuses the features at different levels for the reconstruction of lost or damaged areas of the video, outputting reconstructed frames. The discriminator calculates the adversarial loss based on the reconstructed frames output by the generator to constrain the training of the video error hiding model and optimize the model performance.

[0061] The system for hiding video errors in this embodiment uses techniques for the specific implementation of each module, which can be referred to in the corresponding steps of the method, and will not be repeated here.

[0062] Based on the same technical concept, another embodiment of the present invention also provides a terminal, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it is used to execute the method for implementing video error hiding in any of the above embodiments.

[0063] Based on the same technical concept, another embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to perform the method for implementing video error hiding in any of the above embodiments.

[0064] To better illustrate the technical solutions in the above embodiments of the present invention, the following description is based on an application of a preferred embodiment. However, it should be understood that the present invention is not limited to the following preferred embodiment.

[0065] In this preferred embodiment, experiments were conducted in a Windows 10 and PyTorch environment. First, the original dataset was preprocessed to obtain a corresponding video frame training set, and random mask blocks were superimposed to simulate real-world video frame corruption as input to the network. After the corrupted video frames were input into the video error hiding model, they sequentially passed through a shallow feature extraction layer and an encoding layer composed of Video Swing Transformers to obtain and save feature outputs at different levels. Then, four decoding layers composed of convolutional and upsampling layers were used, with each decoding layer's output combined with the same-sized encoding layer's output features via skip connections as input to the next decoding layer. This fully utilizes feature information at different levels to effectively recover lost or corrupted areas of the video frames. For model training, reconstruction loss and adversarial loss were used to jointly constrain the model, resulting in error hiding results whose content and details closely resembled the original uncorrupted video frames. The overall process is as follows: Figure 2 As shown.

[0066] Specifically, refer to Figure 2 As shown in the figure, the method for video error hiding using the Transformer model provided in this preferred embodiment has the following specific implementation steps:

[0067] Step 1. Preprocess the dataset:

[0068] Specific reference Figure 4 As shown:

[0069] First, the video format is converted from the original MP4 format video to YUV(420) format video. Then, 100 videos of different scenes are randomly selected as the test set, and the remaining videos are used as the training set.

[0070] Then, the YUV sequences are compressed and encoded using the VVC reference software (VTM 18.0). The training set is compressed into lossless video through lossless encoding and decoding, while the test set is compressed into lossy video through lossy encoding and decoding.

[0071] Subsequently, frames are extracted from both the training and test sets at a frequency of 8 frames per second to obtain the video frame training set and video frame test set. Of course, in other embodiments, other frequencies can be used to extract frames from the dataset, not limited to 8 frames per second, and the specific frequency can be selected according to the actual application requirements.

[0072] Since the network needs to input damaged video frames, randomly generated mask blocks are added to the original video frame training set to simulate the damage to video frames caused by real packet loss. This operation forms the network training set, which is used as the final input for training the video error hiding model.

[0073] Step 2. Design the overall network model:

[0074] The overall network model is based on the generative adversarial network framework, which includes two parts: a generator and a discriminator.

[0075] In this embodiment, the encoder part of the generator uses the Video Swin Transformer as the base network, which contains four stages. The number of Video Swin Transformer Blocks in each stage is 2, 2, 6, and 2 respectively. The feature maps obtained by shallow feature extraction of the video frame sequence through 3D convolutional layers are first processed by patch partitioning and linear embedding to obtain 3D tokens. These tokens are then fed into the Video Swin TransformerBlock for feature extraction and downsampling, and the feature output of each stage is saved.

[0076] In this embodiment, the decoder part of the generator consists of convolutional layers and an upsampling module, containing four network layers. The first three layers use a two-dimensional convolutional layer and bilinear interpolation upsampling method, while the last layer uses a three-dimensional convolutional layer and trilinear interpolation upsampling method. A jump linking method of concat in the channel dimension is selected to combine the output of each layer with encoder features of the same size as the input of the next layer, gradually restoring the image size and time dimension, thereby completing the reconstruction of the lost region.

[0077] In this embodiment, the discriminator uses the fully convolutional network PatchGAN as the main body, which consists of 6 cascaded 3D convolutional layers. The discriminator obtains the corresponding feature maps based on the reconstructed frames of the generator and the ground truth (original undamaged frames), and uses them to calculate the adversarial loss to constrain the model, so that the model can enhance spatiotemporal consistency and pay more attention to the details of the image during the training process.

[0078] The overall network model constructed in this embodiment is as follows: Figure 3 As shown, the specific network characteristics are detailed as follows: Figure 6 As shown, the model uses a generative adversarial network (GAN) as its basic framework, comprising a generator and a discriminator. The generator is further divided into an encoder and a decoder. The encoder uses the Video Swin Transformer as its base network, while the decoder uses convolutional layers. The specific construction process of the above model includes:

[0079] Step 2.1 For the input damaged video frame (T×W×H×3), it first passes through a 3D convolutional layer to extract shallow features and obtain the output feature map (T×W×H×C), which is then fed into the Video Swin Transformer network.

[0080] Step 2.2 Construct the Video Swin Transformer network. The overall architecture is as follows: Figure 2 The encoder part is shown in Figure 2.1. The feature map output from step 2.1 is obtained by patch partitioning and linear embedding. 3D tokens serve as input to the subsequent structure. The subsequent structure is the encoder part of the generator, consisting of four stages. Each stage contains 2, 2, 6, and 2 Video Swin Transformer Blocks respectively. The first three stages also contain a Patch Merging structure. The four stages are concatenated sequentially, and the Video Swin Transformer Blocks (VSTBs in the overall network model diagram) within each stage are also concatenated. The Patch Merging structure is concatenated at the end of each stage; see the overall network model diagram for details.

[0081] Typically, VideoSwin Transformer Blocks appear in pairs, implementing window multi-head self-attention (Video W-MSA) and sliding window multi-head self-attention (Video SW-MSA) respectively. The internal structure of two consecutive VideoSwin Transformer Blocks is as follows: Figure 5 As shown, the mathematical expression for its calculation is as follows:

[0082]

[0083]

[0084]

[0085]

[0086] Where 3DW_MSA represents self-attention based on 3D regular windows, 3DSW_MSA represents self-attention based on 3D sliding windows, and FFN is a multilayer perceptron based on GELU. Figure 5 In MLP, LN represents layer normalization. Comparing the internal structure diagrams of two consecutive Video Swin Transformer Blocks, we can see that fl-1 This represents the input of the current first block, which is the output of the previous block; f represents the intermediate output of the current first block. l This represents the final output of the first block, and also the input of the second block. This represents the intermediate output of the current second block, f. l+1 This indicates the final output of the current second block.

[0087] Patch merging is analogous to Max Pooling in convolutional neural networks. It merges the features of spatially adjacent tokens and applies a linear layer to reduce the dimensionality of the merged features.

[0088] Step 2.3 Construct the decoder module of the generator. This module consists of four network layers. The first three layers are composed of two-dimensional convolution and bilinear upsampling, and their outputs are concatenated with the encoder output of the corresponding size in the channel dimension as the input of the next network layer. The last layer is composed of three-dimensional convolution and trilinear upsampling, and the final output is a reconstructed video frame of the same size as the original video frame.

[0089] Step 3. Design the loss function:

[0090] In this embodiment, pixel-level reconstruction loss (L1 loss) and adversarial loss (Adversarial Loss) are used as loss functions. The reconstruction loss for damaged regions (including missing regions) and undamaged regions may have different impacts on model training. Therefore, pixel-level reconstruction losses for both regions are calculated separately, and content reconstruction training for damaged frames is achieved by minimizing the pixel differences between the reconstructed frames and the original video frames. Using the feature map output by the PatchGAN discriminator to calculate the adversarial loss allows the model to pay more attention to detail and texture, resulting in reconstruction results that are closer to the original video frames.

[0091] Specifically, the loss function includes pixel-level reconstruction loss and adversarial loss, and the reconstruction loss of damaged and undamaged regions is calculated separately. Its specific mathematical expression is as follows:

[0092] Step 3.1. Construct the reconstruction loss function

[0093] The reconstruction loss of the damaged area is

[0094]

[0095] The reconstruction loss of the undamaged area is

[0096]

[0097] Where M represents the mask block and Y represents the original undamaged video frame. The synthesized video frames output by the network are shown in the diagram. ⊙ denotes element-wise multiplication, and the final result is normalized according to the size of the corresponding region. Content reconstruction training of damaged frames is achieved by minimizing the pixel differences between the synthesized and original video frames.

[0098] Step 3.2. Construct the adversarial loss function

[0099] Since the discriminator needs to distinguish between the synthesized video frames and the original video frames, and the generator needs to synthesize video frames that can "fool" the discriminator, the adversarial loss function of the discriminator is:

[0100]

[0101] The adversarial loss function of the generator is:

[0102]

[0103] Therefore, the final joint loss function of the generator can be expressed as:

[0104]

[0105] Where, ω hole ω valid ω adv These represent the weights of the corresponding loss function, and in this embodiment, the specific parameter value is ω. hole =1, ω valid =1, ω adv =0.01. The discriminator can distinguish between true and false spatiotemporal features, thereby constraining model training through adversarial loss, which can enhance the spatiotemporal consistency of the model output and improve the perceptual details of the reconstructed frames.

[0106] Step 4. Model Training:

[0107] In this embodiment, the dataset (training set) preprocessed in step 1 is used as the input to the network model. The batch size is 1, the sample length of the video frame sequence is 16, the two loss functions in step 3 are used as model constraints, the Adam optimizer is used to train the network, the learning rate is set to 0.0001, the exponential decay rate beta1 of the Adam first moment estimation is 0, and the exponential decay rate beta2 of the Adam second moment estimation is 0.9.

[0108] After the above training, a video error hiding model is obtained. The video frame test set processed in step 1 is input into the video error hiding model, and the reconstructed frames are output to complete the video error hiding.

[0109] In this preferred embodiment, a generative adversarial network (GAN) is used as the basic framework of the model. The generator first uses an encoder composed of a Video Swin Transformer network to extract multi-layer features from the input damaged video frames and saves the feature output. Then, a decoding layer composed of convolutional layers and upsampling layers is used to reconstruct the frames. The output of each decoding layer is combined with the output features of the encoding layer of the same size in a skip-link manner as the input of the next decoding layer, thereby making full use of feature information at different levels to effectively restore the damaged areas of the video frames. The reconstructed frames and ground truth are both input to the discriminator to calculate the adversarial loss. The adversarial loss, combined with the pixel-level reconstruction loss, constrains the model training to optimize model performance, so that the reconstructed frames output by the model are more consistent with the original undamaged video frames in terms of content, spatiotemporal consistency, and perceptual detail. The Video Swin Transformer network selected in this embodiment introduces hierarchy, locality, and translation invariance into the Transformer network structure design, which can realize multi-scale feature extraction while greatly reducing computational complexity. This allows for full utilization of the spatiotemporal information between video frames to complete the effective video error hiding task. This embodiment breaks through the limitations of feature extraction in traditional methods and solves the problem of not being able to effectively restore damaged areas of VVC standard compressed videos. At the same time, it overcomes the drawbacks of traditional convolutional neural networks that cannot take into account both parallel execution and long-term time dependency modeling capabilities.

[0110] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding modules, devices, units, etc. in the system. Those skilled in the art can implement the steps of the method by referring to the technical solution of the system. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0111] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0112] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A method for hiding video errors, characterized in that, include: The original video is processed to form a test set and a training set. The training set is a compressed lossless video obtained through lossless encoding and decoding, and the test set is a compressed lossy video obtained through lossy encoding and decoding. Frame extraction is performed on the test set and the training set to obtain a video frame test set and a video frame training set. A mask block is added to the video frame training set to obtain the network training set. Construct a network model and train the network model using the network training set to obtain a video error concealment model; The video frame test set is input into the video error hiding model, and the reconstructed frames are output to complete the video error hiding; The video error concealment model uses a generative adversarial network (GAN) as its basic framework, comprising a generator and a discriminator. The generator employs a Video Swin Transformer network to extract multi-layer features from the input video frame test set, and then fuses the features from different levels for the reconstruction of lost or damaged areas of the video, outputting reconstructed frames. The discriminator calculates an adversarial loss based on the reconstructed frames output by the generator to constrain the training of the video error concealment model and optimize its performance.

2. The method for hiding video errors according to claim 1, characterized in that, The generator is set as follows: First, an encoder composed of a Video Swin Transformer network is used to extract multi-layer features from the damaged video frames in the input video frame test set and save the feature output. Then, the decoder performs frame reconstruction, and the output of each decoding layer is combined with the encoder output features of the same size in a skip-link manner as the input of the next decoding layer, so as to make full use of the feature information of different levels to recover the lost or damaged areas of the video frame.

3. The method for hiding video errors according to claim 2, characterized in that, The encoder layer comprises n stages, implementing feature extraction and downsampling at n levels, and storing the feature output of each level; n is a natural number greater than or equal to 2. The decoding layer consists of a convolutional layer and an upsampling module, which realizes pixel reconstruction of video frames.

4. The method for hiding video errors according to claim 3, characterized in that, The encoding layer consists of four stages, each containing 2, 2, 6, and 2 Video Swin Transformer Blocks respectively. The first three stages also contain a Patch Merging structure.

5. The method for hiding video errors according to claim 3, characterized in that, The decoder consists of four network layers, wherein: The first three layers consist of two-dimensional convolution and bilinear upsampling modules. The outputs of these three layers are concatenated with the outputs of the corresponding coding layers in the channel dimension as the input of the next network layer. The final layer consists of a 3D convolution and a trilinear upsampling module, which ultimately outputs a reconstructed video frame of the same size as the original video frame.

6. The method for hiding video errors according to claim 1, characterized in that, The video error hiding model is constrained by using pixel-level reconstruction loss and adversarial loss as loss functions.

7. The method for hiding video errors according to any one of claims 1-6, characterized in that, Training the network model using the network training set further includes: The input damaged video frame first passes through a 3D convolutional layer to extract shallow features, resulting in an output feature map; The output feature map is then partitioned into blocks and linearly embedded, and the result is used as the input to the Video Swin Transformer network.

8. A system for hiding video errors, characterized in that, include: Dataset preprocessing module: Processes the original video to form a test set and a training set. The training set is a compressed lossless video obtained through lossless encoding and decoding, and the test set is a compressed lossy video obtained through lossy encoding and decoding. Frame extraction is performed on the test set and the training set to obtain a video frame test set and a video frame training set. Mask blocks are added to the video frame training set to obtain the network training set. Network model construction and training module: Constructs a network model and trains the network model using the network training set obtained from the dataset preprocessing module to obtain a video error hiding model; Reconstruction module: Inputs the video frame test set obtained by the dataset preprocessing module into the video error hiding model, outputs reconstructed frames, and completes video error hiding; In the network model construction and training module, the video error hiding model uses a generative adversarial network as its basic framework, including a generator and a discriminator. The generator uses a Video Swin Transformer network to extract multi-layer features from the input video frame test set, and then fuses the features at different levels for the reconstruction of lost or damaged areas of the video, outputting reconstructed frames. The discriminator calculates adversarial loss based on the reconstructed frames output by the generator, and combines it with pixel-level reconstruction loss to jointly constrain the training of the video error hiding model to optimize model performance.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it is used to perform the method for implementing video error hiding as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program is used to perform the method for implementing video error hiding as described in any one of claims 1-7.

Images