A video inpainting method based on time attention and generative adversarial network
By employing a video restoration method based on temporal attention and generative adversarial networks, and utilizing the content continuity between video frames to limit a reasonable range of attention queries, the problems of high computational cost and inappropriate content filling in existing technologies are solved, thereby improving the quality and efficiency of video restoration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2023-05-29
- Publication Date
- 2026-06-02
AI Technical Summary
Existing attention-based video restoration methods fail to effectively utilize the content continuity between video frames, resulting in an unreasonable attention query range, high computational cost, or inability to find suitable filling content, thus affecting the quality of video restoration.
We employ a temporal attention and generative adversarial network (GAN) approach. By training a generator and a discriminator, we leverage the content continuity between video frames to define a reasonable range of attention queries. We then use a temporal Transformer and a T-PatchGAN discriminator to optimize the generator's loss function and perform video frame inpainting.
It improves the quality and efficiency of video restoration, solves the problems of large computational load and inappropriate content filling, and has engineering application value.
Smart Images
Figure CN116703760B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, specifically relating to a video restoration method based on temporal attention and generative adversarial networks. Background Technology
[0002] Video restoration aims to fill in missing areas of a video with realistic and coherent content. It has wide applications in film post-processing, video editing, and other fields, such as wire work removal, target removal, and repairing damaged videos. With the rapid development of artificial intelligence, video processing technology has entered a comprehensive AI era. Utilizing AI techniques for video restoration can effectively improve the quality and efficiency of the restoration process.
[0003] Existing attention-based video restoration methods generally do not give sufficient consideration to the continuity of content between video frames, resulting in attention query ranges that are either too large or too small. An excessively large query range leads to a high computational cost for attention and may result in inappropriate or even incorrect fill content being retrieved within that range; conversely, a small query range may result in the absence of suitable fill content within that narrow range. Both of these situations directly impact the quality of video restoration. Summary of the Invention
[0004] To address the problems and shortcomings of existing technologies, the present invention aims to provide a video restoration method based on temporal attention and generative adversarial networks. Based on the continuous nature of content between video frames, this invention proposes temporal attention, solving the problem of inappropriate attention query region size and providing a reasonable query region. This, in turn, solves the problem of high computational cost of the network model and the inability to find suitable filling content, thereby improving the quality of video restoration.
[0005] Based on the above objectives, the technical solution adopted by the present invention is as follows:
[0006] This invention provides a video restoration method based on temporal attention and generative adversarial networks, comprising the following steps:
[0007] Step 1: Preprocess the videos in the training set and randomly generate a binary mask for each video frame;
[0008] Step 2: Input the video frames and their corresponding mask images into the generative adversarial network to train the deep network model;
[0009] Step 3: Input the damaged video frames into the network model trained in Step 2, and output the repair results.
[0010] The specific operation of step 1 is as follows:
[0011] 1-1: The training set uses video frame sequences. In order to make the video frames meet the training requirements, the resolution of the video frames is adjusted and scaled to 240*432.
[0012] 1-2: Generate a binary mask for each video frame, where 0 represents a damaged area and 1 represents an undamaged area, with a size of 240*432; the height of the mask in the mask image is between 120 and 239, the width is between 216 and 431, and the shape and relative position in the image are random.
[0013] The specific operation of step 2 is as follows:
[0014] 2-1: The network model consists of a generator and a discriminator. The generator consists of five parts: an encoder, an overlap segmentation module, a temporal Transformer, an overlap synthesis module, and a decoder.
[0015] 2-2: Randomly select 5 video frames from each video in the preprocessed training set, called the original video frames, and arrange them in chronological order to form the original video frame set X = {x0, x1, x2, x3, x4}, where x0, x1, x2, x3, x4 represent a single video frame; the set of mask images corresponding to these 5 video frames is M = {m0, m1, m2, m3, m4}, where m0, m1, m2, m3, m4 represent a single mask image;
[0016] 2-3: The generator takes the original video frame set X and the mask image set M as input. Where h = 240, w = 432 represent the height and width of the video frame, and 3 is the length of the video frame channel. Represents the set of real numbers; a set of mask images Where 1 represents the length of the video frame channel; each video frame in the original video frame set is multiplied pixel-by-pixel with the corresponding single mask in the mask image set to obtain the video frame set containing the missing region. Set I is encoded to obtain the latent feature set c is the channel length; F obtains the latent feature set after passing through the overlapping segmentation module. d is the channel length; E is the latent feature set obtained after passing through 8 cascaded time Transformers. The latent feature set is obtained through the overlapping synthesis module. The decoder produces a set of repaired video frames Y = {y0, y1, y2, y3, y4}, where y0, y1, y2, y3, and y4 represent the repaired individual video frames.
[0017] 2-4: The input to the discriminator is the repaired video frame set Y and the original video frame set X. The discriminator discriminates the input video frames and outputs a number between 0 and 1, where 0 indicates that the input video frame is false and 1 indicates that the input video frame is true.
[0018] 2-5: The training objective of the discriminator is to distinguish the repaired video frames from the original video frames, while the training objective of the generator is to prevent the discriminator from identifying the generated video frames as fake. The objective function of the generator consists of three loss terms: damaged area reconstruction loss, effective area reconstruction loss, and generator adversarial loss. The expression of the objective function is as follows:
[0019] L=λ hole ·L hole +λ valid ·L valid +λ adv ·L adv
[0020] In the formula, L represents the objective function of the generator, L hole L represents the mean absolute error of the damaged area in a video frame. valid L represents the mean absolute error of the undamaged region in a video frame. adv Indicates generator adversarial loss; λ hole , λ valid and λ adv They are L hole L valid and L adv The corresponding weighting coefficient, L hole The value is 1, L valid The value is 1, L adv The value is 0.01;
[0021] L hole The expression is as follows:
[0022]
[0023] L valid The expression is as follows:
[0024]
[0025] In the formula, Y is the set of repaired video frames, X is the set of original video frames, M is the set of mask images, ⊙ represents pixel-by-pixel multiplication, and |||| represents the calculation of the arithmetic mean.
[0026] Adversarial loss L of the discriminator D The detailed expression is as follows:
[0027] L D =EX [logD(X)]+E Y [log(1-D(Y))]
[0028] Where D represents the discriminator; X represents the original set of video frames; Y represents the repaired set of video frames; D(X) represents the output generated by inputting the original set of video frames into the discriminator D; D(Y) represents the output generated by inputting the repaired set of video frames into the discriminator D; E X E represents the arithmetic mean; Y This indicates the calculation of the arithmetic mean;
[0029] Generator's adversarial loss L adv The expression is as follows:
[0030] L adv =E Y [logD(Y)]
[0031] 2-6: The network model uses the Adam optimizer with an initial learning rate of 0.0001 and a total of 500,000 iterations. When the number of iterations exceeds 400,000, the learning rate decays to 0.00001, and the batch size is set to 8.
[0032] The encoder consists of convolutional layers and activation functions, and its function is to extract latent features from video frames.
[0033] The overlapping segmentation module differs from the commonly used direct cutting method (where there is no overlap between the cut image blocks). It cuts the latent features in an overlapping manner to generate image blocks, meaning that the edges of two adjacent image blocks overlap.
[0034] The overlapping synthesis module operates in the reverse order of the overlapping segmentation module. This module stitches image patches together with overlapping edges to generate latent features. This segmentation and synthesis method not only allows information from different image patches to be aggregated at overlapping locations, but also helps to smooth the boundaries of image patches and expand the receptive field.
[0035] The Temporal Transformer comprises temporal attention, a fusion feedforward layer, and two normalization layers. Its function is to find suitable content from latent features to fill missing regions. The Temporal Transformer includes temporal attention, a fusion feedforward layer, and two normalization layers. The temporal attention layer includes dimensionality splitting, top-bottom temporal attention, left-right temporal attention, dimensionality concatenation, and a linear layer. The input to the temporal attention layer is a set of feature maps. First, feature set The feature set is divided into two equal parts along the channel dimension, resulting in the first feature set. Second feature set Next, the first feature set Second feature set Enter vertical time attention and horizontal time attention respectively. Vertical time attention is... In terms of height dimension Divide into equal parts, and obtain upper half feature set and The lower half of the feature set and Each input is processed by a multi-head attention mechanism, and the outputs of the multi-head attention mechanism are then concatenated in a high-dimensional array to obtain the desired result. This represents the first feature set after processing; left and right temporal attention are... In the width dimension Divide into equal parts, and obtain The left half of the feature set and The right half of the feature set and Each input is processed by a multi-head attention module, and the outputs of the multi-head attention module are concatenated along the width dimension to obtain the result. This represents the second feature set after processing; then, the output of the up-and-down time attention. and the output of attention over time The latent feature set is obtained by concatenating the data along the channel dimension. Finally, the final output latent features are obtained after passing through a linear layer. The basic structure of the fusion feedforward layer is overlapping segmentation and overlapping synthesis. The processed latent features first pass through a linear layer, then their shape is adjusted to generate image patches. These image patches are then used to synthesize latent feature maps using overlapping synthesis, followed by overlapping segmentation to divide them into image patches. These patches are then adjusted to the shape of the input latent features and output through a linear layer. The fusion feedforward layer replaces the commonly used MLP layer, thus enhancing information exchange between image patches.
[0036] The decoder consists of upsampling operations, convolutional layers, and activation functions, and its function is to restore the latent features into video frames of the same size as the input.
[0037] The discriminator uses a T-PatchGAN discriminator, which consists of six 3D convolutional layers. The T-PatchGAN discriminator can not only focus on different spatiotemporal features, but also make full use of all global and local image features as well as temporal information.
[0038] The specific operation of step 3 is as follows:
[0039] 3-1: Select video frames whose serial numbers are multiples of 10 from the input damaged video frames as reference video frames. For example, if there are 73 damaged video frames in total, with serial numbers from 0 to 72, then select frames 0, 10, 20, 30, 40, 50, 60, and 70 as reference video frames.
[0040] 3-2: Use batch repair to repair damaged video frames. Select 10 damaged video frames sequentially each time. For example, the sequence numbers of the batch of damaged video frames selected sequentially are 10, 11, 12, 13, 14, 15, 16, 17, 18, and 19.
[0041] 3-3: Each input consists of a selected damaged video frame and a reference video frame. Based on the examples in steps 3-1 and 3-2, the sequence numbers of the input video frames are: 0, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 30, 40, 50, 60, 70. The generator trained in step 2 is used to repair these video frames, outputting the repaired video frames. This batch repair process continues until all damaged video frames are repaired.
[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0043] (1) Starting from the importance of the continuity of content between video frames, this invention proposes a video restoration method based on temporal attention and generative adversarial networks, and explores a new method for video restoration that makes full use of the continuity of content between video frames.
[0044] (2) This invention uses temporal attention to limit the scope of attention queries, which can solve the problem of unreasonable query regions.
[0045] This solves problems such as large computational load and difficulty in finding suitable content to fill in.
[0046] (3) Compared with existing methods, the present invention has certain improvements in repair quality and efficiency, and has engineering application value. Attached Figure Description
[0047] Figure 1 This is a generator network architecture for a video inpainting method based on temporal attention and generative adversarial networks.
[0048] Figure 2 The network architecture for the discriminator.
[0049] Figure 3 The network architecture for the time-based Transformer.
[0050] Figure 4 A network architecture for time attention, vertical time attention, and horizontal time attention.
[0051] Figure 5 The network architecture for the overlapping synthesis module and the overlapping segmentation module.
[0052] Figure 6 A network architecture for integrating feedforward layers.
[0053] Figure 7 The network model trained using this method is applied to the restoration results of damaged videos.
[0054] Figure 8 The network model trained using this method is applied to the repair results of target removal. Detailed Implementation
[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0056] This invention provides a video restoration method based on temporal attention and generative adversarial networks, comprising the following steps:
[0057] Step 1: Preprocess the videos in the training set by randomly generating a binary mask for each video frame. The specific steps are as follows:
[0058] 1-1: The training set consists of video frame sequences. To make the video frames meet the training requirements, the resolution of the video frames is adjusted and scaled to 240*432;
[0059] 1-2: Generate a binary mask for each video frame, where 0 represents a damaged area and 1 represents an undamaged area, with a size of 240*432. The height of the mask in the mask image is between 120 and 239, the width is between 216 and 431, and the shape and relative position in the image are random.
[0060] Step 2: Input the video frames and their corresponding mask images into the generative adversarial network (GAN) to train the deep network model. The specific operations are as follows:
[0061] 2-1: The network model consists of a generator and a discriminator. The generator consists of five parts: an encoder, an overlap segmentation module, a temporal Transformer, an overlap synthesis module, and a decoder.
[0062] 2-2: Randomly select 5 video frames from each video in the preprocessed training set, called the original video frames, and arrange them in chronological order to form the original video frame set X = {x0, x1, x2, x3, x4}, where x0, x1, x2, x3, x4 represent individual video frames. The set of mask images corresponding to these 5 video frames is M = {m0, m1, m2, m3, m4}, where m0, m1, m2, m3, m4 represent individual mask images.
[0063] 2-3: For example Figure 1 As shown, the generator takes the original video frame set X and the mask image set M as input. Where h = 240, w = 432 represent the height and width of the video frame, and 3 is the length of the video frame channel. Represents the set of real numbers; a set of mask images Where 1 represents the length of the video frame channel. Each video frame in the original video frame set is multiplied pixel-by-pixel by the corresponding single mask in the mask image set to obtain the video frame set containing the missing region. I obtains the latent feature set through the encoder. c is the channel length; F obtains the latent feature set after passing through the overlapping segmentation module. d is the channel length; E is the latent feature set obtained after passing through 8 cascaded time Transformers. The latent feature set is obtained through the overlapping synthesis module. The decoder produces a set of repaired video frames Y = {y0, y1, y2, y3, y4}, where y0, y1, y2, y3, and y4 represent the repaired individual video frames.
[0064] 2-4: The input to the discriminator is the repaired video frame set Y and the original video frame set X. The discriminator discriminates the input video frames and outputs a number between 0 and 1, where 0 indicates that the input video frame is false and 1 indicates that the input video frame is true.
[0065] 2-5: The training objective of the discriminator is to distinguish the repaired video frames from the original video frames, while the training objective of the generator is to prevent the discriminator from identifying the generated video frames as fake. The generator's objective function consists of three loss terms: damaged area reconstruction loss, effective area reconstruction loss, and generator adversarial loss. The expression for the objective function is as follows:
[0066] L=λ hole ·L hole +λ valid ·L valid +λ adv ·Ladv
[0067] In the formula, L represents the objective function of the generator, L hole L represents the mean absolute error of the damaged area in a video frame. valid L represents the mean absolute error of the undamaged region in a video frame. adv This indicates that the generator is adversarial against loss. λ hole , λ valid and λ adv They are L hole L valid and L adv The corresponding weighting coefficient, L hole The value is 1, L valid The value is 1, L adv The value is 0.01.
[0068] L hole The expression is as follows:
[0069]
[0070] L valid The expression is as follows:
[0071]
[0072] In the formula, Y is the set of repaired video frames, X is the set of original video frames, M is the set of mask images, ⊙ represents pixel-by-pixel multiplication, and |||| represents the calculation of the arithmetic mean.
[0073] Adversarial loss L of the discriminator D The expression is as follows:
[0074] L D =E X [logD(X)]+E Y [log(1-D(Y))]
[0075] Where D represents the discriminator; X represents the original set of video frames; Y represents the repaired set of video frames; D(X) represents the output generated by inputting the original set of video frames into the discriminator D; D(Y) represents the output generated by inputting the repaired set of video frames into the discriminator D; E X E represents the arithmetic mean; Y This indicates the calculation of the arithmetic mean;
[0076] Generator's adversarial loss L adv The expression is as follows:
[0077] L adv =E Y [logD(Y)]
[0078] 2-6: The network model uses the Adam optimizer, where the momentum gradient descent factors β1 = 0 and β2 = 0.99. The initial learning rate is 0.0001, and the total number of iterations is 500,000. When the number of iterations exceeds 400,000, the learning rate decays to 0.00001. The batch size is set to 8.
[0079] The encoder consists of convolutional layers and activation functions, and its function is to extract latent features from video frames.
[0080] The overlapping segmentation module differs from the commonly used direct cutting method (where there is no overlap between the cut image blocks). It cuts the latent features in an overlapping manner to generate image blocks, meaning that the edges of two adjacent image blocks overlap.
[0081] The overlapping synthesis module, in the reverse process of the overlapping segmentation module, stitches image patches together with overlapping edges to generate latent features. This segmentation and synthesis method not only allows information from different image patches to be aggregated at overlapping locations, but also helps to smooth the boundaries of image patches and expand the receptive field. The overlapping synthesis and segmentation module, as shown... Figure 5 As shown.
[0082] The temporal Transformer includes temporal attention, a fusion feedforward layer, and two normalization layers, such as... Figure 3 As shown, its function is to find suitable content from latent features to fill in missing regions. The Temporal Transformer includes temporal attention, a fusion feedforward layer, and two normalization layers. Temporal attention includes dimensionality splitting, top-bottom temporal attention, left-right temporal attention, dimensional concatenation, and a linear layer, as shown... Figure 4 As shown in (a), the input to temporal attention is a set of feature maps. First, feature set The first feature set is obtained by dividing it into two equal parts along the channel dimension. Second feature set Next, the first feature set Second feature set Entering vertical time attention and horizontal time attention respectively, as follows: Figure 4 As shown in (b) and 4(c). Attention is focused on the preceding and following time periods. In terms of height dimension Divide into equal parts, and obtain upper half feature set and The lower half of the feature set and Each input is processed by a multi-head attention module, and the outputs of the multi-head attention module are then concatenated along the height dimension to obtain the result. This represents the first feature set after processing. Left and right temporal attention are... In the width dimension Divide into equal parts, and obtain The left half of the feature set and The right half of the feature set and Each input is processed by a multi-head attention module, and the outputs of the multi-head attention module are concatenated along the width dimension to obtain the result. Then, the output of attention in the preceding and following time periods. and the output of attention over time The latent feature set is obtained by concatenating the data along the channel dimension. Finally, the final output latent features are obtained after passing through a linear layer. The basic structure of the fusion feedforward layer is overlapping segmentation and overlapping synthesis, such as... Figure 6 As shown, the processed latent features first pass through a linear layer, then their shape is adjusted to generate image patches. These image patches are first synthesized into a latent feature map through overlapping synthesis, then divided into image patches through overlapping segmentation, and finally adjusted to the shape of the input latent features before passing through a linear layer for output. The fusion feedforward layer replaces the commonly used MLP layer, which enhances the information interaction between image patches.
[0083] The decoder consists of upsampling operations, convolutional layers, and activation functions, and its function is to restore the latent features into video frames of the same size as the input.
[0084] The discriminator uses a T-PatchGAN discriminator, which consists of six 3D convolutional layers, such as... Figure 2 As shown, the T-PatchGAN discriminator can not only focus on different spatiotemporal features, but also make full use of all global and local image features as well as temporal information.
[0085] Step 3: Input the damaged video frames into the network model trained in Step 2, and output the repair results. The specific operation is as follows:
[0086] 3-1: Select video frames whose serial numbers are multiples of 10 from the input damaged video frames as reference video frames. For example, if there are 73 damaged video frames in total, with serial numbers from 0 to 72, then select frames 0, 10, 20, 30, 40, 50, 60, and 70 as reference video frames.
[0087] 3-2: Use batch repair to repair damaged video frames. Select 10 damaged video frames sequentially each time. For example, the sequence numbers of the batch of damaged video frames selected sequentially are 10, 11, 12, 13, 14, 15, 16, 17, 18, and 19.
[0088] 3-3: Each input consists of a selected damaged video frame and a reference video frame. Based on the examples in steps 3-1 and 3-2, the sequence numbers of the input video frames are: 0, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 30, 40, 50, 60, 70. The generator trained in step 2 is used to repair these video frames, outputting the repaired video frames. This batch repair process continues until all damaged video frames are repaired.
[0089] This invention provides a video restoration method based on temporal attention and generative adversarial networks, which can utilize the continuity of content between video frames to restore damaged videos. A schematic diagram of the generator structure is shown below. Figure 1 As shown, the structural schematic diagram of the discriminator is as follows: Figure 2 As shown, the time Transformer is as follows Figure 3 As shown, time attention, left-right time attention, and up-down time attention are as follows: Figure 4 As shown in (a), (b), and (c), overlapping synthesis and overlapping segmentation are as follows: Figure 5 As shown, the fusion feedforward layer is as follows Figure 6 As shown. Figure 7 The restoration results of repairing damaged video frames using the present invention are shown, wherein, Figure 7 (a) is the original video frame. Figure 7 (b) is a mask diagram indicating the damaged area. Figure 7 (c) is the repaired video frame. Figure 8 The results of target removal using this invention are shown, wherein... Figure 8 (a) is the original video frame. Figure 8 (b) A mask map indicating the target area. Figure 8 (c) represents the repaired video frame. This invention, based on temporal attention and generative adversarial networks, proposes a video restoration method that leverages the continuity of content between video frames. This approach specifies a reasonable attention query region, addressing the problem of unreasonable attention query regions and effectively improving the restoration quality of damaged videos. This invention can be applied to fields such as film post-processing and video editing.
[0090] The above embodiments are intended to illustrate the essential content of the present invention, but are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of protection of the present invention.
Claims
1. A video restoration method based on temporal attention and generative adversarial networks, characterized in that, Includes the following steps: Step 1: Preprocess the videos in the training set and randomly generate a binary mask for each video frame; Step 2: Input the video frames and their corresponding mask images into the Generative Adversarial Network (GAN) for deep network model training. The network model consists of a generator and a discriminator. The generator consists of five parts: an encoder, an overlap segmentation module, a temporal Transformer, an overlap synthesis module, and a decoder. The temporal Transformer includes temporal attention, a fusion feedforward layer, and two normalization layers. The temporal attention includes dimensionality splitting, top-bottom temporal attention, left-right temporal attention, dimensionality concatenation, and a linear layer. Step 3: Input the damaged video frames into the network model trained in Step 2, and output the repair results.
2. The video restoration method based on temporal attention and generative adversarial networks according to claim 1, characterized in that, The preprocessing in step 1 includes the following steps: 1-1: The training set uses video frame sequences. In order to make the video frames meet the training requirements, the resolution of the video frames is adjusted and scaled to 240*432. 1-2: Generate a binary mask for each video frame, where 0 represents a damaged area and 1 represents an undamaged area, with a size of 240*432; the height of the mask in the mask image is between 120 and 239, the width is between 216 and 431, and the shape and relative position in the image are random.
3. The video restoration method based on temporal attention and generative adversarial networks according to claim 1, characterized in that, Step 2, the network model training, includes the following steps: 2-1: The network model consists of a generator and a discriminator. The generator consists of five parts: an encoder, an overlap segmentation module, a temporal Transformer, an overlap synthesis module, and a decoder. 2-2: Randomly select 5 video frames from each video in the preprocessed training set, which are called the original video frames, and arrange them in chronological order to form the original video frame set. ,in This represents a single video frame; the set of mask images corresponding to these 5 video frames. ,in Represents a single mask image; 2-3: The generator uses the original set of video frames. X and mask image set M As input, the original set of video frames ,in h =240, w =432, representing the height and width of the video frame, and 3 is the length of the video frame channel. Represents the set of real numbers; the mask image is a set. Where 1 is the length of the video frame channel; each video frame in the original video frame set is multiplied pixel-by-pixel by the corresponding single mask in the mask set to obtain the video frame set containing the missing region. ;gather I The encoder obtains the latent feature set. , where c is the length of the channel; F The latent feature set is obtained after the overlap segmentation module. d is the length of the channel; E The latent feature set is obtained after passing through 8 cascaded time-transformers. ; The latent feature set is obtained through the overlapping synthesis module. ; The video frame set after being repaired by the decoder ,in This indicates a single video frame that has been repaired. ; 2-4: The input to the discriminator is the repaired set of video frames. Y and the original video frame set X The discriminator identifies the input video frame and outputs a number between 0 and 1, where 0 indicates that the input video frame is false and 1 indicates that the input video frame is true. 2-5: The training objective of the discriminator is to distinguish the repaired video frames from the original video frames, while the training objective of the generator is to prevent the discriminator from identifying the generated video frames as fake. The objective function of the generator consists of three loss terms: damaged area reconstruction loss, effective area reconstruction loss, and generator adversarial loss. The expression of the objective function is as follows: ； In the formula L Describes the objective function of the generator. This represents the mean absolute error of the damaged area in a video frame. This represents the mean absolute error of the undamaged area in a video frame. This indicates that the generator is adversarial against loss; , and They are , and The corresponding weighting coefficients, The value is 1. The value is 1. The value is 0.01; The expression is as follows: ； The expression is as follows: ； in, Y This is the collection of repaired video frames. X The original set of video frames. M For a set of mask images, This indicates pixel-by-pixel multiplication; This indicates the calculation of the arithmetic mean; Adversarial loss of the discriminator The expression is as follows: ； in, D Indicates the discriminator; X Represents the original set of video frames; Y This represents the set of repaired video frames; This represents the output generated when the original set of video frames is input into the discriminator D; This represents the output generated when the repaired set of video frames is input into the discriminator D. This indicates the calculation of the arithmetic mean; This indicates the calculation of the arithmetic mean; Generator's adversarial loss The expression is as follows: ； 2-6: The network model uses the Adam optimizer with an initial learning rate of 0.0001 and a total of 500,000 iterations. When the number of iterations exceeds 400,000, the learning rate decays to 0.00001, and the batch size is set to 8.
4. The video restoration method based on temporal attention and generative adversarial networks according to claim 3, characterized in that, The encoder consists of convolutional layers and activation functions, and its function is to extract latent features from video frames. The overlapping segmentation module cuts the latent features in an overlapping manner to generate image blocks, that is, the edges of two adjacent image blocks overlap. The overlapping synthesis module, which operates in the opposite manner to the overlapping segmentation module, stitches together image blocks with overlapping edges to generate latent features. This method of segmentation and synthesis not only allows overlapping areas to aggregate information from different image patches, but also helps to smooth the boundaries of image patches and expand the receptive field. The temporal Transformer includes temporal attention, a fusion feedforward layer, and two normalization layers. Its function is to find suitable content from latent features to fill in the missing regions. The decoder consists of upsampling operations, convolutional layers, and activation functions, and its function is to restore the latent features into video frames of the same size as the input. The discriminator uses a T-PatchGAN discriminator, which consists of 6 3D convolutional layers. The T-PatchGAN discriminator can not only focus on different spatiotemporal features, but also make full use of all global and local image features and temporal information.
5. The video restoration method based on temporal attention and generative adversarial networks according to claim 4, characterized in that, The temporal Transformer includes temporal attention, a fusion feedforward layer, and two normalization layers; Temporal attention includes dimensionality splitting, top-bottom temporal attention, left-right temporal attention, dimensionality concatenation, and linear layers; the input to temporal attention is a set of feature maps. First, the feature set The feature set is divided into two equal parts along the channel dimension, resulting in the first feature set. Second feature set Next, the first feature set Second feature set Separately enter vertical time attention and horizontal time attention; vertical time attention is in In terms of height dimension Divide into equal parts, and obtain upper half feature set and The lower half of the feature set , and Each input is processed by a multi-head attention mechanism, and the outputs of the multi-head attention mechanism are then concatenated in a high-dimensional array to obtain the desired result. , This represents the first feature set after processing; left and right temporal attention are... In the width dimension Divide into equal parts, and obtain The left half of the feature set and The right half of the feature set , and Each input is processed by a multi-head attention module, and the outputs of the multi-head attention module are concatenated along the width dimension to obtain the result. , This represents the second feature set after processing; Then, the output of attention in the preceding and following time periods. and the output of attention over time The latent feature set is obtained by concatenating the data along the channel dimension. Finally, the final output latent features are obtained after passing through a linear layer. ; The basic structure of the fusion feedforward layer is overlapping segmentation and overlapping synthesis. The processed latent features first pass through a linear layer, and then the shape is adjusted to generate image blocks. The image blocks are first synthesized into latent feature maps by overlapping synthesis, and then divided into image blocks by overlapping segmentation. They are then adjusted to the shape of the latent features at the input and output through a linear layer. The fusion feedforward layer replaces the commonly used MLP layer and enhances the information interaction between image blocks.
6. The video restoration method based on temporal attention and generative adversarial networks according to claim 1, characterized in that, The damaged video repair in step 3 includes the following steps: 3-1: Select video frames whose serial numbers are multiples of 10 from the input damaged video frames as reference video frames; 3-2: Use batch repair to repair damaged video frames, selecting 10 damaged video frames sequentially each time; 3-3: Each input consists of a selected damaged video frame and a reference video frame; the generator trained in step 2 is used to repair these video frames, and the repaired video frames are output. Then, batch repair is performed until all damaged video frames are repaired.