Video image edge information restoration method based on deep learning

Abstract

The invention discloses a video image edge information restoration method based on deep learning. Belongs to the image processing field. According to the method, a video edge connection (VEC) model is designed, an Encoder and a Decoder are adopted by the model to carry out coding and decoding on image and edge information, and feature interaction and propagation are realized on the basis of a space-time decoupling Transform. The model is trained through YouTube-vos and Davis data sets, and the data sets cover various scenes such as urban streetscape, natural wind and light, indoor scenes and character activities and have rich and diversified edge features, so that the model can accurately learn a feature mode of a normal video image edge. Compared with a method for repairing edge information of a single image by Edsection, the method has the advantages that the time sequence information of the video is fully utilized, and the transmission of the time sequence information among the video frames is realized through a Transform network architecture, so that the repairing result is more coherent in time sequence and more credible in semantics; and the problem of inter-frame flicker of a video image edge information repairing result is relieved to a certain extent.

Description

Technical Field

[0001] The invention belongs to the technical field of video image processing, and in particular relates to an edge information restoration technology. Background Art

[0002] With the rapid development of video technology, video data occupies an increasingly critical position in people's daily life, work and study. Whether in the fields of film and television creation, video editing or target recognition, high-quality video images are the basis for ensuring the smooth progress of various tasks. However, in the transmission, storage or processing of videos, many factors may lead to the loss of video image information, which in turn causes the loss and damage of edge information. For example, during network transmission, unstable network signals are prone to packet loss, resulting in the loss of some image data; storage media may also damage the stored video image data due to aging, physical damage and other reasons; and some traditional processing algorithms have limitations when dealing with complex image transformations, which will also cause the loss of image information. These problems seriously affect the quality and availability of video images, and bring great obstacles to subsequent video editing, image recognition and film and television production.

[0003] In the field of edge information repair, deep learning technology has made certain progress. Among them, Edgeconnect, as a method based on deep learning, can repair the edge information of a single image. It builds a neural network structure through Encoder, ResNet, and Decoder to learn the characteristics of the image, thereby realizing the reconstruction of the damaged edge. However, there is an essential difference between video images and single images. Videos are composed of a series of continuous image frames and contain rich temporal information. Edgeconnect, a repair method for a single image, cannot effectively utilize this temporal information when processing video images. This leads to problems such as flickering, incoherence, and unreasonable semantics between frames in the repaired video image. For example, in film and television post-production, if there is a problem of temporal incoherence in video edge repair, the audience will clearly notice the abruptness and unnaturalness of the picture; in video surveillance scenarios, flickering between frames and unreasonable semantics may affect the accurate identification and tracking of surveillance targets, seriously reducing the value of video images in practical applications.

[0004] With the development of deep learning technology, the present invention introduces the Transformer network architecture. With its unique attention mechanism, this architecture breaks the limitations of traditional neural networks in processing sequence information and can accurately capture the subtle correlations between video frames. By performing cross-frame modeling on the edge features of each frame in the video, the Transformer network architecture realizes the efficient transmission and fusion of temporal information between video frames. Different from Edgeconnect which only focuses on single-frame image restoration, this method can use the temporal features of the previous and subsequent frames as a reference to dynamically adjust the restoration strategy when restoring the edge information of video images. This not only makes the restored edge information seamlessly connect in the time dimension but also highly conforms to the video content at the semantic level, effectively alleviating the problem of frame-by-frame flickering in the restoration results of video image edge information and providing a better solution for video image edge information restoration. Summary of the Invention

[0005] The present invention aims to solve the problems that in the process of video image transmission, storage or processing, due to improper operations, edge information is missing or damaged, and existing restoration methods cannot effectively utilize video temporal information, resulting in frame-by-frame flickering, temporal incoherence and semantic unreasonableness in the restoration results. A method for restoring video image edge information based on deep learning is provided. It is expected to achieve high-precision restoration of missing or damaged video image edge information, make the restoration results coherent in time and credible in semantics, improve the quality and usability of video images, and be widely applied in fields such as video restoration, film and television production, and image recognition.

[0006] The technical solution adopted by the present invention is as follows: A method for restoring video image edge information based on deep learning, comprising:

[0007] S1. Preprocessing of the video image data to be restored. (1) Obtain the damaged area mask masks of the video image frames, where masks is a binary image (0 represents the normal area, 1 represents the damaged area); (2) Calculate the edge information edges of the video frames frames through the canny operator; (3) Obtain masked_frames and masked_edges through frames×(1 - masks) and edges×(1 - masks);

[0008] S2. Construct a dataset according to (2) in step S1; perform data preprocessing on the train, test, and val video segments in the YouTube–vos and Davis datasets according to (2) in step S1 to obtain the edges required for training;

[0009] S3. Establish a VEC (Video Edge Connect) model that uses an Encoder and a Decoder to encode and decode images and edge information, and realizes feature interaction and propagation based on a spatio-temporal decoupled Transformer;

[0010] S4. Divide the dataset processed in step S2 into training sets, validation sets, and test sets according to the YouTube–vos and Davis datasets (to further improve efficiency, the training set and the test set can be combined). Then, send the masked_frames, masked_edges, and masks obtained in step S1 into the VEC model in step S3 for training;

[0011] S5. Select video segments not used for model training, that is, video segments in neither the training set nor the validation set in step S4. Perform data preprocessing through step S1 and use it as the input of the VEC model for model inference to complete the repair of video image edge information.

[0012] Further, step S1 is specifically as follows:

[0013] S11. Obtain the damaged area masks masks of the video images frames. Use the publicly available dataset QD-IMD: QuickDraw Irregular Mask Dataset, or generate a random polygon mask as masks through an algorithm;

[0014] S12. Read the video images frames through OpenCV-Python and convert them to grayscale images. Use the cv2.Canny function to set appropriate thresholds to process frames to obtain the edge ground truth Ground Truth edges;

[0015] S13. Obtain masked_frames and masked_edges through formula (3) in step S1;

[0016] Further, step S3 is specifically as follows:

[0017] S31. The VEC network structure includes three major modules: an encoder module, a spatio-temporal decoupled attention module, and a decoder module;

[0018] S32. Build the VEC network structure using the pytorch framework;

[0019] S33. Set appropriate neural network training parameters;

[0020] Further, step S31 is specifically as follows:

[0021] S311. Data preprocessing. The damaged video images masked_frames and the damaged edge images masked_edges obtained in step (3) of step S1 are respectively concatenated with the masks masks in the feature channel dimension through the torch.cat function in the pytorch framework. For details, refer to the attached Figure 1 .

[0022] S312. Encoder module, that is, the VEC network is a feature extraction module that transfers the image from the spatial domain to the feature domain. It consists of an Edge encoder and a Frame encoder. For details, refer to the attached drawings of the specification Figure 2 and Figure 3 for construction;

[0023] S313. The Edge encoder and the Frame encoder in the encoder module respectively process the Input edges and Input frames obtained in step S311 to obtain outputs x0 and x1;

[0024] S314. Spatiotemporal decoupled attention module, that is, the VEC network is a propagation module that respectively performs temporal and spatial information interaction. It consists of Edge patch2vec, Frame patch2vec, Pos embedding, the temporal attention module TTransformer, the spatial attention module S Transformer, and Transformer blocks. Among them, Edge patch2vec and Frame patch2vec are two independent Conv2d networks. The position encoding Pos embedding is constructed in detail referring to the attached drawings of the specification Figure 4 for construction. TTransformer and S Transformer are constructed referring to the FGT (Flow-Guided Transformer for Video Inpainting) paper. Transformer blocks are multiple stacked TTransformer and S Transformer (2 in this method);

[0025] S315. Edge patch2vec and Frame patch2vec respectively process the outputs x0 and x1 obtained in step S313, and convert their common patch data format in the convolutional neural network into the token form commonly used in the attention network to obtain outputs x2 and x3;

[0026] In S316, the x2 obtained in step S315 sequentially passes through a T Transformer and a Pos embedding to obtain outputs x4 and x5 respectively; x5 and the x3 obtained in step S315 are jointly input into an S Transformer to obtain an output x6; x6 and x3 are used as inputs to pass through Transformer blocks to obtain an output x7;

[0027] S317. The decoder module, i.e., the VEC network, decodes and restores the feature domain information to the spatial domain to obtain the repair result. It consists of Vec2patch and Decoder, and the detailed structure is shown in the attached drawings of the specification Figure 5 and Figure 6 is constructed accordingly;

[0028] In S318, the output x7 obtained in step S316 passes through Vec2patch to be transformed from the common token form of the attention network back to the common format patch of the convolutional neural network and obtain an output x8; x8 and the output x0 obtained in step S313 are subjected to pixel-level addition to obtain x9, and x9 is then upsampled by the Decoder to obtain the result x10;

[0029] Furthermore, the specific steps of step S4 are as follows:

[0030] S41. For each video clip in the YouTube–vos and Davis datasets, the ground truth edges of the video images are obtained according to the method of step S2, i.e., step S12.

[0031] S42. For each video clip, the selected video frames are processed through steps S11 and S13 to obtain masked_frames, masked_edges, and masks, which are then fed into the VEC model in step S3 for training.

[0032] Furthermore, the specific steps of step S5 are as follows:

[0033] S51. Select a video clip f0 that has not been used for model training, and obtain its masked_frames, masked_edges, and masks through steps S11 and S13 as the input of the VEC model for model inference to obtain the output x10 in step S318;

[0034] S52. Obtain the final repair result x11 through f0×(1 - mask)+x10×mask.

[0035] Compared with the edge repair method of Edgeconnect for a single image, this method makes full use of the temporal information of the video, and realizes the transfer of temporal information between video frames through the Transformer network architecture, making the repair results more coherent in time series and more credible semantically, and alleviating the problem of frame-by-frame flicker in the repair results of video image edge information to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of the training data preprocessing of the present invention.

[0037] Figure 2 It is a structural diagram of the Edge encoder network of the present invention.

[0038] Figure 3 It is a structural diagram of the Frame encoder network of the present invention.

[0039] Figure 4 It is a structural diagram of the Pos embedding network of the present invention.

[0040] Figure 5 It is a structural diagram of the Vec2patch network of the present invention.

[0041] Figure 6 It is a structural diagram of the Decoder network of the present invention.

[0042] Figure 7 It is a structural diagram of the VEC network of the present invention.

[0043] Figure 8 It is a schematic diagram of the edge repair result in the specific implementation manner of the present invention. SPECIFIC IMPLEMENTATION MANNER

[0044] To facilitate the understanding of the technical content of the present invention by those skilled in the art, the content of the present invention will be further explained below with reference to the accompanying drawings.

[0045] A method for repairing video image edge information based on deep learning according to the present invention first calculates the edge ground truth edges of video segments of the YouTube–vos and Davis datasets to obtain the dataset required for training. Then, the damaged area masks are obtained, and masked_frames and masked_edges are obtained according to the formulas frames×(1 - masks) and edges×(1 - masks). Then, masked_frames, masked_edges, and masks are sent into the VEC network for training, and finally, the trained edge information repair model is used for inference calculation. The implementation process of the present invention includes the following steps:

[0046] S1. Preprocessing of video image data to be repaired. (1) Obtain the damaged area masks masks of the video images frames, where masks is a binary image (0 represents the normal area, and 1 represents the damaged area); (2) Calculate the edge information edges of the video frames frames through the canny operator; (3) Obtain masked_frames and masked_edges through frames×(1 - masks) and edges×(1 - masks);

[0047] S11. Obtain the damaged area masks masks of the video images frames. Use the publicly available dataset QD-IMD: QuickDraw Irregular Mask Dataset, or generate a random polygon mask as masks through an algorithm;

[0048] S12. Read the video images frames through OpenCV-Python and convert them to grayscale images. Use the cv2.Canny function to set appropriate thresholds to process frames to obtain the edge ground truth Ground Truth edges;

[0049] S13. Obtain masked_frames and masked_edges through the formula in step (3) of S1;

[0050] S2. Construct a dataset according to step (2) in S1; perform data preprocessing on the train, test, and val video segments in the YouTube–vos and Davis datasets according to step (2) in S1 to obtain the edges required for training;

[0051] S3. Establish a VEC (video edge connect) model that encodes and decodes images and edge information using an Encoder and a Decoder, and realizes feature interaction and propagation based on a spatio-temporal decoupled Transformer;

[0052] S31. The VEC network structure includes three major modules: an encoder module, a spatio-temporal decoupled attention module, and a decoder module;

[0053] S311. Data preprocessing. Concatenate the damaged video image masked_frames and the damaged edge image masked_edges obtained in step (3) of S1 with the mask masks respectively in the feature channel dimension through the torch.cat function in the pytorch framework. For detailed reference, see the attached Figure 1 .

[0054] S312. The encoder module, i.e., the feature extraction module of the VEC network that transfers the image from the spatial domain to the feature domain. It consists of an Edge encoder and a Frame encoder. For details, refer to the accompanying drawings of the specification. Figure 2 and Figure 3 are constructed; among them, the ConvBlock convolutional layer of the Edge encoder is VanillaConv, the LeakyReLU layer of the Frame encoder selects the parameter 0.2, and the Conv2d convolutional layers of the [4, 5, 6, 7, 8] layer networks are grouped convolutions, and the groups are [1, 2, 4, 8, 1] respectively;

[0055] S313. In the encoder module, the Edge encoder and the Frame encoder respectively process the Input edges and Input frames obtained in step S311 to obtain the outputs x0 and x1;

[0056] S314. The spatio-temporal decoupled attention module, i.e., the propagation module of the VEC network that performs temporal and spatial information interaction respectively. It consists of Edge patch2vec, Frame patch2vec, Pos embedding, and the temporal attention module TTransformer, the spatial attention module S Transformer, and Transformer blocks. Among them, Edge patch2vec and Frame patch2vec are two independent Conv2d networks, and the position encoding Pos embedding is constructed in detail referring to the accompanying drawings of the specification Figure 4 (where the Conv2d convolutional layer is a grouped convolution with groups = 512), TTransformer and S Transformer are constructed referring to the FGT (Flow-Guided Transformer for Video Inpainting) paper, and Transformer blocks are multiple stacked T Transformers and S Transformers (in this method, there are 2);

[0057] S315. Edge patch2vec and Frame patch2vec respectively process the outputs x0 and x1 obtained in step S313, and convert them from the common patch data format of the convolutional neural network to the token form commonly used in the attention network to obtain the outputs x2 and x3;

[0058] In S316, the x2 obtained in step S315 sequentially passes through a T Transformer and a Pos embedding to obtain outputs x4 and x5 respectively; x5 and the x3 obtained in step S315 are jointly input into an S Transformer to obtain an output x6; x6 and x3 are used as inputs to pass through Transformer blocks to obtain an output x7;

[0059] S317. The decoder module, i.e., the VEC network, decodes and restores the feature domain information to the spatial domain to obtain the restoration result. It consists of Vec2patch and Decoder, and for the detailed structure, refer to the attached drawings of the specification Figure 5 and Figure 6 is built; among them, the deconvolution layer of the DeconvBlock in the Decoder is VanillaDeconv, and the ConvBlock is VanillaConv;

[0060] In S318, the output x7 obtained in step S316 passes through Vec2patch to be transformed from the common token form of the attention network back to the common format patch of the convolutional neural network and obtains an output x8; x8 and the output x0 obtained in step S313 perform pixel-level addition to obtain x9, and x9 is then upsampled by the Decoder to obtain the result x10;

[0061] S32. Use the pytorch framework to build the VEC network structure;

[0062] S33. Set appropriate neural network training parameters as shown in the hyperparameter settings in Table 1;

[0063] S4. The processed dataset in step S2, according to the training set, validation set, and test set divided by the YouTube–vos and Davis datasets (to further improve the efficiency and effect, the training set and test set can be combined), passes through step S1 to obtain masked_frames, masked_edges, and masks and is sent into the VEC model in step S3 for training;

[0064] S41. For each video segment of the YouTube–vos and Davis datasets, obtain the ground truth edges of the video images according to step S2, that is, the method in step S12;

[0065] S42. Select video frames from each video segment and send the masked_frames, masked_edges, and masks obtained through steps S11 and S13 into the VEC model in step S3 for training;

[0066] S5. Select the video clips not used for model training, i.e., the video clips in the training set and validation set in step S4. Perform data preprocessing through step S1 and use it as the input of the VEC model for model inference to complete the repair of the edge information of the video image;

[0067] S51. Select the video clip f0 not used for model training, and obtain its masked_frames, masked_edges, and masks through steps S11 and S13 as the input of the VEC model for model inference to obtain the output x10 in step S318;

[0068] S52. Obtain the final repair result x11 through f0×(1 - mask)+x10×mask

[0069] Table 1 Hyperparameter settings

[0070]

[0071] In the above table, L1V and L1M are the L1 losses of the unoccluded region and the occluded region respectively.

[0072] With this improved neural network structure and training method, high-precision repair of the missing or damaged edge information of video images is achieved, making the repair results coherent in time series and credible in semantics, effectively alleviating the problem of frame flickering in the repair results of video image edge information, and providing a better solution for the repair of video image edge information. Finally, the performance of the trained model is evaluated using the validation set. According to the evaluation results of the test set, adjust the hyperparameters and structure of the model, including learning rate, batch size parameters, etc., to improve the generalization ability and performance of the model.

[0073] Evaluate the performance of the model trained under the current parameters and structure through the validation set, and further optimize the model according to the evaluation results to improve its generalization ability and performance. In terms of performance evaluation, objective evaluation metrics peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used, and the calculation methods are as follows:

[0074]

[0075] Among them, Max I is the maximum value of the image pixel points (e.g., if the pixel point value range is 0 - 255, then Mxa I is 255), and MSE is the mean square error between the test image I and the ground truth image I'. If the image to be calculated is a color image, the common processing methods are:

[0076] (1) Calculate PSNR separately in the three RGB channels and then calculate their average value;

[0077] (2) The MSE is calculated and summed separately in the three RGB channels, and then the average value of the MSE is obtained for calculating the overall PSNR.

[0078] For the peak signal-to-noise ratio (PSNR), the smaller the MSE, the larger the PSNR, which also represents better image quality. However, the peak signal-to-noise ratio ignores the human visual characteristics. Generally, the human eye is more sensitive to brightness, spatial frequency contrast, and the structure of the perceived area. Therefore, if only PSNR is used as an objective evaluation index, it may lead to inconsistent evaluation results of the restored image with the subjective feeling of people. For this reason, this paper adopts structural similarity (SSIM) as another objective evaluation index, which measures the similarity of images from three factors: brightness, spatial frequency contrast, and structure. The calculation formula is as follows:

[0079] SSIM(I, I') = [l(I, I')^α · c(I, I') β · s(I, I') γ

[0080] Among them, l(I, I'), c(I, I'), and s(I, I') are the components of luminance, contrast, and structure respectively, and the calculation formulas are as follows:

[0081]

[0082] Among them, μ I , μ I' represent the mean values of images I and I' respectively, σ I , σ I' represent the standard deviations of images I and I' respectively, represent the variances of images I and I' respectively, σ II' represents the covariance of images I and I', and C1, C2, and C3 are constants to avoid system errors caused by the denominator being zero. The calculation formulas are respectively:

[0083]

[0084] C1 = (K1Max I ) 2 , C2 = (K2Max I ) 2 ,

[0085] Generally, taking K1 = 0.01, K2 = 0.03, α = β = γ = 1, we can get:

[0086]

[0087] ​The structural similarity value ranges from 0 to 1, and the larger the SSIM value, the smaller the image distortion.

[0088] For the video image edge information restoration method based on deep learning described in the present invention, the reliability of the method has been proven on public datasets such as YouTube–vos and Davis.

[0089] Among them, for the edge restoration result obtained in step S52, it is verified on the YouTube–vos validation set. The peak signal-to-noise ratio of the restored edge image compared with the original image is increased by about 11.2% compared with that before restoration, and the structural similarity is increased by about 1.24% from 0.9861 to 0.9983. Figure 8 From top to bottom, the original image, the damaged image, and the edge images before and after restoration (the red part is the restored edge) are shown in sequence.

Claims

1. A method for restoring edge information of a video image based on deep learning, the method comprising the following steps: S1. Preprocessing of the video image data to be repaired; (1) Obtaining the damaged area mask of the video image frames, where masks is a binary image (0 represents a normal area and 1 represents a damaged area); (2) Calculating the edge information edges of the video frame frames by using the canny operator; (3) Obtaining masked_frames and masked_edges by using frames×(1-masks) and edges×(1-masks); S2. Construct a dataset according to step (2) in step S1; perform data preprocessing on the train, test, and val video clips in the YouTube-vos and Davis datasets according to step (2) in step S1 to obtain the edges required for training; S3. Establish a VEC model that uses Encoder and Decoder to encode and decode images and edge information, and realizes feature interaction and propagation based on spatiotemporal decoupling Transformer; S4, the data set processed in step S2 is divided into training set, validation set and test set according to YouTube-vos and Davis data sets, and the masked_frames, masked_edges and masks obtained in step S1 are sent to the VEC model in step S3 for training; S5. Select video clips that are not used for model training, that is, video clips that are not in the training set and validation set in step S4; perform data preprocessing through step S1, use them as VEC model input for model reasoning, and complete video image edge information restoration.

2. According to the method for repairing edge information of video images based on deep learning in claim 1, step S1 specifically comprises: S11, obtaining damaged area masks of video image frames; using the public dataset QD-IMD: Quick Draw Irregular Mask Dataset, or generating random polygonal masks through algorithms as masks; S12. Read the video image frames through OpenCV-Python and convert them into grayscale images. Use the cv2.Canny function to set a suitable threshold to process the frames and obtain the ground truth edges. S13. Obtain masked_frames and masked_edges using formula (3) in step S1.

3. According to the method for repairing edge information of video images based on deep learning in claim 1, step S3 is specifically: S31,VEC network structure includes three modules: encoder module, spatiotemporal decoupling attention module, and decoder module; S32, use pytorch framework to build VEC network structure; S33. Set appropriate neural network training parameters.

4. According to the method for repairing edge information of video images based on deep learning in claim 1, step S31 is specifically: S311, data preprocessing; the damaged video image masked_frames and the damaged edge image masked_edges obtained in step S1 (3) are concatenated with the mask masks in the feature channel dimension through the torch.cat function in the pytorch framework, for details, refer to FIG1 of the specification; S312, encoder module, i.e., feature extraction module of VEC network that converts image from spatial domain to feature domain; it is composed of Edge encoder and Frame encoder, and is constructed with reference to Figures 2 and 3 of the attached drawings of the specification for details; S313, the Edge encoder and the Frame encoder in the encoder module respectively process the Input edges and Input frames obtained in step S311 to obtain outputs x0 and x1; S314, spatiotemporal decoupling attention module, i.e., the propagation module of the VEC network for temporal and spatial information interaction respectively; it is composed of Edge patch2vec, Frame patch2vec, Pos embedding, temporal attention module TTransformer, spatial attention module S Transformer and Transformer blocks; wherein Edgepatch2vec and Frame patch2vec are two independent Conv2d networks, the position encoding Pos embedding is constructed in detail with reference to Figure 4 of the attached figure of the specification, TTransformer and S Transformer are constructed with reference to the FGT paper, and Transformer blocks are multiple stacked TTransformers and S Transformers; S315, Edge patch2vec, and Frame patch2vec process the outputs x0 and x1 obtained in step S313 respectively, and convert them from the common patch data format of the convolutional neural network to the token form commonly used in the attention network to obtain outputs x2 and x3; S316, x2 obtained in step S315 is passed through T Transformer and Pos embedding in turn to obtain outputs x4 and x5 respectively; x5 and x3 obtained in step S315 are input into S Transformer together to obtain output x6; x6 and x3 are used as inputs and passed through Transformer blocks to obtain output x7; S317, decoder module, that is, the VEC network decodes the feature domain information and restores it to the spatial domain to obtain the repair result; it is composed of Vec2patch and Decoder, and is constructed in detail with reference to Figures 5 and 6 of the attached drawings of the specification; S318. The output x7 obtained in step S316 is converted from the common token form of the attention network to the common format patch of the convolutional neural network through Vec2patch and the output x8 is obtained; x8 is added to the output x0 obtained in step S313 at the pixel level to obtain x9, and x9 is then upsampled by the Decoder to obtain the result x10.

5. According to the method for repairing edge information of video images based on deep learning in claim 1, step S4 is specifically: S41, for each video clip of YouTube-vos and Davis datasets, obtain the ground truth edges of the video image according to step S2, i.e., the method of step S12; S42, select video frames from each video clip and obtain masked_frames, masked_edges and masks through steps S11 and S13, and send them to the VEC model in step S3 for training.

6. According to the method for restoring edge information of video images based on deep learning in claim 1, step S5 is specifically: S51, select the video clip f0 that is not used for model training, obtain its masked_frames, masked_edges and masks through steps S11 and S13 as VEC model input for model inference, and obtain the output x10 of step S318; S52. Obtain the final repair result x11 through f0×(1-mask)+x10×mask.

Images