Video repair method and apparatus

Abstract

The present disclosure relates to a video repairing method and device. The video repairing method comprises: obtaining a video to be repaired; obtaining deformation information between each image frame of the video to be repaired based on image features of each image frame of the video to be repaired; performing deformation processing on each image frame of the video to be repaired based on the deformation information; and obtaining a repaired video based on each image frame after deformation processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of video processing, and in particular, to a video inpainting method and device. BACKGROUND

[0002] Video inpainting is to inpaint the unknown (damaged) region according to the information of the known region of the video, to obtain the reconstructed region, so that the video after inpainting is visually coherent and natural, that is, when the part of pixels that needs to be inpainted is deleted and the spatial and temporal holes are filled with reasonable content in the video, a reasonable background is reconstructed. Video inpainting can help many video editing and restoration tasks, such as removing unwanted objects, scratch or damage recovery, and repositioning, and more importantly, video inpainting can also be used in combination with augmented reality (AR) to provide a better visual experience.

[0003] At present, the commonly used video inpainting method is the STTN method, which is based on a space-time joint Transformer for video inpainting. The Transformer is a model proposed according to the self-attention mechanism to solve the Seq2Seq problem. In order to capture as much context information as possible, the STTN method uses different sizes of image blocks for cutting on different heads, so that when the features of the missing region are not rich enough, the attention mechanism based on large image blocks can effectively utilize more known information; when the features of the missing region are rich, the attention mechanism based on small image blocks helps the model to focus on more subtle changes. However, for videos with large changes between image frames, using the STTN method to inpaint the video will cause the inpainted video to have a distorted phenomenon. SUMMARY

[0004] The present disclosure provides a video inpainting method and device to at least solve the problem that the video inpainting method in the related art cannot be applied to videos with large changes between image frames.

[0005] According to a first aspect of an embodiment of the present disclosure, a video inpainting method is provided, comprising: obtaining a to-be-inpainted video; obtaining deformation information between each image frame of the to-be-inpainted video based on image features of each image frame of the to-be-inpainted video; performing deformation processing on each image frame of the to-be-inpainted video based on the deformation information; and obtaining an inpainted video based on each image frame after the deformation processing.

[0006] Optionally, obtaining the deformation information between each image frame of the to-be-inpainted video based on the image features of each image frame of the to-be-inpainted video comprises: obtaining similarity information based on sub-image features of each image block in each image frame, wherein the image block is one of a predetermined number of image blocks obtained by dividing each image frame; and obtaining the deformation information based on the similarity information.

[0007] Optionally, the similarity information is obtained based on the sub-image features of each image block in each image frame, including: obtaining image features of each image frame of the video to be repaired; obtaining a Query matrix, a Key matrix and a Value matrix based on the image features of each image frame; and obtaining a similarity matrix as the similarity information based on the similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix.

[0008] Optionally, the similarity matrix is obtained based on the similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix, including: obtaining a mask of each image block corresponding to the Query matrix and a mask of each image block corresponding to the Key matrix; in a case where the mask of the image block in the Query matrix and the mask of the image block in the Key matrix are both 1, taking a product of the Query sub-matrix of the image block in the Query matrix and the Key sub-matrix of the image block in the Key matrix as the similarity between the two image blocks; in a case where either of the mask of the image block in the Query matrix and the mask of the image block in the Key matrix is 0, taking 0 as the similarity between the two image blocks; and obtaining the similarity matrix by merging after normalization of the similarity.

[0009] Optionally, the deformation information is obtained based on the similarity information, including: obtaining a deformation parameter matrix as the deformation information based on the similarity matrix.

[0010] Optionally, the deformation processing is performed on each image frame of the video to be repaired based on the deformation information, including: performing deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix.

[0011] Optionally, the deformation processing is performed on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix, including: for each image block, respectively deforming a Key sub-matrix of the image block in the Key matrix and a Value sub-matrix of the image block in the Value matrix by a deformation parameter sub-matrix of the image block in the deformation parameter matrix to obtain a deformed Key sub-matrix and a deformed Value sub-matrix of the image block; splicing the deformed Key sub-matrices to obtain the deformed Key matrix; and splicing the deformed Value sub-matrices to obtain the deformed Value matrix.

[0012] Optionally, based on each image frame after deformation processing, the repaired video is obtained, including: based on the Query matrix, the deformed Key matrix and the deformed Value matrix, the repaired feature matrix of each image frame is obtained; based on the repaired feature matrix, the repaired video is obtained.

[0013] Optionally, based on the Query matrix, the deformed Key matrix and the deformed Value matrix, the repaired feature matrix of each image frame is obtained, including: based on the Query matrix, the deformed Key matrix, the deformed Value matrix and the deformation parameter matrix, the repaired feature matrix of each image frame is obtained.

[0014] Optionally, the video repair method is executed through a pre-trained video repair model, and the video repair model includes an encoder, a deformation parameter prediction network, a repair network and a decoder; wherein the image features of each image frame of the video to be repaired are obtained, including: inputting the video to be repaired into the encoder to obtain the feature matrix of each image frame of the video to be repaired as the image features; wherein the deformation parameter matrix is obtained based on the similarity matrix, including: inputting the similarity matrix into the deformation parameter prediction network to obtain the deformation parameter matrix; wherein the repaired feature matrix of each image frame is obtained based on the Query matrix, the deformed Key matrix and the deformed Value matrix, including: inputting the Query matrix, the deformed Key matrix and the deformed Value matrix into the repair network to obtain the repaired feature matrix of each image frame; wherein the repaired video is obtained based on the repaired feature matrix, including: inputting the repaired feature matrix of each image frame into the decoder to obtain the repaired video.

[0015] Optionally, the video inpainting model is trained by: obtaining a training sample set, wherein the training sample set includes a plurality of training videos and a clear video corresponding to each training video; inputting the training video into an encoder to obtain a feature matrix of each image frame of the training video; obtaining a Query matrix, a Key matrix and a Value matrix based on the feature matrix of each image frame; obtaining a similarity matrix based on a similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix, wherein the image block is one of a predetermined number of image blocks obtained by respectively cutting each image frame; inputting the similarity matrix into a deformation parameter prediction network to obtain a deformation parameter matrix; performing deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix; inputting the Query matrix, the deformed Key matrix and the deformed Value matrix into an inpainting network to obtain an estimated deformed feature matrix of each image frame; inputting the estimated deformed feature matrix into a decoder to obtain an estimated deformed video; and adjusting parameters of the encoder, the deformation parameter prediction network, the inpainting network and the decoder based on the estimated deformed video and the clear video corresponding to the training video.

[0016] According to a second aspect of the embodiments of the present disclosure, a video inpainting apparatus is provided, including: a video obtaining unit configured to obtain a video to be inpainted; a deformation information obtaining unit configured to obtain deformation information between each image frame of the video to be inpainted based on image features of each image frame of the video to be inpainted; a deformation processing unit configured to perform deformation processing on each image frame of the video to be inpainted based on the deformation information; and an inpainting unit configured to obtain an inpainted video based on each image frame after the deformation processing.

[0017] Optionally, the deformation information obtaining unit is further configured to obtain similarity information based on sub-image features of each image block in each image frame, wherein the image block is one of a predetermined number of image blocks obtained by respectively cutting each image frame; and obtain the deformation information based on the similarity information.

[0018] Optionally, the deformation information obtaining unit is further configured to obtain image features of each image frame of the video to be inpainted; obtain a Query matrix, a Key matrix and a Value matrix based on the image features of each image frame; obtain a similarity matrix based on a similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix as the similarity information.

[0019] Optionally, the deformation information obtaining unit is further configured to obtain a mask of each image block corresponding to the Query matrix and a mask of each image block corresponding to the Key matrix; in a case that the mask of the image block in the Query matrix and the mask of the image block in the Key matrix are both 1, taking a product of a Query sub-matrix of the image block in the Query matrix and a Key sub-matrix of the image block in the Key matrix as the similarity of the two image blocks; in a case that any one of the mask of the image block in the Query matrix and the mask of the image block in the Key matrix is 0, taking 0 as the similarity of the two image blocks; performing merging after normalization processing of the similarity to obtain a similarity matrix.

[0020] Optionally, the deformation information obtaining unit is further configured to obtain a deformation parameter matrix based on the similarity matrix as the deformation information.

[0021] Optionally, the deformation processing unit is further configured to perform deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix.

[0022] Optionally, the deformation processing unit is further configured to, for each image block, deform a Key sub-matrix of the image block in the Key matrix and a Value sub-matrix of the image block in the Value matrix respectively by a deformation parameter sub-matrix of the image block in the deformation parameter matrix to obtain a deformed Key sub-matrix and a deformed Value sub-matrix of the image block; splice the deformed Key sub-matrices to obtain the deformed Key matrix; and splice the deformed Value sub-matrices to obtain the deformed Value matrix.

[0023] Optionally, the repairing unit is further configured to obtain a repaired feature matrix of each image frame based on the Query matrix, the deformed Key matrix and the deformed Value matrix; and obtain a repaired video based on the repaired feature matrix.

[0024] Optionally, the repairing unit is further configured to obtain a repaired feature matrix of each image frame based on the Query matrix, the deformed Key matrix, the deformed Value matrix and the deformation parameter matrix.

[0025] Optionally, the video repairing method is performed by a pre-trained video repairing model, the video repairing model comprising an encoder, a deformation parameter prediction network, a repairing network and a decoder; wherein the deformation information acquisition unit is further configured to input the video to be repaired into the encoder to obtain a feature matrix of each image frame of the video to be repaired as image features; the deformation information acquisition unit is further configured to input the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix; wherein the repairing unit is further configured to input the Query matrix, the deformed Key matrix and the deformed Value matrix into the repairing network to obtain a repaired feature matrix of each image frame; and the repairing unit is further configured to input the repaired feature matrix of each image frame into the decoder to obtain a repaired video.

[0026] Optionally, the video repairing model is trained by: obtaining a training sample set, wherein the training sample set comprises a plurality of training videos and a clear video corresponding to each training video; inputting the training video into the encoder to obtain a feature matrix of each image frame of the training video; obtaining a Query matrix, a Key matrix and a Value matrix based on the feature matrix of each image frame; obtaining a similarity matrix based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, wherein the image block is one of a predetermined number of image blocks obtained by respectively cutting each image frame; inputting the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix; performing deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix; inputting the Query matrix, the deformed Key matrix and the deformed Value matrix into the repairing network to obtain an estimated repaired feature matrix of each image frame; inputting the estimated repaired feature matrix into the decoder to obtain an estimated repaired video; and adjusting the parameters of the encoder, the deformation parameter prediction network, the repairing network and the decoder based on the estimated repaired video and the clear video corresponding to the training video.

[0027] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the video repairing method according to the present disclosure.

[0028] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is caused to perform the video repairing method according to the present disclosure.

[0029] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer instructions which, when executed by a processor, implement the video inpainting method according to the present disclosure.

[0030] The technical solutions provided by the embodiments of the present disclosure at least have the following beneficial effects:

[0031] According to the video inpainting method and device, the deformation parameter prediction network is introduced to predict the deformation information between different image frames, and the non-linear deformation between different image frames can be adjusted based on the deformation information before inpainting the video, so that the video can be inpainted based on the image frames after adjusting the deformation, and the motion video with non-linear deformation can be better inpainted, and the phenomenon of image distortion of the inpainted video is avoided. Therefore, the present disclosure solves the problem that the video inpainting method in the related art cannot be applied to the video with large variation between image frames.

[0032] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings incorporated in the specification hereof and forming a part thereof illustrate embodiments consistent with the present disclosure and together with the description serve to explain the principles of the present disclosure, and are not intended to limit the present disclosure.

[0034] Figure 1 is a schematic diagram of an implementation scenario of a video inpainting method according to an exemplary embodiment of the present disclosure;

[0035] Figure 2 is a flowchart of a video inpainting method according to an exemplary embodiment;

[0036] Figure 3 is a structural schematic diagram to which a video inpainting method according to an exemplary embodiment is applied;

[0037] Figure 4 is a structural schematic diagram of a homography estimator according to an exemplary embodiment;

[0038] Figure 5 is a structural schematic diagram of a deformation parameter prediction network according to an exemplary embodiment;

[0039] Figure 6 is a flowchart of a training method of a video inpainting model according to an exemplary embodiment;

[0040] Figure 7 is a verification result schematic diagram according to an exemplary embodiment Figure 1 ;

[0041] Figure 8 is a verification result diagram shown according to an exemplary embodiment Figure 2 ;

[0042] Figure 9 is a verification result diagram shown according to an exemplary embodiment Figure 3 ;

[0043] Figure 10 is a verification result diagram shown according to an exemplary embodiment Figure 4 ;

[0044] Figure 11 is a verification result diagram shown according to an exemplary embodiment Figure 5 ;

[0045] Figure 12 is a block diagram of a video repair device according to an exemplary embodiment;

[0046] Figure 13 is a block diagram of a training device of a video repair model according to an exemplary embodiment;

[0047] Figure 14 is a block diagram of an electronic device 1400 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0049] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or a chronological sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0050] It should be noted that "at least one of a plurality" appearing in the present disclosure represents three types of parallel cases including "any one of the plurality", "a combination of any multiple of the plurality", and "all of the plurality". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step one and step two" represents the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing step one and step two.

[0051] To solve the above problems, the present disclosure provides a video repair model method, which can be applied to videos with relatively large variation between image frames. The following takes a video of an athlete running outdoors as an example for illustration.

[0052] Figure 1 is a schematic diagram of an implementation scenario of a video repair method according to an example embodiment of the present disclosure, as Figure 1 The implementation scenario includes a server 100, a user terminal 110, and a user terminal 120. The user terminals are not limited to two, and include but are not limited to mobile phones, personal computers, and other devices. The user terminals can install an application for video repair. The server can be one server, a server cluster composed of several servers, a cloud computing platform, or a virtualization center.

[0053] After the user terminal 110 or the user terminal 120 obtains a video to be repaired (a video of an athlete running outdoors), the user terminal 110 or the user terminal 120 sends the video to be repaired to the server 100. The server 100 inputs the video to be repaired to an encoder, obtains a feature matrix of each image frame of the video to be repaired, and obtains a Query matrix, a Key matrix, and a Value matrix based on the feature matrix of each image frame. Then, the server 100 obtains a similarity matrix based on the similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix, where the image block is one of a predetermined number of image blocks obtained by dividing each image frame. After obtaining the similarity matrix, the server 100 inputs the similarity matrix to a deformation parameter prediction network, obtains a deformation parameter matrix, performs deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix, obtains a deformed Key matrix and a deformed Value matrix, and then inputs the Query matrix, the deformed Key matrix, and the deformed Value matrix to a repair network to obtain a repaired feature matrix of each image frame. Finally, the server 100 inputs the repaired feature matrix to a decoder to obtain a repaired video. Although the video to be repaired has relatively large variation between image frames, the repaired video obtained by the video repair method of the present disclosure does not appear distorted.

[0054] Hereinafter, a video repairing method and device according to an example embodiment of the present disclosure will be described with reference to the accompanying drawings. Figures 2 to 13 A video repairing method and device according to an example embodiment of the present disclosure will be described in detail.

[0055] Figure 2 is a flowchart of a video repairing method according to an example embodiment, as shown in Figure 2 The video repairing method includes the following steps:

[0056] In step S201, a video to be repaired is obtained. The video to be repaired can be temporarily obtained through a terminal camera or can be a locally stored video, which is not limited by the present disclosure.

[0057] In step S202, based on the image features of each image frame of the video to be repaired, deformation information between each image frame of the video to be repaired is obtained. It should be noted that what is needed to be obtained here is the deformation information between each image frame two by two.

[0058] According to an example embodiment of the present disclosure, based on the image features of each image frame of the video to be repaired, the deformation information between each image frame of the video to be repaired is obtained, including: based on the sub-image features of each image block in each image frame, similarity information is obtained, wherein the image block is one of a predetermined number of image blocks obtained by respectively cutting each image frame; based on the similarity information, the deformation information is obtained. According to the present embodiment, the deformation information can be obtained conveniently and quickly.

[0059] According to an example embodiment of the present disclosure, based on the sub-image features of each image block in each image frame, the similarity information is obtained, including: obtaining the image features of each image frame of the video to be repaired; based on the image features of each image frame, a Query matrix, a Key matrix and a Value matrix are obtained; based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, a similarity matrix is obtained as the similarity information. A conventional video encoder can be used by the encoder, which is not limited by the present disclosure.

[0060] According to an example embodiment of the present disclosure, based on the image features of each image frame, the Query matrix, the Key matrix and the Value matrix can include: the image features of each image frame are respectively cut into a predetermined number of matrices, wherein the predetermined number of matrices correspond to the predetermined number of image blocks; all the cut matrices are spliced in the channel dimension to obtain a spliced matrix; the spliced matrix is respectively input to the corresponding convolution to obtain the Query matrix, the Key matrix and the Value matrix. Through the present embodiment, the three matrices can be obtained conveniently and quickly.

[0061] For example, the video to be repaired is composed of T image frames, each of which is a picture with a width of W and a height of H, Figure 3 is a structural schematic diagram to which a video repair method according to an example embodiment is applied, as shown in Figure 3 The Encoder encodes each image frame of the video to be repaired. After the tth image frame is encoded by the Encoder, a feature matrix f T is generated, where f T has a dimension of c x h x w, c is the number of channels, h corresponds to the height H of the picture, and w corresponds to the width W of the picture. Then, each image frame is divided into a predetermined number of image blocks, for example, each image frame is divided into N p image blocks, that is, f T is divided into N p sub-feature matrices. Figure 3 In the above embodiment, N p = 4, the width and height of each image block are w / 2 and h / 2 respectively, and the entire video to be repaired has T*N p image blocks in total. Then, the feature matrices of the T*N p image blocks are concatenated in the channel dimension to obtain the feature matrix F of the video to be repaired (i.e., the above-mentioned concatenated matrix). The concatenated matrix is input into corresponding convolutions to obtain Query matrices, Key matrices, and Value matrices. The corresponding convolutions are not limited by the present disclosure and can each be a 1x1 convolution.

[0062] According to an example embodiment of the present disclosure, based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, a similarity matrix is obtained, including: obtaining a mask corresponding to each image block in the Query matrix and a mask corresponding to each image block in the Key matrix; in the case that both the mask of the image block in the Query matrix and the mask of the image block in the Key matrix are 1, the product of the Query sub-matrix of the image block in the Query matrix and the Key sub-matrix of the image block in the Key matrix is taken as the similarity between the two image blocks; in the case that either the mask of the image block in the Query matrix or the mask of the image block in the Key matrix is 0, 0 is taken as the similarity between the two image blocks; after normalization, the similarities are merged to obtain a similarity matrix. Through this embodiment, the similarity matrix can be obtained conveniently and quickly.

[0063] For example, the similarity matrix in the above embodiment can be implemented by a DePtH in Figure 3 , which is a DePtH based on a Transformer. Figure 4 is a structural schematic diagram of a homography estimator according to an example embodiment, as shown inFigure 4 As shown, F is input into DePtH, and three 1x1 convolutions are performed to generate three matrices required for three self-attention operations, i.e., a Query matrix, a Key matrix, and a Value matrix. For example, a patch encoder deforms F to generate the Query matrix, the Key matrix, and the Value matrix, where the Query matrix, the Key matrix, and the Value matrix are determined by Q, K, V = M q (f i ), M k (f i ), M v (f i ), M is a 1x1 convolution. Then, the three matrices are input into a patch matching (abbreviated as PM) to obtain a similarity matrix. The specific operation of the PM is as follows: the similarity between each image block in the Query matrix and each image block in the Key matrix is calculated by multiplying the feature matrix of each image block in the Query matrix and the feature matrix of each image block in the Key matrix. For example, as shown in equation (1), C is the cosine similarity matrix calculated between f k (i) T and f q (j), where f k (i) T is the transpose of the channel-normalized feature matrix of each image block in the Key matrix, and f q (j) is the channel-normalized feature matrix of each image block in the Query matrix. Since the video to be modified has a total of T·N p image blocks, for the i-th image block in the Key matrix and the j-th image block in the Query matrix, only the part containing valid information in which neither of the two image blocks is occluded can be calculated. To this end, a down-sampled binary mask is used to filter the result, i.e., only when m k (i) and m q (j) are both 1, the similarity matrix of the two image blocks is calculated.

[0064]

[0065] According to an example embodiment of the present disclosure, the deformation information is obtained based on the similarity information, including: obtaining a deformation parameter matrix as the deformation information based on the similarity matrix.

[0066] For example, as Figure 4As shown, the output of the PM is input to a Deformed Transformer estimator, i.e., the deformed parameter prediction network described above, to obtain a predicted bias coefficient matrix θ (i.e., the deformed parameter matrix described above), which contains the deformed parameters between each image block in the Query matrix and each image block in the Key matrix. θ is a matrix with a dimension of T·Np×2×3, where for T·Np image blocks, each image block has a matrix θ' composed of 2×3 = 6 numbers as a transformation parameter sub-matrix for each image block.

[0067] According to an exemplary embodiment of the present disclosure, the deformed parameter prediction network includes a convolution layer, a pooling layer, and a fully connected layer. The similarity information is input to the deformed parameter prediction network to obtain the deformed information, including: inputting the similarity information to the convolution layer to obtain the information after convolution; inputting the information after convolution to the pooling layer to obtain the information after pooling; and inputting the information after pooling to the fully connected layer to obtain the deformed information. Through this embodiment, a simple structure of the deformed parameter prediction network is given, and the deformed parameter matrix can be obtained conveniently and quickly.

[0068] For example, Figure 5 is a structure diagram of a deformed parameter prediction network according to an exemplary embodiment, as Figure 5 As shown, the left and right of the network represent the input and output, respectively. The network can be composed of 3×3 convolution with a step of 2, where the number of channels of the middle layer is 128, and Average Pool represents the average pooling at all positions h×w. In addition, ReLU is used as a nonlinear activation function after all convolution layers except the last layer of the decoder.

[0069] In step S203, based on the deformed information, the deformed processing is performed on each image frame of the video to be repaired.

[0070] According to an exemplary embodiment of the present disclosure, based on the deformed information, the deformed processing is performed on each image frame of the video to be repaired, including: based on the deformed parameter matrix, performing the deformed processing on the Key matrix and the Value matrix to obtain the deformed Key matrix and the deformed Value matrix. The above θ acts on the Key matrix and the Value matrix respectively, and the Key matrix and the Value matrix can be aligned with reference to the Query matrix to generate the deformed Key matrix (Deformed K) and the deformed Value matrix (Deformed V) respectively, while the Query matrix remains unchanged.

[0071] According to an exemplary embodiment of this disclosure, deformation processing is performed on the Key matrix and Value matrix based on the deformation parameter matrix to obtain the deformed Key matrix and the deformed Value matrix. This includes: for each image block, deforming the Key submatrix of the image block in the Key matrix and the Value submatrix of the image block in the Value matrix using the deformation parameter submatrix of the image block in the deformation parameter matrix, respectively, to obtain the deformed Key submatrix and the deformed Value submatrix of the image block; concatenating the deformed Key submatrixes to obtain the deformed Key matrix; and concatenating the deformed Value submatrixes to obtain the deformed Value matrix. Through this embodiment, rotating on an image block basis allows for more accurate deformation results.

[0072] For example, such as Figure 4 The θ shown is a matrix of dimension T·Np×2×3. For T·Np image blocks, each image block has a matrix θ′ consisting of 2×3=6 numbers as the transformation parameter submatrix for each image block. The Key matrix and Value matrix are rotated and transformed according to the corresponding θ′ for each image block to obtain the transformed Key matrix (Deformed K, also known as the deformed Key matrix above) and the transformed Value matrix (Deformed V, also known as the deformed Value matrix above).

[0073] In step S204, the repaired video is obtained based on each image frame after deformation processing.

[0074] According to an exemplary embodiment of this disclosure, obtaining a repaired video based on each deformed image frame includes: obtaining a repaired feature matrix for each image frame based on a Query matrix, a deformed Key matrix, and a deformed Value matrix; and obtaining the repaired video based on the repaired feature matrix. For example, as... Figure 3 As shown, the network can be repaired using Transformer, but this disclosure does not limit this to any particular method.

[0075] According to an exemplary embodiment of this disclosure, the restored feature matrix of each image frame is obtained based on the Query matrix, the deformed Key matrix, and the deformed Value matrix, including: obtaining the restored feature matrix of each image frame based on the Query matrix, the deformed Key matrix, the deformed Value matrix, and the deformation parameter matrix. In this embodiment, by also inputting the deformation parameters into the restoration network, better restoration results can be obtained for the video.

[0076] For example, such as Figure 3As shown, θ' can also be input into the Transformer as a hyperparameter to assist the Transformer operation.

[0077] According to an exemplary embodiment of the present disclosure, based on the repaired feature matrix, the repaired video can be obtained as Figure 3 As shown, the output of the Transformer is T c x h x w matrices (i.e., the repaired feature matrix described above), which can be input into a decoder of an RNN structure for decoding, and the video obtained after decoding is the video repaired by the network for each image frame.

[0078] According to an exemplary embodiment of the present disclosure, the video repair method is executed by a pre-trained video repair model, and the video repair model includes an encoder, a deformation parameter prediction network, a repair network, and a decoder. Wherein, the image features of each image frame of the video to be repaired are obtained, including: inputting the video to be repaired into the encoder to obtain the feature matrix of each image frame of the video to be repaired as the image features; wherein, based on the similarity matrix, the deformation parameter matrix is obtained, including: inputting the similarity matrix into the deformation parameter prediction network to obtain the deformation parameter matrix; wherein, based on the Query matrix, the deformed Key matrix and the deformed Value matrix, the repaired feature matrix of each image frame is obtained, including: inputting the Query matrix, the deformed Key matrix and the deformed Value matrix into the repair network to obtain the repaired feature matrix of each image frame; wherein, based on the repaired feature matrix, the repaired video is obtained, including: inputting the repaired feature matrix of each image frame into the decoder to obtain the repaired video.

[0079] According to an exemplary embodiment of this disclosure, the video restoration model is trained by the following operations: obtaining a training sample set, wherein the training sample set includes multiple training videos and a clear video corresponding to each training video; inputting the training videos into an encoder to obtain a feature matrix for each image frame of the training videos; obtaining a Query matrix, a Key matrix, and a Value matrix based on the feature matrix of each image frame; and obtaining a similarity matrix based on the similarity between the Query submatrix of each image block in the Query matrix and the Key submatrix of each image block in the Key matrix, wherein an image block is one of a predetermined number of image blocks obtained by segmenting each image frame. Image patches; the similarity matrix is ​​input into the deformation parameter prediction network to obtain the deformation parameter matrix; based on the deformation parameter matrix, deformation processing is performed on the key matrix and value matrix to obtain the deformed key matrix and deformed value matrix; the query matrix, deformed key matrix, and deformed value matrix are input into the restoration network to obtain the estimated restored feature matrix for each image frame; the estimated restored feature matrix is ​​input into the decoder to obtain the estimated restored video; based on the estimated restored video and the clear video corresponding to the training video, the parameters of the encoder, deformation parameter prediction network, restoration network, and decoder are adjusted.

[0080] In summary, as Figure 3 The deformation homography estimator shown generates a deformation parameter matrix θ, which is applied to the Query and Value matrices to align the Key and Value submatrices of each image patch with the corresponding submatrices of the Query matrix. The Query matrix, the Key matrix aligned with the Query matrix, and the Value matrix aligned with the Query matrix are then fed into the Transformer as input for Transformer operations. Furthermore, the deformation parameter θ can also be used as a hyperparameter, inputting into the Transformer to assist in its operation.

[0081] To solve the problem of complex rotation transformation between image frames in a video in a complex scene, the disclosure improves the performance and efficiency of the Transformer by introducing a deformed patch in the attention operation, that is, the matrix input into the Transformer is a deformed matrix, so that the attention and alignment of the image block can be better, specifically, the Query matrix, the Key matrix and the Value matrix are used to learn the offset between image frames, so that the Key matrix and the Query matrix are better spatio-temporal sampled without additional supervision. Therefore, even in challenging scenes with rotation transformation or deformation due to rapid movement, the method of the disclosure is applicable. In addition, the deformed Transformer estimator of the disclosure can be separated from the entire network framework as a separate module and can be used to predict the deformation parameter matrix θ for any two images.

[0082] The disclosure also provides a training method of a video restoration model, Figure 6 is a flowchart of a training method of a video restoration model according to an exemplary embodiment, as Figure 6 shown, the video restoration model includes an encoder, a deformation parameter prediction network, a restoration network and a decoder, and the training method includes:

[0083] In step S601, a training sample set is obtained, wherein the training sample set includes a plurality of training videos and a clear video corresponding to each training video.

[0084] In step S602, the training video is input into the encoder to obtain a feature matrix of each image frame of the training video.

[0085] In step S603, based on the feature matrix of each image frame, a Query matrix, a Key matrix and a Value matrix are obtained.

[0086] In step S604, based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, a similarity matrix is obtained, wherein the image block is one of a predetermined number of image blocks obtained by cutting each image frame.

[0087] In step S605, the similarity matrix is input into the deformation parameter prediction network to obtain a deformation parameter matrix.

[0088] In step S606, based on the deformation parameter matrix, deformation processing is performed on the Key matrix and the Value matrix to obtain a deformed Key matrix and a deformed Value matrix.

[0089] In step S607, the Query matrix, the warped Key matrix and the warped Value matrix are input into the inpainting network to obtain an estimated inpainted feature matrix of each image frame.

[0090] In step S608, the estimated inpainted feature matrix is input into the decoder to obtain an estimated inpainted video.

[0091] In step S609, based on the estimated inpainted video and the clear video corresponding to the training video, the parameters of the encoder, the deformation parameter prediction network, the inpainting network and the decoder are adjusted, and the video inpainting model is trained.

[0092] According to an example embodiment of the present disclosure, based on the deformation parameter matrix, the deformation processing is performed on the Key matrix and the Value matrix to obtain the warped Key matrix and the warped Value matrix, including: for each image block, the Key sub-matrix of the image block in the Key matrix and the Value sub-matrix of the image block in the Value matrix are respectively warped by the deformation parameter sub-matrix of the image block in the deformation parameter matrix to obtain the warped Key sub-matrix and the warped Value sub-matrix of the image block; the warped Key sub-matrices are spliced to obtain the warped Key matrix; and the warped Value sub-matrices are spliced to obtain the warped Value matrix.

[0093] According to an example embodiment of the present disclosure, based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, a similarity matrix is obtained, including: obtaining the mask of the sub-matrix of each image block in the Query matrix and the mask of the sub-matrix of each image block in the Key matrix; in the case that the mask of the sub-matrix of the image block in the Query matrix and the mask of the sub-matrix of the image block in the Key matrix are both 1, the product of the Query sub-matrix of the image block in the Query matrix and the Key sub-matrix of the image block in the Key matrix is taken as the similarity of the two image blocks; in the case that either the mask of the sub-matrix of the image block in the Query matrix or the mask of the sub-matrix of the image block in the Key matrix is 0, 0 is taken as the similarity of the two image blocks; and after normalization processing, the similarities are merged to obtain the similarity matrix.

[0094] According to an example embodiment of the present disclosure, the Query matrix, the warped Key matrix and the warped Value matrix are input into the inpainting network to obtain an estimated inpainted feature matrix of each image frame, including: the Query matrix, the warped Key matrix, the warped Value matrix and the deformation parameter matrix are input into the inpainting network to obtain an estimated inpainted feature matrix of each image frame.

[0095] According to an example embodiment of the present disclosure, the deformation parameter prediction network comprises a convolution layer, a pooling layer and a full connection layer, inputting the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix, comprising: inputting the similarity matrix into the convolution layer to obtain a matrix after convolution; inputting the matrix after convolution into the pooling layer to obtain a matrix after pooling; inputting the matrix after pooling into the full connection layer to obtain the deformation parameter matrix.

[0096] To prove the feasibility of the method of the present disclosure, the corresponding verification results are provided below.

[0097] Figure 7 is a verification result diagram according to an example embodiment Figure 1 As shown in Figure 7 , the method of the present disclosure outperforms existing video inpainting methods such as VINet, Deep-Flow, CPT and STTN on the public academic datasets YouTube VOS and DAVIS.

[0098] Figure 8 is a verification result diagram according to an example embodiment Figure 2 , Figure 9 is a verification result diagram according to an example embodiment Figure 3 , Figure 8 shows the qualitative results of the method of the present disclosure compared with other methods on the public academic datasets, Figure 9 shows the quantitative contribution of each sub-module in the model to the entire model. The last row is the PSNR and SSIM evaluation indicators of the entire model on the public academic datasets YouTube VOS and DAVIS. The higher the two evaluation indicators, the better the performance. The first row w / o DePtH is the performance of the model without the DePtH module. It can be seen that DePtH brings a 1.31 PSNR performance improvement and a 0.005 SSIM performance improvement to the model on the YouTube dataset. On the DAVIS dataset, it brings a 0.7 PSNR performance improvement and a 0.014 SSIM performance improvement to the model.

[0099] Figure 10 is a verification result diagram according to an example embodiment Figure 4 As shown in Figure 10 , the attention weights are sorted from high to low in all T image frames, and the attention visualization of the 6 frames with the highest attention weight after the input target frame (the second frame is the target frame in the left figure, and the 33rd frame is the target frame in the right figure) is given. It can be seen that the model in the present disclosure can well select the frames around the target frame for searching, and the found image blocks are all related regions around the filling area, so that the occluded area can be correctly filled.

[0100] Figure 11 is a verification result diagram shown according to an exemplary embodiment Figure 5 As shown in Figure 11 , the first column is an input image, the second column is a result without adding DePtH, and the last column is a result with adding DePtH. It can be seen that when DePtH is not added, the repaired image is relatively blurred, and the line is relatively curved. After adding DePtH, the repair result is obviously much better.

[0101] Figure 12 is a block diagram of a video repair device according to an exemplary embodiment. Referring to Figure 12 , the video repair device includes a video acquisition unit 1202, a deformation information acquisition unit 1204, a deformation processing unit 1206, and a repair unit 1208.

[0102] The video acquisition unit 1202 is configured to acquire a to-be-repaired video; the deformation information acquisition unit 1204 is configured to obtain deformation information between each image frame of the to-be-repaired video based on image features of each image frame of the to-be-repaired video; the deformation processing unit 1206 is configured to perform deformation processing on each image frame of the to-be-repaired video based on the deformation information; and the repair unit 1208 is configured to obtain a repaired video based on each image frame after deformation processing.

[0103] According to an embodiment of the present disclosure, the deformation information acquisition unit 1204 is further configured to obtain similarity information based on sub-image features of each image block in each image frame, wherein the image block is one of a predetermined number of image blocks respectively cut from each image frame; and obtain the deformation information based on the similarity information.

[0104] According to an embodiment of the present disclosure, the deformation information acquisition unit 1204 is further configured to obtain image features of each image frame of the to-be-repaired video; obtain a Query matrix, a Key matrix, and a Value matrix based on the image features of each image frame; and obtain a similarity matrix as the similarity information based on similarity of a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix.

[0105] According to an embodiment of the present disclosure, the deformation information acquisition unit 1204 is further configured to acquire a mask of each image block corresponding to the Query matrix and a mask of each image block corresponding to the Key matrix; in a case where the mask of the image block in the Query matrix and the mask of the image block in the Key matrix are both 1, taking a product of a Query sub-matrix of the image block in the Query matrix and a Key sub-matrix of the image block in the Key matrix as a similarity of the two image blocks; in a case where any one of the mask of the image block in the Query matrix and the mask of the image block in the Key matrix is 0, taking 0 as the similarity of the two image blocks; and performing merging after normalization processing of the similarity to obtain a similarity matrix.

[0106] According to an embodiment of the present disclosure, the deformation information acquisition unit 1204 is further configured to obtain a deformation parameter matrix as the deformation information based on the similarity matrix.

[0107] According to an embodiment of the present disclosure, the deformation processing unit 1206 is further configured to perform deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix.

[0108] According to an embodiment of the present disclosure, the deformation processing unit 1206 is further configured to, for each image block, deform a Key sub-matrix of the image block in the Key matrix and a Value sub-matrix of the image block in the Value matrix respectively by a deformation parameter sub-matrix of the image block in the deformation parameter matrix to obtain a deformed Key sub-matrix and a deformed Value sub-matrix of the image block; and splice the deformed Key sub-matrices to obtain the deformed Key matrix and splice the deformed Value sub-matrices to obtain the deformed Value matrix.

[0109] According to an embodiment of the present disclosure, the repairing unit 1208 is further configured to obtain a repaired feature matrix of each image frame based on the Query matrix, the deformed Key matrix and the deformed Value matrix; and obtain a repaired video based on the repaired feature matrix.

[0110] According to an embodiment of the present disclosure, the repairing unit 1208 is further configured to obtain a repaired feature matrix of each image frame based on the Query matrix, the deformed Key matrix, the deformed Value matrix and the deformation parameter matrix.

[0111] According to an embodiment of the present disclosure, the deformation parameter prediction network comprises a convolution layer, a pooling layer and a full connection layer, and the deformation information acquisition unit 1204 is further configured to input the similarity information into the convolution layer to obtain convolution information; input the convolution information into the pooling layer to obtain pooling information; and input the pooling information into the full connection layer to obtain the deformation information.

[0112] According to an embodiment of the present disclosure, the video repairing method is performed by a pre-trained video repairing model, the video repairing model comprising an encoder, a deformation parameter prediction network, a repairing network and a decoder; wherein the deformation information acquisition unit is further configured to input the video to be repaired into the encoder to obtain a feature matrix of each image frame of the video to be repaired as image features; the deformation information acquisition unit is further configured to input the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix; wherein the repairing unit is further configured to input the Query matrix, the deformed Key matrix and the deformed Value matrix into the repairing network to obtain a repaired feature matrix of each image frame; the repairing unit is further configured to input the repaired feature matrix of each image frame into the decoder to obtain a repaired video.

[0113] According to an embodiment of the present disclosure, the video repairing model is trained by the following operations: obtaining a training sample set, wherein the training sample set comprises a plurality of training videos and a clear video corresponding to each training video; inputting the training video into the encoder to obtain a feature matrix of each image frame of the training video; based on the feature matrix of each image frame, obtaining a Query matrix, a Key matrix and a Value matrix; based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, obtaining a similarity matrix, wherein the image block is one of a predetermined number of image blocks obtained by cutting each image frame; inputting the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix; based on the deformation parameter matrix, performing deformation processing on the Key matrix and the Value matrix to obtain a deformed Key matrix and a deformed Value matrix; inputting the Query matrix, the deformed Key matrix and the deformed Value matrix into the repairing network to obtain an estimated repaired feature matrix of each image frame; inputting the estimated repaired feature matrix into the decoder to obtain an estimated repaired video; based on the estimated repaired video and the clear video corresponding to the training video, adjusting the parameters of the encoder, the deformation parameter prediction network, the repairing network and the decoder.

[0114] Figure 13 is a block diagram of a training device of a video repairing model according to an exemplary embodiment. Referring to Figure 13 , the video repairing model comprises an encoder, a deformation parameter prediction network, a repairing network and a decoder, and the training device comprises a sample acquisition unit 1301, an encoding unit 1302, a matrix acquisition unit 1303, a similarity acquisition unit 1304, a deformation parameter acquisition unit 1305, a deformation processing unit 1306, a repairing unit 1307, a decoding unit 1308 and a training unit 1309.

[0115] The sample obtaining unit 1301 is configured to obtain a training sample set, wherein the training sample set includes a plurality of training videos and a clear video corresponding to each training video; the encoding unit 1302 is configured to input the training video into an encoder to obtain a feature matrix of each image frame of the training video; the matrix obtaining unit 1303 is configured to obtain a Query matrix, a Key matrix and a Value matrix based on the feature matrix of each image frame; the similarity obtaining unit 1304 is configured to obtain a similarity matrix based on the similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix, wherein the image block is one of a predetermined number of image blocks obtained by respectively cutting each image frame; the deformation parameter obtaining unit 1305 is configured to input the similarity matrix into a deformation parameter prediction network to obtain a deformation parameter matrix; the deformation processing unit 1306 is configured to perform deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix; the repairing unit 1307 is configured to input the Query matrix, the deformed Key matrix and the deformed Value matrix into a repairing network to obtain an estimated repaired feature matrix of each image frame; the decoding unit 1308 is configured to input the estimated repaired feature matrix into a decoder to obtain an estimated repaired video; and the training unit 1309 is configured to adjust the parameters of the encoder, the deformation parameter prediction network, the repairing network and the decoder based on the estimated repaired video and the clear video corresponding to the training video, and train the video repairing model.

[0116] According to an embodiment of the present disclosure, the deformation processing unit 1306 is further configured to, for each image block, deform the Key sub-matrix of the image block in the Key matrix and the Value sub-matrix of the image block in the Value matrix by a deformation parameter sub-matrix of the image block in the deformation parameter matrix to obtain a deformed Key sub-matrix and a deformed Value sub-matrix of the image block; and splice the deformed Key sub-matrices to obtain the deformed Key matrix, and splice the deformed Value sub-matrices to obtain the deformed Value matrix.

[0117] According to an embodiment of the present disclosure, the similarity obtaining unit 1304 is further configured to obtain a mask of the sub-matrix of each image block in the Query matrix and a mask of the sub-matrix of each image block in the Key matrix; in a case where the mask of the sub-matrix of the image block in the Query matrix and the mask of the sub-matrix of the image block in the Key matrix are both 1, taking a product of the Query sub-matrix of the image block in the Query matrix and the Key sub-matrix of the image block in the Key matrix as the similarity of the two image blocks; in a case where any one of the mask of the sub-matrix of the image block in the Query matrix and the mask of the sub-matrix of the image block in the Key matrix is 0, taking 0 as the similarity of the two image blocks; and performing merging after normalization processing of the similarity to obtain a similarity matrix.

[0118] According to an embodiment of the present disclosure, the repairing unit 1307 is further configured to input the Query matrix, the deformed Key matrix, the deformed Value matrix, and the deformation parameter matrix into a repairing network to obtain an estimated repaired feature matrix of each image frame.

[0119] According to an embodiment of the present disclosure, the deformation parameter prediction network includes a convolution layer, a pooling layer, and a full connection layer, and the deformation parameter obtaining unit 1305 is further configured to input the similarity matrix into the convolution layer to obtain a convolution matrix; input the convolution matrix into the pooling layer to obtain a pooling matrix; and input the pooling matrix into the full connection layer to obtain a deformation parameter matrix.

[0120] According to an embodiment of the present disclosure, an electronic device can be provided. Figure 14 FIG. 14 is a block diagram of an electronic device 1400 according to an embodiment of the present disclosure. The electronic device includes at least one memory 1401 and at least one processor 1402, and the at least one memory stores a set of computer executable instructions. When the set of computer executable instructions is executed by the at least one processor, a method for training a video repairing model and a method for video repairing according to an embodiment of the present disclosure are performed.

[0121] As an example, the electronic device 1400 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above-mentioned set of instructions. Here, the electronic device 1000 does not have to be a single electronic device, but can also be a collection of any devices or circuits capable of executing the above-mentioned instructions (or set of instructions) individually or jointly. The electronic device 1400 can also be part of an integrated control system or a system manager, or can be configured to interface with a local or remote (e.g., via wireless transmission) portable electronic device.

[0122] In electronic device 1400, processor 1402 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 1402 can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0123] Processor 1402 can execute instructions or code stored in memory, where memory 1401 can also store data. Instructions and data can also be transmitted and received via a network through a network interface device, which can employ any known transmission protocol.

[0124] Memory 1401 can be integrated with processor 1402, e.g., disposed within an integrated circuit microprocessor, etc. Further, memory 1402 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. Memory 1401 and processor 1402 can be operatively coupled, or can communicate with each other, e.g., through I / O ports, network connections, etc., such that processor 1402 can read files stored in memory 1401.

[0125] Further, electronic device 1400 can also include a video display, such as a liquid crystal display, and a user interface interface, such as a keyboard, a mouse, a touch input device, etc. All components of the electronic device can be connected to each other via a bus and / or a network.

[0126] According to an embodiment of the present disclosure, a computer readable storage medium is also provided, wherein when instructions in the computer readable storage medium are run by at least one processor, the at least one processor is caused to perform the training method of the video restoration model and the video restoration method according to the embodiments of the present disclosure. Examples of the computer readable storage medium herein include read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk memory, hard disk drive (HDD), solid state disk (SSD), card memory such as a multimedia card, secure digital (SD) card or extreme digital (XD) card, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store a computer program in a non-transitory manner and provide the computer program to a processor or computer so that the processor or computer can execute the computer program, as well as any associated data, data files and data structures. The computer program in the computer readable storage medium described above can be run in an environment deployed in a computer device such as a client, host, proxy device, server, etc., and in addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed by one or more processors or computers in a distributed manner.

[0127] According to an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the training method of the video restoration model and the video restoration method according to the embodiments of the present disclosure.

[0128] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects disclosed herein. The present disclosure is intended to cover any variations, uses or adaptive changes of the present disclosure following the general principles thereof and including those expressly stated or implied herein. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0129] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of video inpainting, characterized by, The method comprises the following steps: obtaining a video to be repaired; obtaining deformation information between each image frame of the video to be repaired based on image features of each image frame of the video to be repaired; performing deformation processing on each image frame of the video to be repaired based on the deformation information; obtaining a repaired video based on each image frame after deformation processing; wherein the deformation information between each image frame of the video to be repaired is obtained based on image features of each image frame, comprising: obtaining similarity information based on sub-image features of each image block in each image frame, wherein the image block is one of a predetermined number of image blocks obtained by dividing each image frame; obtaining the deformation information based on the similarity information; wherein the similarity information is obtained based on the sub-image features of each image block in each image frame, comprising: obtaining image features of each image frame of the video to be repaired; obtaining a Query matrix, a Key matrix and a Value matrix based on the image features of each image frame; obtaining a similarity matrix as the similarity information based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix.

2. The video repair method of claim 1, wherein, The similarity matrix is obtained based on the similarity between the Query sub-matrix of each image block in the Query matrix and the Key sub-matrix of each image block in the Key matrix, comprising: obtaining a mask of each image block corresponding to the Query matrix and a mask of each image block corresponding to the Key matrix; in the case that the mask of the image block in the Query matrix and the mask of the image block in the Key matrix are both 1, taking the product of the Query sub-matrix of the image block in the Query matrix and the Key sub-matrix of the image block in the Key matrix as the similarity of the two image blocks; in the case that either the mask of the image block in the Query matrix or the mask of the image block in the Key matrix is 0, taking 0 as the similarity of the two image blocks; after normalizing the similarity, merging to obtain the similarity matrix.

3. The video repair method of claim 1, wherein, The deformation information is obtained based on the similarity information, comprising: obtaining a deformation parameter matrix as the deformation information based on the similarity matrix.

4. The video repair method of claim 3, wherein, The deformation processing is performed on each image frame of the video to be repaired based on the deformation information, comprising: performing deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix.

5. The video repair method of claim 4, wherein, The deformation processing is performed on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix, comprising: For each image block, the Key sub-matrix of the image block in the Key matrix and the Value sub-matrix of the image block in the Value matrix are respectively deformed by the deformation parameter sub-matrix of the image block in the deformation parameter matrix, to obtain a deformed Key sub-matrix and a deformed Value sub-matrix of the image block; The deformed Key sub-matrices are spliced to obtain the deformed Key matrix; The deformed Value sub-matrices are spliced to obtain the deformed Value matrix.

6. The video repair method of claim 4, wherein, The video is repaired based on each image frame after deformation processing, including: Based on the Query matrix, the deformed Key matrix and the deformed Value matrix, a repaired feature matrix of each image frame is obtained; Based on the repaired feature matrix, a repaired video is obtained.

7. The video repair method of claim 6, wherein, Based on the Query matrix, the deformed Key matrix and the deformed Value matrix, a repaired feature matrix of each image frame is obtained, including: Based on the Query matrix, the deformed Key matrix, the deformed Value matrix and the deformation parameter matrix, a repaired feature matrix of each image frame is obtained.

8. The video repair method of claim 7, wherein, The video repair method is executed by a pre-trained video repair model, and the video repair model includes an encoder, a deformation parameter prediction network, a repair network and a decoder; The image features of each image frame of the video to be repaired are obtained, including: The video to be repaired is input into the encoder to obtain a feature matrix of each image frame of the video to be repaired as the image features; The deformation parameter matrix is obtained based on the similarity matrix, including: The similarity matrix is input into the deformation parameter prediction network to obtain the deformation parameter matrix; The repaired feature matrix of each image frame is obtained based on the Query matrix, the deformed Key matrix and the deformed Value matrix, including: The Query matrix, the deformed Key matrix and the deformed Value matrix are input into the repair network to obtain the repaired feature matrix of each image frame; The repaired video is obtained based on the repaired feature matrix, including: The repaired feature matrix of each image frame is input into the decoder to obtain the repaired video.

9. The video repair method of claim 8, wherein, The video repair model is trained by the following operations: A training sample set is obtained, wherein the training sample set includes a plurality of training videos and a clear video corresponding to each training video; The training video is input into the encoder to obtain a feature matrix of each image frame of the training video; Based on the feature matrix of each image frame, a Query matrix, a Key matrix and a Value matrix are obtained; obtain a similarity matrix based on similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix, wherein the image block is one of a predetermined number of image blocks respectively split from each image frame; input the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix; perform deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix; input the Query matrix, the deformed Key matrix and the deformed Value matrix into the inpainting network to obtain an estimated deformed feature matrix of each image frame; input the estimated deformed feature matrix into the decoder to obtain an estimated deformed video; adjust parameters of the encoder, the deformation parameter prediction network, the inpainting network and the decoder based on the estimated deformed video and a clear video corresponding to the training video.

10. A video repair apparatus, characterized by comprising: comprise: a video acquisition unit configured to acquire a video to be inpainted; a deformation information acquisition unit configured to obtain deformation information between each image frame of the video to be inpainted based on image features of the each image frame of the video to be inpainted; a deformation processing unit configured to perform deformation processing on each image frame of the video to be inpainted based on the deformation information; an inpainting unit configured to obtain a deformed video based on each image frame after deformation processing; wherein the deformation information acquisition unit is further configured to obtain similarity information based on sub-image features of each image block in the each image frame, wherein the image block is one of a predetermined number of image blocks respectively split from the each image frame; and obtain the deformation information based on the similarity information. wherein the deformation information acquisition unit is further configured to acquire image features of each image frame of the video to be inpainted; obtain a Query matrix, a Key matrix and a Value matrix based on the image features of the each image frame; and obtain a similarity matrix as the similarity information based on similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix.

11. The video repair apparatus of claim 10, wherein, The deformation information acquisition unit is further configured to acquire a mask of each image block corresponding to the Query matrix and a mask of each image block corresponding to the Key matrix; in a case where the mask of the image block in the Query matrix and the mask of the image block in the Key matrix are both 1, taking a product of a Query sub-matrix of the image block in the Query matrix and a Key sub-matrix of the image block in the Key matrix as a similarity of the two image blocks; in a case where any one of the mask of the image block in the Query matrix and the mask of the image block in the Key matrix is 0, taking 0 as the similarity of the two image blocks; performing merging after normalization processing of the similarity to obtain the similarity matrix.

12. The video repair apparatus of claim 10, wherein, The deformation information acquisition unit is further configured to obtain a deformation parameter matrix as the deformation information based on the similarity matrix.

13. The video repair apparatus of claim 12, wherein, The deformation processing unit is further configured to perform deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix.

14. The video repair apparatus of claim 13, wherein, The deformation processing unit is further configured to, for each image block, deform a Key sub-matrix of the image block in the Key matrix and a Value sub-matrix of the image block in the Value matrix respectively by a deformation parameter sub-matrix of the image block in the deformation parameter matrix to obtain a deformed Key sub-matrix and a deformed Value sub-matrix of the image block. The deformed Key sub-matrices are spliced to obtain the deformed Key matrix, and the deformed Value sub-matrices are spliced to obtain the deformed Value matrix.

15. The video repair apparatus of claim 13, wherein, The repair unit is further configured to obtain a repaired feature matrix of each image frame based on the Query matrix, the deformed Key matrix and the deformed Value matrix. A repaired video is obtained based on the repaired feature matrix.

16. The video repair apparatus of claim 15, wherein, The repair unit is further configured to obtain a repaired feature matrix of each image frame based on the Query matrix, the deformed Key matrix, the deformed Value matrix and the deformation parameter matrix.

17. The video repair apparatus of claim 16, wherein, The video repair device is executed through a pre-trained video repair model, and the video repair model includes an encoder, a deformation parameter prediction network, a repair network and a decoder. The deformation information acquisition unit is further configured to input the to-be-repaired video into the encoder to obtain a feature matrix of each image frame of the to-be-repaired video as the image feature. The deformation information acquisition unit is further configured to input the similarity matrix into the deformation parameter prediction network to obtain the deformation parameter matrix. The repair unit is further configured to input the Query matrix, the deformed Key matrix and the deformed Value matrix into the repair network to obtain a repaired feature matrix of each image frame. The repair unit is further configured to input the repaired feature matrix of each image frame into the decoder to obtain a repaired video.

18. The video repair apparatus of claim 17, wherein, The video repair model is trained by: obtaining a training sample set, wherein the training sample set includes a plurality of training videos and a clear video corresponding to each training video; inputting the training video into the encoder to obtain a feature matrix of each image frame of the training video; obtaining a Query matrix, a Key matrix and a Value matrix based on the feature matrix of each image frame; obtaining a similarity matrix based on a similarity between a Query sub-matrix of each image block in the Query matrix and a Key sub-matrix of each image block in the Key matrix, wherein the image block is one of a predetermined number of image blocks obtained by respectively cutting the each image frame; inputting the similarity matrix into the deformation parameter prediction network to obtain a deformation parameter matrix; performing deformation processing on the Key matrix and the Value matrix based on the deformation parameter matrix to obtain a deformed Key matrix and a deformed Value matrix; inputting the Query matrix, the deformed Key matrix and the deformed Value matrix into the repair network to obtain an estimated repaired feature matrix of each image frame; inputting the estimated repaired feature matrix into the decoder to obtain an estimated repaired video; and adjusting parameters of the encoder, the deformation parameter prediction network, the repair network and the decoder based on the estimated repaired video and the clear video corresponding to the training video.

19. An electronic device, comprising: comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the video repair method of any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium, when the instructions therein are executed by at least one processor, causes the at least one processor to perform the video repair method of any one of claims 1 to 9.

21. A computer program product comprising computer instructions, characterized in that, The computer instructions, when executed by a processor, implement the video repair method of any one of claims 1 to 9.

Images