Method executed by electronic equipment, electronic equipment and storage medium

Abstract

The invention relates to a method executed by electronic equipment, the electronic equipment and a storage medium. The method comprises the following steps: extracting at least one key frame and at least one non-key frame from a video; repairing the at least one key frame based on at least one mask corresponding to the at least one key frame; and repairing the at least one non-key frame based on the repaired at least one key frame.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular, to a method executed by an electronic device, an electronic device, and a storage medium. Background Art

[0002] In many application scenarios of videos, there is a technical demand for removing objects from video frames and completing missing areas in frames. In particular, with the popularization of mobile terminals such as smart phones and tablets, people's demand for using mobile terminals to shoot and process videos has gradually increased. However, in related technologies, technologies such as removing objects from frames or completing missing areas are inefficient, waste too much resources, have slow processing speeds, are difficult to process long videos, or have unstable completion effects.

[0003] How to efficiently remove objects or complete missing areas to better meet user needs is a technical problem that technicians in this field have been working hard to study. Summary of the invention

[0004] In order to at least solve the above problems existing in the prior art, the present invention provides a method executed by an electronic device, an electronic device and a storage medium.

[0005] According to a first aspect of an embodiment of the present application, a method performed by an electronic device is provided, comprising: extracting at least one key frame and at least one non-key frame from a video; repairing the at least one key frame based on at least one mask corresponding to the at least one key frame; and repairing the at least one non-key frame based on the repaired at least one key frame.

[0006] Optionally, the step of extracting at least one key frame and at least one non-key frame from the video includes: extracting the at least one key frame and the at least one non-key frame from the video according to decoding information of the video.

[0007] Optionally, the step of extracting the at least one key frame from the video according to the decoded information of the video includes: extracting at least one frame from the video according to the decoded information of the video; and determining the at least one key frame from the extracted frames based on a predetermined time interval.

[0008] Optionally, based on at least one mask corresponding to the at least one key frame, the step of repairing the at least one key frame includes: repairing each group of key frames in the at least one key frame by the following operations: extracting features of a group of key frames to be repaired; based on features of the repaired key frames, processing features of the group of key frames; based on the processed features, decoding the group of key frames; wherein the features of the repaired key frames include: a first feature related to all groups of repaired key frames and / or a second feature related to the last group of repaired key frames.

[0009] Optionally, the method further includes: extracting a third feature from the processed features based on semantic relevance; fusing the third feature with the stored first feature, and storing the fusion result as the updated first feature.

[0010] Optionally, the method further includes: updating the second feature based on the processed feature.

[0011] Optionally, based on the features of the repaired key frames, the step of processing the features of the group of key frames includes: based on a set of masks corresponding to the group of key frames, processing the features of the group of key frames to obtain preliminary repair features of the group of key frames; based on the first feature, processing the preliminary repair features of the group of key frames to obtain processed preliminary repair features; splicing the second feature with the processed preliminary repair features; and obtaining the processed features by processing the splicing result.

[0012] Optionally, based on the first feature, the preliminary restoration features of the group of key frames are processed, and the step of obtaining the processed preliminary restoration features includes: obtaining a group of fourth features by dividing the preliminary restoration features of the group of key frames in the time dimension, wherein each fourth feature in the group of fourth features corresponds to a key frame in the group of key frames; for each fourth feature in the group of fourth features, extracting a fifth feature about the area corresponding to the object to be removed from the first feature; and obtaining the processed preliminary restoration features by splicing the fifth feature extracted for each fourth feature in the group of fourth features, and adding the splicing result to the preliminary restoration features of the group of key frames.

[0013] Optionally, for each fourth feature in the group of fourth features, the step of extracting a fifth feature about the area corresponding to the object to be removed from the first feature includes: tiling a fourth feature in the group of fourth features and the first feature, respectively; calculating a similarity matrix for the tiled fourth feature and the tiled first feature; obtaining a weight matrix by normalizing the similarity matrix; and obtaining a fifth feature about the area corresponding to the object to be removed based on the first feature and the weight matrix.

[0014] Optionally, the step of updating the second feature based on the processed feature includes: obtaining a group of sixth features by dividing the processed features of the group of key frames in the time dimension, wherein each sixth feature in the group of sixth features corresponds to a key frame in the group of key frames; updating the second feature by processing the group of sixth features using a neural network formed by at least one gated recurrent unit GRU module connected in series.

[0015] Optionally, based on semantic relevance, the step of extracting a third feature from the processed features includes: obtaining a group of seventh features by dividing the processed features of the group of key frames in the time dimension, wherein each seventh feature in the group of seventh features corresponds to a key frame in the group of key frames; obtaining a group of eighth features by performing feature compression on each seventh feature in the group of seventh features in the spatial dimension; and obtaining a third feature by performing feature compression on the splicing result of the group of eighth features in the time dimension.

[0016] Optionally, the step of obtaining a group of eighth features by performing feature compression on each seventh feature in the group of seventh features in the spatial dimension includes: performing the following operations on each seventh feature to obtain an eighth feature: performing feature block tiling on a seventh feature; obtaining a ninth feature corresponding to the seventh feature by performing the following operations: for each feature block in the seventh feature, calculating a similarity matrix between the feature blocks in the seventh feature and each feature block in the seventh feature, and fusing the feature blocks based on the similarity matrix; obtaining a tenth feature corresponding to the seventh feature by performing full connection on the feature blocks of the ninth feature; and obtaining an eighth feature by rearranging the tenth feature.

[0017] Optionally, the step of obtaining the third feature by performing feature compression on the splicing result of the group of eighth features in the time dimension includes: performing feature block tiling on the splicing result of the group of eighth features; obtaining the eleventh feature corresponding to the splicing result of the group of eighth features by the following operations: for each feature block in the splicing result of the group of eighth features, calculating the similarity matrix between each feature block in the splicing result of the group of eighth features and the splicing result of the group of eighth features, and fusing the feature blocks based on the similarity matrix; obtaining the twelfth feature corresponding to the splicing result of the group of eighth features by performing full connection on the feature blocks of the eleventh feature; and obtaining the third feature by rearranging the twelfth feature.

[0018] Optionally, based on the at least one repaired key frame, the step of repairing the at least one non-key frame includes: repairing each of the at least one non-key frame by the following operations: obtaining at least one aligned first key frame by aligning at least one first key frame related to the current non-key frame to the current non-key frame; and repairing the current non-key frame based on the at least one aligned first key frame.

[0019] Optionally, the step of obtaining at least one aligned first key frame includes: performing the following operation for each first key frame to obtain an aligned first key frame: obtaining an aligned first key frame based on motion vector information of a current non-key frame relative to a first key frame.

[0020] Optionally, the step of obtaining an aligned first key frame based on the motion vector information of the current non-key frame relative to a first key frame includes: obtaining the masked motion vector information of the current non-key frame based on the mask corresponding to the first key frame and the motion vector information of the current non-key frame relative to the first key frame; for the first key frame, obtaining an affine transformation matrix based on the masked motion vector information; and obtaining an aligned first key frame by processing the first key frame based on the affine transformation matrix.

[0021] Optionally, based on the at least one aligned first key frame, the step of repairing the current non-key frame includes: obtaining a thirteenth feature of the current non-key frame based on the current non-key frame and the corresponding mask; obtaining a fourteenth feature of each of the at least one aligned first key frame; obtaining a fifteenth feature of the current non-key frame by fusing the thirteenth feature and at least one fourteenth feature of the at least one aligned first key frame; and obtaining a repaired current non-key frame based on the fifteenth feature.

[0022] Optionally, the step of obtaining the fifteenth feature of the current non-key frame by fusing the thirteenth feature and at least one fourteenth feature of the at least one aligned first key frame includes: obtaining at least one similarity matrix based on the thirteenth feature and the at least one fourteenth feature; obtaining at least one weight matrix based on the at least one similarity matrix; obtaining at least one sixteenth feature based on the at least one weight matrix and the at least one fourteenth feature; and obtaining the fifteenth feature of the current non-key frame by fusing the at least one sixteenth feature and the thirteenth feature.

[0023] Optionally, based on the at least one similarity matrix, the step of obtaining at least one weight matrix includes: obtaining at least one pooled similarity matrix by performing channel dimension average pooling on each of the at least one similarity matrix; processing the at least one pooled similarity matrix based on temporal position information related to the at least one aligned key frame; and normalizing the processed at least one pooled similarity matrix to obtain at least one weight matrix.

[0024] Optionally, the method also includes: after repairing the at least one key frame, displaying a video containing the repaired at least one key frame; receiving a continue repair instruction input by the user, wherein the step of repairing the at least one non-key frame based on the repaired at least one key frame includes: repairing the at least one non-key frame based on the repaired at least one key frame according to the continue repair instruction; and displaying a video containing the repaired at least one key frame and the repaired at least one non-key frame.

[0025] Optionally, the method further includes: after repairing the at least one key frame, displaying a video including the at least one repaired key frame; receiving a re-repair instruction input by a user; and re-executing the extraction according to the re-repair instruction.

[0026] According to a second aspect of an embodiment of the present application, a method performed by an electronic device is provided, including: determining a target repair area in a first video; using a first neural network to repair the target repair area in a key frame of the first video; displaying a second video containing the repaired key frame; receiving a continue repair instruction input by a user; according to the continue repair instruction, using a second neural network to repair the target repair area in a non-key frame of the first video based on the repaired key frame; and displaying the repaired first video.

[0027] Optionally, the step of determining a target repair area in the first video includes: determining the target repair area in the first video according to user selection.

[0028] Optionally, the method further includes: after displaying the second video, receiving a re-repair instruction input by a user; redetermining a target repair area in the first video according to the re-repair instruction; and repairing the re-determined target repair area in a key frame of the first video using a first neural network.

[0029] Optionally, the method further includes: after displaying the second video, receiving setting instructions input by the user for setting other target repair areas; determining the other target repair areas in the repaired key frame according to the setting instructions; and repairing the other target repair areas in the repaired key frame using the first neural network.

[0030] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the method performed by the electronic device as described above.

[0031] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium storing instructions is provided, wherein, when the instructions are executed by at least one processor, the at least one processor is prompted to execute the method performed by the electronic device as described above.

[0032] The beneficial effects brought about by the technical solutions provided by the embodiments of the present application will be described below in conjunction with specific optional embodiments, or can be learned from the description of the embodiments, or can be known through the implementation of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly and easily illustrate and understand the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in describing the embodiments of the present application.

[0034] Figure 1 is a flowchart illustrating a method performed by an electronic device according to an exemplary embodiment of the present application.

[0035] Figure 2 The present invention is a process diagram showing a method executed by an electronic device according to an exemplary embodiment of the present application.

[0036] Figure 3 A decoding schematic diagram is shown.

[0037] Figure 4 is a schematic diagram showing the distribution of I frames, P frames and B frames.

[0038] Figure 5The present invention is a process diagram showing a method executed by an electronic device according to another exemplary embodiment of the present application.

[0039] Figure 6 The present invention is a process diagram showing a method executed by an electronic device according to another exemplary embodiment of the present application.

[0040] Figure 7 is a flow chart illustrating a process of repairing a set of key frames according to an exemplary embodiment of the present application.

[0041] Figure 8 is a processing diagram showing a spatiotemporal memory Transformer module according to an exemplary embodiment of the present application.

[0042] Fig. 9 is a flowchart illustrating a process of processing features of a group of key frames based on features of restored key frames according to an exemplary embodiment of the present application.

[0043] Fig.10 is a schematic diagram showing the processing process of the long-term memory reading module according to an exemplary embodiment of the present application.

[0044] Fig.11 is a schematic diagram showing updating of spatiotemporal memory according to an exemplary embodiment of the present application.

[0045] Fig.12 is a block diagram illustrating a short-term memory update module according to an exemplary embodiment of the present application.

[0046] Fig.13 is a flowchart illustrating a process of extracting a third feature according to an exemplary embodiment of the present application.

[0047] Fig.14 is a block diagram illustrating a long-term memory update module according to an exemplary embodiment of the present application.

[0048] Fig.15 is a schematic diagram illustrating a process of repairing a non-key frame according to an exemplary embodiment of the present application.

[0049] Fig.16 is a schematic diagram illustrating a process of repairing a non-key frame according to an exemplary embodiment of the present application.

[0050] Fig.17 is a schematic diagram showing the structure of an alignment module according to an exemplary embodiment of the present application.

[0051] Fig.18 is a schematic diagram illustrating a process of repairing a non-key frame according to another exemplary embodiment of the present application.

[0052] Fig.19 Detailed description is a flowchart illustrating a process of obtaining an aligned first key frame based on motion vector information of a current non-key frame relative to a first key frame according to an exemplary embodiment of the present application.

[0053] Fig. 20 is a flowchart illustrating a process of repairing a current non-key frame based on at least one aligned first key frame according to an exemplary embodiment of the present application.

[0054] Fig.21a is a flowchart illustrating a process of fusing a thirteenth feature with at least one fourteenth feature of the at least one aligned first key frame to obtain a fifteenth feature of a current non-key frame according to an exemplary embodiment of the present application.

[0055] Figure 21b is a schematic diagram showing the processing process of the temporal attention fusion module according to an exemplary embodiment of the present application.

[0056] Fig. 22 is a schematic diagram showing time location information according to an exemplary embodiment of the present application.

[0057] Fig.23 is a schematic diagram showing the processing process of the temporal attention fusion module according to another exemplary embodiment of the present application.

[0058] Fig.24 is a flowchart illustrating a method performed by an electronic device according to another exemplary embodiment of the present application.

[0059] Fig.25a is a flowchart illustrating an example of applying a method performed by an electronic device according to an exemplary embodiment of the present application.

[0060] Fig.25b is a flowchart illustrating another example of applying a method performed by an electronic device according to an exemplary embodiment of the present application.

[0061] Fig.26 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0063] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B". When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A including A1 or A2 or A3, and can also be implemented as parameter A including at least two of the three items A1, A2, A3.

[0064] Figure 1 is a flowchart illustrating a method performed by an electronic device according to an exemplary embodiment of the present application. Figure 2 The present invention is a process diagram showing a method executed by an electronic device according to an exemplary embodiment of the present application.

[0065] In step S110 , at least one key frame and at least one non-key frame are extracted from the video.

[0066] In an exemplary embodiment of the present disclosure, the video in step S110 may be obtained by decoding a compressed video file with a code to be masked by a decoder. The step of extracting at least one key frame and at least one non-key frame from the video may include: extracting at least one key frame and at least one non-key frame from the video according to the decoded information of the video. In an exemplary embodiment, an intra-coded (I) frame and / or a forward prediction (P) frame is extracted as at least one key frame, and a bi-directional interpolated prediction (B) frame is extracted as at least one non-key frame. In addition, when extracting at least one non-key frame from the video, motion vector information of at least one non-key frame can also be obtained from the decoded information of the video.

[0067] like Figure 2 As shown in , the compressed video file with the mask is first decoded by a video decoder, wherein the video decoder can support various video codec standards, such as MPEG4, H.264, HEVC and other standards, and the video decoder can be, for example, an FFmpeg decoder, but the present application is not limited thereto. After decoding, the key frames and non-key frames of the decoded video can be extracted according to the decoding information of the video, and the motion vector information of each non-key frame can also be obtained according to the decoding information of the video. For example, Figure 3 A schematic diagram of video decoding is shown. Figure 3 In the embodiment of the present invention, when a compressed video is decoded using a video decoder (such as an FFmpeg decoder), three types of frames can be extracted, namely, I frames, P frames, and B frames, wherein the I frame retains the complete information of the original image, the P frame retains the motion vector information relative to the I frame, and the B frame retains the motion vector information relative to the I frame and the P frame. Since the I frame and the P frame contain a large amount of useful information, and the B frame contains a large amount of redundant or repeated information, in an exemplary embodiment of the present application, the I frame and the P frame can be classified as key frames, and the B frame can be classified as a non-key frame. In addition, when the same video is encoded using different video codec standards, different proportions of I frames, P frames, and B frames will be generated. For example, Figure 4 The distribution of I frames, P frames and B frames in a video encoded using the H.264 standard is shown, wherein the ratio of I frames to P frames (i.e., key frames) is approximately 20% to 30%, and the ratio of B frames (i.e., non-key frames) is approximately 76.8%. However, this is only an example, and the present application can use various video encoding and decoding standards to decode videos encoded according to the corresponding standards, and accordingly extract I frames and / or P frames as key frames, and extract B frames as non-key frames.

[0068] In the exemplary embodiments described above, I frames and / or P frames are extracted as key frames, and B frames are extracted as non-key frames. The present disclosure proposes that the number of key frames can be further reduced, and the step of extracting at least one key frame from a video according to the decoded information of the video may include: extracting at least one frame from the video according to the decoded information of the video, and determining at least one key frame from the extracted frames based on a predetermined time interval. That is, first, at least one frame consisting of I frames and / or P frames is extracted from the video according to the decoded information of the video, and then at least one frame of the extracted at least one frame is extracted as a key frame according to a predetermined time interval. In addition, frames in the video other than the extracted key frames may be extracted as non-key frames. Specifically, as Figure 5As shown in , it shows a process diagram of a method executed by an electronic device according to another exemplary embodiment of the present application, for example, at least one frame is extracted from a video according to decoding information, a key frame is selected from the at least one extracted frame according to a predetermined frame interval, and other unselected frames in the at least one frame are set as non-key frames, for example, one frame is selected from the at least one extracted frame as a key frame every 3 frames, and all frames in the video except the selected key frame are set to (or extracted as) non-key frames. In this case, since the number of key frames is reduced, the overall processing speed can be further improved.

[0069] In another exemplary embodiment of the present application, the step of extracting at least one key frame and at least one non-key frame from a video may include: extracting frames from the video at a predetermined frame interval to obtain at least one key frame; extracting at least one frame in the video other than the at least one key frame as at least one non-key frame.

[0070] Specifically, if Figure 6 As shown in the above reference Figure 2 and Figure 5 The described exemplary embodiment of extracting key frames and non-key frames from a video based on decoded information is different. In this exemplary embodiment, key frames can be extracted from the decoded video at a predetermined frame interval, and the remaining frames can be extracted as non-key frames. For example, one frame is extracted from every 5 frames in the video as a key frame, so that at least one key frame can be obtained, and the remaining frames are extracted as non-key frames.

[0071] Return to reference Figure 1 In step S120, at least one key frame is repaired based on at least one mask corresponding to the at least one key frame. In the present application, each group of key frames in the at least one key frame is repaired in a common processing manner, wherein each group of key frames includes a predetermined number of key frames, wherein the predetermined number can be any positive integer greater than or equal to 1, for example, every five key frames arranged consecutively in the at least one key frame are regarded as a group. Figure 7 and Figure 8 A detailed description of the process of repairing a set of keyframes. Figure 7 is a flow chart illustrating a process of repairing a set of key frames according to an exemplary embodiment of the present application. Figure 8 is a processing diagram showing a spatiotemporal memory Transformer module according to an exemplary embodiment of the present application.

[0072] Specifically, in step S710, features of a set of key frames to be repaired are extracted. Specifically, features of a set of key frames to be repaired can be obtained by encoding the features of the set of key frames to be repaired. In this article, "features" can also be referred to as "semantic features". Figure 8 As shown in th The block is input into the block embedding module for feature encoding to obtain block features. In the following description, for ease of understanding, it is assumed that the number of frames in each group of key frames is 5, but the present application is not limited to this, and it can be any positive integer.

[0073] In step S720, based on the features of the repaired key frames, the features of the group of key frames are processed, wherein the features of the repaired key frames include: first features related to all groups of repaired key frames and / or second features related to the last group of repaired key frames. Figure 8 and Fig. 9 The process of processing the features of a set of key frames to obtain processed features is described in detail.

[0074] Fig. 9 is a flowchart illustrating a process of processing features of a set of key frames based on features of restored key frames according to an exemplary embodiment of the present application.

[0075] In step S7210, based on a set of masks corresponding to the set of key frames, features of the set of key frames are processed to obtain preliminary restoration features of the set of key frames.

[0076] like Figure 8 As shown in , the features of the set of key frames obtained by the block embedding module and a set of masks corresponding to the set of key frames (not shown) are input into a Transformer module (e.g., a lightweight Transformer module) to process the features of the set of key frames, thereby obtaining preliminary repair features of the set of key frames. That is, by using a lightweight Transformer module to roughly repair the area corresponding to the mask in each of the set of key frames, a coarse feature Fi of a set of key frames is obtained. th , where the coarse feature Fi th Contains all features after a set of keyframes are initially restored.

[0077] In step S7220, based on the first feature, the preliminary restoration features of the group of key frames are processed to obtain the processed preliminary restoration features. In this application, the "first feature" is a feature related to all the groups of key frames that have been restored, which can also be called "long-term memory". Figure 8As shown in FIG, the memory reading module uses the long-term memory (i.e., the first feature) to read the rough features Fi of a set of key frames. th (i.e., the preliminary restoration features of the set of key frames) are processed (e.g., refined) to obtain the processed preliminary restoration features. Fig.10 The processing process of the memory reading module (ie, the process of processing the preliminary restoration features of the set of key frames based on long-term memory to obtain the processed preliminary restoration features) is described in detail.

[0078] Fig.10 is a schematic diagram showing the processing process of the long-term memory reading module according to an exemplary embodiment of the present application.

[0079] First, a set of fourth features is obtained by dividing the preliminary restoration features of the set of key frames in the time dimension, wherein each fourth feature in the set of fourth features corresponds to a key frame in the set of key frames. Fig.10 As shown in , assuming a set of keyframe coarse features Fi th (i.e., the preliminary restoration features of the set of key frames) is [5, 256, 32, 32]. The partitioning module first divides the coarse features Fi in the time dimension. th Divided into a group of fourth feature maps, i.e., 5 fourth features, wherein the group of key frames includes 5 key frames in total, i.e., the 1st key frame to the 5th key frame, and each fourth feature corresponds to a corresponding one of the 1st key frame to the 5th key frame, for example, the 1st fourth feature 1 corresponds to the 1st key frame, the 2nd fourth feature corresponds to the 2nd key frame, the 3rd fourth feature corresponds to the 3rd key frame, the 4th fourth feature corresponds to the 4th key frame, and the 5th fourth feature corresponds to the 5th key frame.

[0080] Then, for each fourth feature in the set of fourth features, a fifth feature about the area corresponding to the object to be removed is extracted from the first feature (i.e., long-term memory). In the present application, the fifth feature may also be referred to as a useful feature. In addition, the object to be removed may be an object or subject selected by a user in the process of browsing the video, but the present application is not limited thereto.

[0081] Specifically, the memory reading block tiles one of the fourth features and the first feature (i.e., long-term memory) in the set of fourth features respectively; the memory reading block calculates a similarity matrix for the tiled fourth feature and the tiled first feature (i.e., long-term memory); the memory reading block obtains a weight matrix by normalizing the similarity matrix; the memory reading block obtains a fifth feature of the area corresponding to the object to be removed based on the first feature (i.e., long-term memory) and the weight matrix. Fig.10As shown in , suppose that the first fourth feature [1,256,32,32] is combined with the long-term memory Li th Each of [1,256,8,8] is tiled to obtain a one-dimensional feature block vector corresponding to the first fourth feature and a long-term memory Li th The corresponding one-dimensional feature block vector; then, the memory reading block calculates the one-dimensional feature block vector corresponding to the first fourth feature and the one-dimensional feature block vector corresponding to the long-term memory Li th The similarity matrix between the corresponding one-dimensional feature block vectors, for example, is obtained Fig.10 The similarity matrix [1, 8×8, 32×32] shown in FIG is obtained; thereafter, the memory reading block uses the Softmax activation function to normalize the similarity matrix to obtain a weight matrix; then, the weight matrix is ​​compared with the long-term memory Li th [1,256,8,8] are multiplied (ie, matrix multiplication, MatMul) to obtain the fifth feature [1,256,32,32] for the area corresponding to the object to be removed.

[0082] After extracting a fifth feature about the area corresponding to the object to be removed from the first feature (i.e., long-term memory) for each of the fourth features in the set of fourth features, the processed preliminary restoration features are obtained by splicing the fifth features extracted for each of the fourth features in the set of fourth features and adding the splicing result to the preliminary restoration features of the set of key frames. Fig.10 As shown in, the fifth feature [1,256,32,32] output by all five memory read blocks is concatenated to obtain a concatenated result [5,256,32,32], and then the concatenated result [5,256,32,32] is concatenated with the coarse feature Fi of the set of key frames. th [5,256,32,32] are added together to finally obtain the processed preliminary restoration features [5,256,32,32].

[0083] Return to reference Fig. 9 In step S7230, the second feature is concatenated with the processed preliminary repair feature. In this application, the "second feature" is a feature related to the previous set of key frames that have been repaired, which can be called "short-term memory". Figure 8 As shown in , the short-term memory S(i-1) th Splice with the processed preliminary repair features.

[0084] In step S7240, the splicing result is processed to obtain the processed features, that is, the final restoration features of the set of key frames. Figure 8As shown in , the splicing result is input into the second Transformer module (e.g., a lightweight Transformer module) for processing to obtain the final repair feature Ri of the i-th group of key frames th .

[0085] return Figure 7 In step S730, the group of key frames is decoded based on the processed features. Specifically, the group of key frames after restoration is obtained by performing feature decoding on the final restoration features of the group of key frames. Figure 8 As shown in , by the final restoration feature Ri of the i-th group of key frames th Perform feature decoding and finally obtain a set of repaired feature frames Pi th .

[0086] In the above reference Figures 7 to 10 In the process described above, by introducing a memory mechanism consisting of a first feature (i.e., long-term memory) and a second feature (i.e., short-term memory), the redundant calculations caused by overlap can be eliminated while improving the video restoration effect. In this application, the first feature and the second feature need to be updated during use, that is, spatiotemporal memory update is required. Specifically, Figure 1 The method shown in may also include the step of updating the first feature, that is, extracting a third feature from the processed feature based on semantic relevance; and fusing the third feature with the stored first feature, and storing the fusion result as the updated first feature. In addition, the method may also include the step of updating the second feature, that is, updating the second feature based on the processed feature. Fig.11 Let's introduce the spatiotemporal memory update process proposed in this application.

[0087] Fig.11 is a schematic diagram showing updating of spatiotemporal memory according to an exemplary embodiment of the present application.

[0088] In the present application, the second feature (i.e., short-term memory) stores the features of the previous set of key frames of the iteration, and helps to ensure that the time series of the restored video has good continuity. The first feature (i.e., long-term memory) is updated based on the final repair features of the current set of key frames and the previously stored first feature (i.e., the previously stored long-term memory), which is used to store long-term features. In addition, due to considerations of memory usage and computational complexity, a common memory queue is not used to store these long-term memories and short-term memories. Instead, a long-term memory update module (also referred to as a first feature update module) is used to update the long-term memory.

[0089] Specifically, if Fig.11As shown in , when the first group of key frames (i.e., 5 key frames) are repaired, both the short-term memory and the long-term memory are empty. When the second group of key frames (i.e., 5 key frames) are restored, the short-term memory is generated by the final repair features of the first group of key frames, and the long-term memory is generated by the final repair features of the first group of key frames. When the third group of key frames is restored, the short-term memory is generated (or updated) by the final restored features of the second group of key frames, and the long-term memory is generated (or updated) by the final restored features of the second group of key frames and the previous long-term memory (i.e., the second long-term memory), and so on.

[0090] Next, we will refer to Fig.12 The short-term memory updating module (also referred to as the second feature updating module) shown is used to describe the detailed process of updating the short-term memory (ie, the second feature).

[0091] Fig.12 is a block diagram illustrating a short-term memory update module according to an exemplary embodiment of the present application.

[0092] First, a group of sixth features is obtained by dividing the processed features (i.e., final repair features) of the group of key frames in the time dimension, wherein each sixth feature in the group of sixth features corresponds to a key frame in the group of key frames.

[0093] like Fig.12 As shown in , the short-term memory update module includes a partitioning module, which partitions a set of processed features (i.e., final repair features) Ri of key frames in the time dimension. th [5,256,32,32] is divided into a group of sixth features, for example, 5 sixth features. Specifically, the group of key frames includes 5 key frames in total, namely, the 1st key frame to the 5th key frame, and each sixth feature corresponds to a corresponding one of the 1st key frame to the 5th key frame. For example, the 1st sixth feature corresponds to the 1st key frame, the 2nd sixth feature corresponds to the 2nd key frame, the 3rd sixth feature corresponds to the 3rd key frame, the 4th sixth feature corresponds to the 4th key frame, and the 5th sixth feature corresponds to the 5th key frame.

[0094] Then, the second feature is updated, i.e., the short-term memory is updated, by processing the set of sixth features using a neural network formed by at least one gated recurrent unit (GRU) module connected in series, wherein the input of each GRU module includes a corresponding sixth feature in the set of sixth features, and the input of the remaining GRU modules except the first GRU module also includes the output of the previous GRU module connected in series.

[0095] like Fig.12As shown in , the short-term memory update module also includes a plurality of GRU modules connected in series, and the number of the plurality of GRU modules is equal to the number of features included in a set of sixth features. Fig.12 In the example, the number of multiple GRU modules is 5, and the five divided sixth features are continuously fed to the series-connected GRU modules. Specifically, the first sixth feature [1, 256, 32, 32] is input into the first GRU module as the hidden state at the previous moment and the input information at the current moment, respectively. The output of the first GRU module and the second sixth feature [1, 256, 32, 32] are input into the second GRU module as the hidden state at the previous moment and the input information at the current moment, respectively. By analogy, the output of the fourth GRU module and the fifth sixth feature [1, 256, 32, 32] are input into the fifth GRU module as the hidden state at the previous moment and the input information at the current moment, respectively. Finally, the output of the fifth GRU module is the updated second feature (i.e., short-term memory) Si th [1,256,32,32].

[0096] In the short-term memory update module, the GRU module filters out important features from the features of a key frame and passes these features to the next GRU module in series, and finally generates a short-term memory of a set of features, which can store more detailed texture features than long-term memory. The short-term memory proposed in this application retains the most important information in adjacent frames. Since this short-term memory is not compressed by the spatial scale, it helps to restore more detailed content in the input key frame.

[0097] Refer to the following Fig.13 and Fig.14 To describe the process of updating the first feature (i.e. long-term memory). Fig.13 is a flowchart illustrating a process of extracting a third feature according to an exemplary embodiment of the present application, wherein, as described above, the third feature is fused with the previously stored first feature, and the fusion result is stored as the updated first feature. Fig.14 is a block diagram illustrating a long-term memory update module according to an exemplary embodiment of the present application.

[0098] First, in step S1310, a set of seventh features is obtained by dividing the processed features (i.e., the final repair features) of the set of key frames in the time dimension, wherein each seventh feature in the set of seventh features corresponds to a key frame in the set of key frames. Fig.14 As shown in , the long-term memory update module includes a feature compression module and a feature fusion module, and the feature compression module includes a partitioning module. Here, the partitioning module converts the final restoration features Ri of a set of key frames in the time dimension. th[5,256,32,32] is divided into a set of seventh features, i.e., for example Fig.14 5 seventh features, specifically, the group of key frames includes 5 key frames in total, for example, the 1st key frame to the 5th key frame, each seventh feature corresponds to a corresponding one of the 1st key frame to the 5th key frame, for example, the 1st seventh feature corresponds to the 1st key frame, the 2nd seventh feature corresponds to the 2nd key frame, the 3rd seventh feature corresponds to the 3rd key frame, the 4th seventh feature corresponds to the 4th key frame, and the 5th seventh feature corresponds to the 5th key frame.

[0099] Then, in step S1320, a set of eighth features is obtained by performing feature compression on each of the set of seventh features in the spatial dimension. Specifically, Fig.14 As shown in , the feature compression module also includes multiple spatial pruning attention modules, wherein the number of spatial pruning attention modules is equal to the number of seventh features included in a set of seventh features. Fig.14 In the figure, the number of multiple spatial pruning attention modules is 5, and the five seventh features divided are fed to five spatial pruning attention modules respectively. Then, each spatial pruning attention module processes a corresponding seventh feature to obtain an eighth feature, which is described in detail below.

[0100] Specifically, if Fig.14 As shown in , first, the spatial pruning attention module flattens the feature block of the input seventh feature [1, 256, 32, 32].

[0101] Then, the ninth feature corresponding to the seventh feature is obtained by the following operation: for each feature block in the seventh feature, the spatial pruning attention module calculates the similarity matrix between each feature block in the seventh feature, and fuses the feature blocks based on the similarity matrix, for example, the most similar first k feature blocks for each feature block are fused. k is a configurable value, which can be set according to user needs, for example, to 5, 10, 15, etc. Specifically, Fig.14As shown in , there are 1024 feature blocks in total for the seventh feature [1,256,32,32]. Assuming that the current feature block is the first feature block after tiling, it is necessary to calculate the similarity matrix between each feature block in the seventh feature (including the current feature block and the remaining feature blocks in the seventh feature) to determine the top k feature blocks that are most similar to the current feature block. Thereafter, the top k feature blocks that are most similar to the current feature block are fused (that is, the current feature block is fused with the top k-1 feature blocks that are most similar to it) to obtain the fused current feature block. According to a similar process, the fused feature blocks of each feature block in the seventh feature can be obtained, and then the ninth feature corresponding to the seventh feature is obtained. For example, Fig.14 The ninth feature in [1,256,1024].

[0102] Thereafter, a tenth feature corresponding to the seventh feature is obtained by performing full connection on the feature block of the ninth feature. Specifically, Fig.14 As shown in , the 1024 feature blocks of the seventh feature [1,256,1024] can be fully connected to reduce the number of feature blocks, thereby obtaining the tenth feature [1,256,64]. However, this is only an example, and the number of feature blocks in the tenth feature is not limited thereto, and can also be other values.

[0103] Then, by rearranging the tenth feature, an eighth feature is obtained, that is, a feature after feature compression in the spatial dimension is obtained. Fig.14 As shown in , after rearranging the tenth feature [1,256,64], the eighth feature [1,256,8,8] is obtained.

[0104] Return to reference Fig.13 In step S1330, the third feature is obtained by performing feature compression on the splicing result of the set of eighth features in the time dimension. Specifically, Fig.14 As shown in , the feature compression module also includes a time pruning attention module. In this application, the structure of the time pruning attention module is the same as Fig.14 The structure of the spatial pruning attention module shown in is the same.

[0105] Specifically, first, a set of eighth feature concatenation results [5,256,8,8] are tiled by feature blocks. The concatenation result [5,256,8,8] is obtained by concatenating the outputs of 5 spatial pruning attention modules, and here, the concatenation result [5,256,8,8] can be regarded as a feature. Therefore, after tiling it according to the feature block, a feature [1,256,320] is obtained, where 320=8×8×5.

[0106] Thereafter, the eleventh feature corresponding to the splicing result of the set of the eighth features is obtained by the following operation: for each feature block in the splicing result of the set of the eighth features, a similarity matrix between the feature blocks and each feature block in the splicing result of the set of the eighth features is calculated, and the feature blocks are merged based on the similarity matrix, for example, the first m feature blocks that are most similar to each feature block are merged. Similar to k, m is also a configurable value.

[0107] Then, by fully connecting the feature blocks of the eleventh feature, a twelfth feature corresponding to the concatenation result of the set of eighth features is obtained. Specifically, the twelfth feature [1,256,64] can be obtained by fully connecting the 320 feature blocks of the eleventh feature [1,256,320] to reduce the number of feature blocks, but this is only an example, and the number of feature blocks in the twelfth feature is not limited thereto, and can also be other values.

[0108] Thereafter, the third feature is obtained by rearranging the twelfth feature, that is, the feature after feature compression in the time dimension (that is, the compressed feature) is obtained, thereby obtaining the result enhanced by time attention. Specifically, after rearranging the twelfth feature [1,256,64], the third feature [1,256,8,8] is obtained.

[0109] Through the above Fig.13 After the third feature is extracted through the process, the third feature is fused with the stored first feature, and the fusion result is stored as the updated first feature.

[0110] Specifically, if Fig.14 As shown in , the feature fusion module includes a multi-head attention module, a first normalization module, a feedforward neural network and a second normalization module. The current compressed feature [1,256,8,8] is used as the input q and the previously stored first feature (i.e., long-term memory) L(i-1) is used as the input q. th [1,256,8,8] is input as input quantities k and v to the multi-head attention module for processing, and then the first normalization module is used to normalize the processing results of the multi-head attention module and the current compressed features [1,256,8,8]. Thereafter, the normalized results are fed to the second normalization module for normalization using a feedforward neural network, and finally an updated first feature can be obtained. The updated first feature can combine the feature information of the current moment with the feature information of all previous moments. That is to say, the first feature can store effective features for a longer time, and the feature fusion module can compress these features in terms of time and space scales.

[0111] Return to reference Figure 1In step S130, based on the repaired at least one key frame, the at least one non-key frame is repaired. Here, each non-key frame in the at least one non-key frame is repaired through a similar process. Fig.15 and Fig.16 This is described in detail.

[0112] Fig.15 is a flowchart illustrating a process of repairing a non-key frame according to an exemplary embodiment of the present application. Fig.16 It is a schematic diagram showing a process of repairing a non-key frame according to an exemplary embodiment of the present application, wherein the repair process is implemented by a fast matching convolutional neural network proposed in the present application, wherein the fast matching convolutional neural network includes two alignment modules, multiple encoders, a temporal attention fusion module, and a decoder. Fig.17 is a schematic diagram showing the structure of an alignment module according to an exemplary embodiment of the present application.

[0113] In step S1510, at least one aligned first key frame is obtained by aligning at least one first key frame related to the current non-key frame to the current non-key frame. Specifically, the step of obtaining at least one aligned first key frame includes: performing the following operation for each first key frame to obtain an aligned first key frame: obtaining an aligned first key frame based on motion vector information of the current non-key frame relative to a first key frame. Here, the term "first key frame" is used to represent a key frame related to the current non-key frame among the extracted at least one key frame.

[0114] In an exemplary embodiment of the present application, the alignment module may adopt Fig.17 The alignment module shown in , in this case, Fig.16 Refine to Fig.18 shown in . Specifically, Fig.17 The alignment module shown is an alignment regression network based on motion vector information, so it is required that non-key frames have motion vector information relative to key frames. Fig.17 and Fig.19 The process of obtaining an aligned first key frame based on motion vector information of a current non-key frame relative to a first key frame is described in detail.

[0115] Fig.19 Detailed description is a flowchart illustrating a process of obtaining an aligned first key frame based on motion vector information of a current non-key frame relative to a first key frame according to an exemplary embodiment of the present application.

[0116] In step S1910, based on a mask corresponding to a first key frame and motion vector information of a current non-key frame relative to the first key frame, the masked motion vector information of the current non-key frame is obtained, that is, the masked motion vector information of the current non-key frame relative to the first key frame, wherein, as described above, the motion vector information of the current non-key frame relative to the first key frame is obtained from decoded information of decoding the video.

[0117] like Fig.18 As shown in the figure, the current non-key frame B i There are two closest key frames in different time directions, that is, the current non-key frame B i The left side of the nearest left side has been fixed keyframe P L and the current non-key frame B i The right side of the nearest right side has been fixed keyframe P R , in this application, the left side has been repaired keyframe P L and the right side has fixed keyframe P R Both may be referred to as the first key frame or as an adjacent key frame.

[0118] like Fig.18 As shown in the signal flow in the upper middle, the right-hand repaired key frame P R , current non-key frame B i Relative to the fixed keyframe P on the right R Motion vector information MV R→i , and the fixed keyframe P on the right R The corresponding mask is input to the alignment regression module above, where the motion vector information MV R→i Indicates that the key frame P has been repaired from the right R To the current non-key frame B i The motion vector defines the displacement of pixels between two frames. Here, the motion vector information MV R→i Defines the right fixed keyframe P R With the current non-key frame B i The displacement of pixels between L→i Indicates that the key frame P has been fixed from the left L To the current non-key frame B i motion vector.

[0119] like Fig.17 As shown in , the alignment regression module is based on the repaired key frame P on the right. R The corresponding mask for the current non-key frame B i Relative to the fixed keyframe P on the right R Motion vector information MVR→i Perform mask operation to obtain the current non-key frame B i Relative to the fixed keyframe P on the right R The motion vector information MV with mask R→i , the mask is the mask of the object, which represents the object area. Since the motion vector of the object area is often inaccurate, the mask operation process can eliminate the influence of the foreground area (ie, the object area) and only retain the motion vector information of the background area.

[0120] In step S1920, for the one first key frame, an affine transformation matrix is ​​extracted from the masked motion vector information.

[0121] Specifically, if Fig.17 As shown in , the alignment module is used for the repaired key frame P on the right. R From the current non-key frame B i Relative to the fixed keyframe P on the right R The motion vector information MV with mask R→i Specifically, since the motion change between frames is slight, a lightweight model can be used to extract the affine transformation matrix from the motion vector information, such as Fig.17 As shown in FIG, the motion vector information MV with mask can be coded by a lightweight model. R→i Perform a series of downsampling, convolution, average pooling and full connection operations to obtain the key frame P that will be applied to the right side. R The affine transformation matrix of .

[0122] S1930: Obtain an aligned first key frame by processing the first key frame based on an affine transformation matrix.

[0123] like Fig.17 As shown in , the alignment module aligns the right repaired key frame P based on the extracted affine transformation matrix. R Process (for example, multiply) to obtain the same value as the current non-key frame B i Right side of alignment fixed keyframe P R '.

[0124] Similarly, if Fig.18 As shown in the figure, the fixed keyframe P on the left can be L , current non-key frame B i Relative to the fixed keyframe P on the left L Motion vector information MV L→i , and the fixed keyframe P on the left L The corresponding mask is input to the alignment module below, where the motion vector information MV L→i Indicates that the key frame P has been fixed from the leftL To the current non-key frame B i The motion vector of the current non-key frame B is obtained by i Right side of alignment fixed keyframe P R ' is similar to the process, the alignment module below can obtain the current non-key frame B according to the similar process i Aligned left fixed keyframe P L '.

[0125] In another exemplary embodiment of the present application, the alignment module may not use Fig.17 Instead of using the alignment module shown in , the traditional alignment method or the pixel-based depth alignment model is used to align the left repaired key frame P L and the right side has fixed keyframe P R Align to current non-keyframe B i , and then get the current non-key frame B i Aligned left fixed keyframe P L ' and the current non-key frame B i Right side of alignment fixed keyframe P R ', which will not be described in detail here, and any method that can align one frame to another frame can be applied to the alignment module of the present application.

[0126] Return to reference Fig.15 In step S1520, the current non-key frame is repaired based on the at least one aligned first key frame. Fig.16 and Fig. 20 Describe this.

[0127] Fig. 20 is a flowchart illustrating a process of repairing a current non-key frame based on at least one aligned first key frame according to an exemplary embodiment of the present application.

[0128] In step S2010, based on the current non-key frame and the corresponding mask, the thirteenth feature of the current non-key frame is obtained.

[0129] Specifically, step S2010 may include: fusing the current non-key frame with a mask corresponding to the current non-key frame to obtain the fused current non-key frame, and encoding the fused current non-key frame to obtain the thirteenth feature of the current non-key frame. Fig.16 or Fig.18 As shown in the figure, the current non-key frame B i Before encoding features through the encoder, it is necessary to use the current non-key frame B i The mask of the current non-key frame B iA mask operation (i.e., fusion processing) is performed to remove the object area to be removed, thereby obtaining the fused current non-key frame. Thereafter, the fused current non-key frame B is fused by using a shared lightweight encoder. i Encode and extract the current non-key frame B i Features F i , which is the thirteenth characteristic.

[0130] In step S2020, a fourteenth feature of each of the at least one aligned first key frame is obtained.

[0131] Specifically, Fig.16 As shown in FIG, the light-weight encoder shared with the current non-key frame B can be used to respectively i Aligned left fixed keyframe P L ' and the current non-key frame B i Right side of alignment fixed keyframe P R 'Encode to extract the aligned left fixed keyframe P L 'Feature F L and aligned right side fixed keyframe P R 'Feature F R , that is, the fourteenth feature of each of the two aligned key frames is obtained.

[0132] In step S2030, the thirteenth feature and at least one fourteenth feature of the at least one aligned first key frame are fused to obtain the fifteenth feature of the current non-key frame. Fig.21a This is described in detail.

[0133] Fig.21a is a flowchart illustrating a process of fusing a thirteenth feature with at least one fourteenth feature of the at least one aligned first key frame to obtain a fifteenth feature of a current non-key frame according to an exemplary embodiment of the present application.

[0134] In step S2110, at least one similarity matrix is ​​obtained based on the thirteenth feature of the current non-key frame and the at least one fourteenth feature.

[0135] like Figure 21b As shown in , assuming that the current non-key frame B i There are two key frames closest to it in time, that is, the current non-key frame B i The left side of the nearest left side has been fixed keyframe P L and the current non-key frame B i The right side of the nearest right side has been fixed keyframe P R , accordingly, through the above reference Fig.16The fused current non-key frame B can be obtained by i The Thirteenth Feature F i , aligned left fixed keyframe P L 'The fourteenth characteristic F L and aligned right side fixed keyframe P R 'The fourteenth characteristic F R ,These features can be input into the temporal attention fusion module.

[0136] First, the thirteenth feature Fi of the current non-key frame after fusion and the aligned left repaired key frame P are calculated respectively. L 'The fourteenth characteristic F L The similarity matrix M between L , and the thirteenth feature Fi of the current non-key frame after fusion and the aligned right repaired key frame P R 'The fourteenth characteristic F R The similarity matrix M between R .

[0137] In step S2120, at least one weight matrix is ​​obtained based on the at least one similarity matrix.

[0138] Specifically, based on the two similarity matrices M obtained L and M R , respectively obtain the similarity matrix M L and M R The corresponding weight matrix AT R and AT L Specifically, based on the at least one similarity matrix, the step of obtaining at least one weight matrix includes: obtaining at least one pooled similarity matrix by performing channel dimension average pooling on each of the at least one similarity matrix; processing the at least one pooled similarity matrix based on time position information related to the at least one aligned key frame; and normalizing the processed at least one pooled similarity matrix to obtain at least one weight matrix.

[0139] like Figure 21b As shown in , first, by respectively obtaining the two similarity matrices M L and M R Perform average pooling in the channel dimension to obtain the similarity matrix M L The corresponding pooled similarity matrix A L And the similarity matrix M R The corresponding pooled similarity matrix A R , where A L and A RThe feature sizes of are all 1×1×C, where C represents the number of channels. Specifically, here, C represents the dimensions of AL and AR respectively.

[0140] Then, based on the aligned left fixed keyframe P L 'Corresponding time position information T L For the pooled similarity matrix A L Processing (for example, enhancement processing) is performed, and based on the aligned right repaired key frame P R 'Corresponding time position information T R For the pooled similarity matrix A R Processing (for example, enhancement processing) is performed to obtain the processed pooled similarity matrix A L ' and A R Specifically, the time position information T L Add to the pooled similarity matrix A L Each element in , so that the pooling similarity matrix A L The temporal representation is obtained to characterize the temporal continuity of video restoration. Similarly, the temporal position information T R Add to the pooled similarity matrix A R Each element in , so that the pooling similarity matrix A R Get tense expression. Fig. 22 As shown in the figure, the key frame P on the right has been fixed R and the fixed keyframe P on the left L are two adjacent key frames, both of which serve as reference frames for the current non-key frame Bi. L The position of distance Bi relative to P in time R Closer, therefore, the left side has been modified keyframe P L The impact on non-key frames Bi is greater, so the corresponding weight w in its weight matrix li Should be larger than the fixed keyframe P on the right R The corresponding weight w in the weight matrix ri , the temporal position information can be used to learn this temporal representation. In other words, the temporal position information can be used to learn the temporal relationship between key frames and non-key frames. Therefore, the temporal position information T L and T R The initial value of can be obtained from the fixed key frame P on the left L With the fixed keyframe P on the right R The number of non-key frames between and the time position of the current non-key frame is determined. For example, assuming that the number of these non-key frames is 10 and the current non-key frame B i It is the fixed keyframe P on the left. LThe second non-key frame on the right (i.e. the time position of the current non-key frame is 2), then T L It can be 2 / 10=0.2, T R It is 1-0.2=0.8, but this is only an example and the present application is not limited thereto.

[0141] After that, the Softmax activation function is used to process the pooled similarity matrix A in the channel dimension. L ' and A R 'Normalize and obtain the normalized weight matrix AT R and AT L Specifically, A L ' and A R ', normalize the two elements at each corresponding position in, for example, assuming A L ' and A R The elements at the first position in ' are 0.3 and 0.9 respectively. By normalizing these two values, we can get 0.3 / (0.3+0.9)=0.25 and 0.9 / (0.3+0.9)=0.75, which are used as A L ' and A R 'The normalized element (i.e. weight) at the first position in .

[0142] In step S2130, based on the at least one weight matrix and the at least one fourteenth feature, at least one sixteenth feature is obtained. Specifically, based on each of the at least one weight matrix, a corresponding one of the at least one fourteenth feature is weighted to obtain at least one sixteenth feature. Figure 21b As shown in , based on the normalized weight matrix AT R Fixed keyframe P on the right side of the alignment R 'The fourteenth characteristic F R Weighted, the sixteenth feature AF on the right is obtained R , similarly, based on the normalized weight matrix AT L Fixed keyframe P on the left side of the alignment L 'The fourteenth characteristic F L Weighted, the sixteenth feature AF on the left is obtained L .

[0143] In step S2140, the fifteenth feature of the current non-key frame is obtained by fusing the at least one sixteenth feature and the thirteenth feature.

[0144] like Figure 21b As shown in i The Thirteenth Feature F i With the sixteenth feature AF on the rightR and the sixteenth feature AF on the left L Perform feature fusion to obtain the current non-key frame B i The fifteenth feature is the feature map after time position enhancement.

[0145] In this application, the temporal attention fusion module uses the temporal position information in the above process to adjust the importance of key frames to non-key frames, enhance the perception of the temporal dimension, ensure the continuity of the temporal dimension, ensure the temporal consistency of video restoration, and solve the frame discontinuity problem that occurs when non-key frames are repaired.

[0146] Return to reference Fig. 20 In step S2040, based on the fifteenth feature, the restored current non-key frame is obtained. Specifically, the restored current non-key frame is obtained by feature decoding the fifteenth feature of the current non-key frame. Fig.16 As shown in , the decoder is used to perform feature decoding on the fifteenth feature of the current non-key frame to obtain the restored current non-key frame.

[0147] In the above referenced drawings (eg Figures 16 to 22 ), it is assumed that the current non-key frame Bi has two key frames, that is, the right side has been repaired key frame P R and the fixed keyframe P on the left L , but the present application is not limited thereto, for example, Figure 5 In the example shown in , in the process of extracting at least one key frame and at least one non-key frame from a video, at least one key frame among the key frames extracted according to the decoded information can be set as a non-key frame, that is, some P frames are set as non-key frames. Different from B frames as non-key frames, the motion vector of P frames is unidirectional. Therefore, when repairing non-key frames of P frame type, Fig.16 In the above example, only one alignment module will be operated, while the other alignment module will not be operated. Similarly, when repairing non-key frames of P frame type, Fig.18 There is only one alignment module in the operation, and the temporal position information is not used for enhancement processing in the temporal attention fusion module, that is, it is necessary to Figure 21b The structure of is simplified to obtain Fig.23 The simplified temporal attention fusion module shown in . Fig.23 In the example, only the aligned left fixed keyframe P is calculated. L 'The fourteenth characteristic F L The thirteenth feature F of the current non-key frame after fusion i Then, the similarity matrix A is obtained by performing average pooling on the channel dimension of the similarity matrix. L, after that, directly use the pooled similarity matrix A L With the fourteenth characteristic F L Multiply to get the sixteenth feature AF on the left L Finally, the thirteenth feature F of the current non-key frame after fusion i With the sixteenth feature AF on the left L Perform feature fusion. Figure 1 In the described method executed by an electronic device, after at least one key frame is repaired through step S120, step S130 is performed, that is, at least one non-key frame is repaired based on the at least one repaired key frame, but the present application is not limited to this. The method may also include: after repairing the at least one key frame through step S120, displaying a video containing the at least one repaired key frame, and receiving a continue repair instruction or a re-repair instruction input by the user. That is, a video consisting of at least one repaired key frame is displayed, and the user previews the video and confirms whether the repair result of the key frame meets the requirements. If the requirements are met, the user inputs a continue repair instruction, and thereafter, the method repairs at least one non-key frame based on the at least one repaired key frame according to the continue repair instruction, and displays a video containing at least one repaired key frame and at least one repaired non-key frame; if the requirements are not met, the user inputs a re-repair instruction, and thereafter, the method can re-execute step S110 according to the re-repair instruction, that is, re-extract at least one key frame and at least one non-key frame. At this time, at least one key frame and at least one non-key frame can be extracted in the same or different manner as the last time, and then subsequent steps S120 and so on are executed sequentially.

[0148] Fig.24 is a flowchart illustrating a method performed by an electronic device according to another exemplary embodiment of the present application.

[0149] In step S2401, a target repair area in a first video is determined. Specifically, the first video may be a video to be processed, and the target repair area may be determined in the first video according to a user selection. For example, the user may select the target repair area in the displayed first video.

[0150] Although Fig.24 Although not shown in the figure, the method may further include: after determining the target repair area, deleting frames that do not include the target repair area from the first video. Through this processing, the number of frames to be processed can be reduced and the operation efficiency can be improved.

[0151] In step S2402, the target repair area in the key frame of the first video is repaired using the first neural network. Specifically, the key frame can be a key frame as described above. Figure 1 The extraction method of step S110 is determined, and the above reference Figure 8 The spatiotemporal memory Transformer module described above is used to perform the repair process of the target repair area in the key frame. Since these contents have been described in detail above, they will not be repeated here.

[0152] In step S2403, a second video including the repaired key frames is displayed. Here, the displayed second video may be a video including only the repaired key frames, so that the user can easily determine whether the repair results of these key frames are satisfactory.

[0153] In step S2404, a continue repair instruction input by the user is received. Specifically, the user may input the continue repair instruction after browsing the second video containing the repaired key frame, for example, if the user is satisfied with the repair result of the key frame, the continue repair instruction may be input.

[0154] In step S2405, according to the continuing repair instruction, the target repair area in the non-key frame of the first video is repaired using the second neural network based on the repaired key frame. Fig.15 The described process is used to repair the target repair area in the non-key frame, and as mentioned above, the repair process can be implemented by the fast matching convolutional neural network proposed in the present application. Since these contents have been described in detail above, they will not be repeated here.

[0155] Although Fig.24 Although not shown, the method may further include: after displaying the second video, receiving a re-repair instruction input by a user; redetermining the target repair area in the first video according to the re-repair instruction; and using the first neural network to repair the re-determined target repair area in the key frame of the first video.

[0156] Specifically, as described above in step S2404, the user can browse the second video containing the repaired key frame and input a continue repair instruction, but the user can also input a re-repair instruction. For example, if the user is not satisfied with the repair result of the key frame, the user can input a re-repair instruction. In this case, the target repair area in the first video can be re-determined according to the re-repair instruction, that is, the target repair area is re-determined in the first video according to the user's selection, and the re-determined target repair area in the key frame of the first video is repaired using the first neural network. Thereafter, the video containing the repaired key frame can be displayed, and the user can browse the video to determine whether he is satisfied with the repair result. If satisfied, subsequent processing can be performed. If not satisfied, the process can be repeated until the user is satisfied.

[0157] In addition, the method may also include: after displaying the second video, receiving setting instructions input by the user for setting other target repair areas; determining the other target repair areas in the repaired key frame according to the setting instructions; and using the first neural network to repair the other target repair areas in the repaired key frame.

[0158] Specifically, as described above in step S2404, the user can browse the second video containing the repaired key frame and input a continue repair instruction, but the user can also input a setting instruction for setting other target repair areas. For example, although the user is satisfied with the current repair result of the key frame, the user wants to further repair other target areas. At this time, the user can input a setting instruction for setting other target repair areas. In this case, other target repair areas can be further determined in the repaired key frame according to the setting instruction, that is, other target repair areas are further determined in the repaired key frame according to the user's selection, and the other target repair areas in the repaired key frame are repaired using the first neural network. After that, the video containing the repaired key frame can be displayed, and the user can browse the video to determine whether he is satisfied with the repair result. If satisfied, subsequent processing can be performed. If not satisfied, the process can be repeated until the user is satisfied.

[0159] In step S2406, the restored first video is displayed. Specifically, the restored first video may be a video including restored key frames and restored non-key frames.

[0160] The following will refer to Fig.25a and Fig.25b To describe in detail Fig.24 An example of a method performed by an electronic device is shown.

[0161] Fig.25a is a flowchart illustrating an example of applying a method performed by an electronic device according to an exemplary embodiment of the present application.

[0162] like Fig.25a As shown in , in step S2410, a video is played on the display, and the object to be removed is set according to the user's selection, that is, the target area to be removed is selected in the displayed video according to the user's selection.

[0163] In step S2420, at least one key frame and at least one non-key frame (for example, step S110) are extracted from the video, and the at least one key frame is repaired according to the above method proposed in the present application (for example, step S120), that is, the object to be removed in the at least one key frame is removed using the spatiotemporal memory Transformer module, and at least one key frame from which the object to be removed is removed is displayed on the display.

[0164] In step S2430, it is determined whether the removal effect of the object to be removed in at least one key frame meets the user's requirements according to the user input. According to this step, the key frame to be repaired, that is, the key frame with the object to be removed removed, can be quickly displayed to the user.

[0165] If it is determined in step S2430 that the removal effect of the object to be removed in at least one key frame does not meet the user requirements, that is, a re-repair instruction input by the user is received, then return to step S2410, reset the object to be removed, that is, redetermine the object to be removed in the video (that is, the target repair area), and proceed to step S2420.

[0166] If it is determined in step S2430 that although the removal effect of the object to be removed in at least one key frame meets the user requirements, a removal request for other objects (i.e., other target repair areas) is received from the user when playing at least one key frame after the repair, that is, a setting instruction input by the user for setting other target repair areas is received, then in step S2440, at least one key frame after the repair is played and other objects to be removed are further set in at least one key frame after the repair according to the user's selection, and then the process returns to step S2420, and the other objects to be removed are further removed on at least one key frame where object removal was previously performed, and then the process proceeds to step S2430 until the user is satisfied.

[0167] If it is determined in step S2430 that the removal effect of the object to be removed in at least one key frame meets the user's requirements, that is, a user input instruction to continue repairing is received, then the process proceeds to step S2450, and non-key frames are repaired based on at least one repaired key frame, that is, all objects to be removed are removed, for example, the fast matching convolutional neural network proposed in this application is used to remove objects from the remaining frames. Finally, in step S2460, the repaired video is output.

[0168] Fig.25b is a flowchart illustrating another example of applying a method performed by an electronic device according to an exemplary embodiment of the present application.

[0169] like Fig.25bAs shown in , in step S2510, a video is played on the display, and the object to be removed is set according to the user's selection, that is, the target area to be removed is selected in the displayed video according to the user's selection.

[0170] In step S2520, the video is preprocessed to delete frames that do not contain the object to be removed, that is, frames that do not include the target repair area are deleted from the video to be processed. Through this preprocessing, the number of frames to be processed can be reduced and the operating efficiency can be improved.

[0171] In step S2530, at least one key frame and at least one non-key frame (for example, step S110) are extracted from the preprocessed video, and the at least one key frame is repaired according to the above method proposed in the present application (for example, step S120), that is, the object to be removed in the at least one key frame is removed using the spatiotemporal memory Transformer module, and at least one key frame from which the object to be removed is removed is displayed on the display.

[0172] In step S2540, it is determined whether the removal effect of the object to be removed in at least one key frame meets the user's requirements (ie, whether the repair result is satisfactory) according to the user input.

[0173] If it is determined in step S2540 that the removal effect of the object to be removed in at least one key frame does not meet the user requirements, that is, a re-repair instruction input by the user is received, then return to step S2510, reset the object to be removed, that is, redetermine the object to be removed in the video (that is, the target repair area), and proceed to step S2520.

[0174] If it is determined in step S2540 that although the removal effect of the object to be removed in at least one key frame meets the user's requirements, a removal request for other objects is received from the user when playing at least one repaired key frame, then in step S2550, at least one repaired key frame is played and other objects to be removed are further set in at least one repaired key frame according to the user's selection, that is, the setting instruction input by the user for setting other target repair areas is received, and then the process returns to step S2530, and the other objects to be removed are further removed on at least one key frame where object removal was previously performed, and then the process proceeds to step S2540 until the user is satisfied.

[0175] If it is determined in step S2540 that the removal effect of the object to be removed in at least one key frame meets the user's requirements, that is, a user input instruction to continue repairing is received, then the process proceeds to step S2560, and non-key frames are repaired based on at least one repaired key frame, that is, objects to be removed in all non-key frames are removed, for example, by using the fast matching convolutional neural network proposed in this application to remove objects from the remaining frames. Finally, in step S2570, the repaired video is output.

[0176] Fig.26 is a block diagram illustrating an electronic device 2600 according to an exemplary embodiment of the present application.

[0177] like Fig.26 As shown in , the electronic device 2600 includes at least one processor 2610 and at least one memory 2620, wherein the at least one memory 2620 can store computer executable instructions, and when the computer executable instructions are executed by the at least one processor 2610, they prompt the at least one processor 2610 to execute the method performed by the electronic device as described above.

[0178] The method proposed in the present application and performed by an electronic device realizes object removal in a video through two stages. First, in the first stage, the objects to be removed in the key frames are removed to obtain the repaired key frames, that is, a small number of key frames are removed by the spatiotemporal memory Transformer module; then, in the second stage, the non-key frames are repaired according to the repaired key frames obtained in the first stage, that is, the objects are removed from the non-key frames by fast matching convolutional neural networks. This can improve the quality of image restoration and solve the problems of low efficiency, excessive resource consumption, slow processing speed, difficulty in processing long videos and unstable completion effects in the prior art. At least one of the above multiple modules can be implemented by an AI model. Functions associated with AI can be executed by non-volatile memory, volatile memory and processor.

[0179] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instruction (or instruction set) individually or jointly. The electronic device may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote interface (e.g., via wireless transmission). The processor may include one or more processors. At this time, one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., a processor only for graphics (e.g., a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor (e.g., a neural processing unit (NPU)). One or more processors control the processing of input data according to predefined operating rules or AI models stored in non-volatile memory and volatile memory. The predefined operating rules or AI models may be provided by training or learning. Here, providing by learning means that by applying a learning algorithm to multiple learning data, a predefined operating rule or AI model with desired characteristics is formed. Learning can be performed in the device itself that performs AI according to the embodiment, and / or can be implemented by a separate server / device / system.

[0180] A learning algorithm is a method of using multiple learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0181] The AI ​​model can be obtained through training. Here, "obtained through training" means training a basic AI model with multiple training data through a training algorithm to obtain a predefined operating rule or AI model, which is configured to perform the desired features (or purposes).

[0182] As an example, the AI ​​model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0183] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0184] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0185] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0186] According to an embodiment of the present application, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the above-mentioned method performed by an electronic device according to the exemplary embodiment of the present application. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The instructions or computer programs in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0187] It should be noted that the terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0188] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless there is a clear explanation in this article, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. In different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0189] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A method performed by an electronic device, comprising: Extracting at least one key frame and at least one non-key frame from the video; Repairing the at least one key frame based on at least one mask corresponding to the at least one key frame; The at least one non-key frame is repaired based on the at least one repaired key frame.

2. The method of claim 1, wherein: The step of extracting at least one key frame and at least one non-key frame from the video comprises: The at least one key frame and the at least one non-key frame are extracted from the video according to the decoded information of the video.

3. The method of claim 2, wherein: The step of extracting the at least one key frame from the video according to the decoded information of the video comprises: Extracting at least one frame from the video according to the decoded information of the video; The at least one key frame is determined from the extracted frames based on a predetermined time interval.

4. The method of claim 1, wherein: The step of repairing the at least one key frame based on at least one mask corresponding to the at least one key frame comprises: repairing each group of key frames in the at least one key frame by: Extracting features of a set of key frames to be repaired; Based on the features of the restored key frames, processing the features of the set of key frames; Based on the processed features, decoding the set of key frames; The features of the repaired key frames include: first features related to all groups of repaired key frames and / or second features related to the last group of repaired key frames.

5. The method of claim 4, further comprising: extracting a third feature from the processed features based on semantic relevance; The third feature is fused with the stored first feature, and the fusion result is stored as the updated first feature.

6. The method of claim 4, further comprising: The second feature is updated based on the processed feature.

7. The method of claim 4, wherein: Based on the features of the restored key frames, the step of processing the features of the set of key frames comprises: Based on a set of masks corresponding to the set of key frames, features of the set of key frames are processed to obtain preliminary restoration features of the set of key frames; Based on the first feature, the preliminary restoration features of the set of key frames are processed to obtain processed preliminary restoration features; splicing the second feature with the processed preliminary repair feature; The processed features are obtained by processing the splicing results.

8. The method of claim 7, wherein: Based on the first feature, the step of processing the preliminary restoration features of the set of key frames to obtain the processed preliminary restoration features comprises: By dividing the preliminary restoration features of the group of key frames in the time dimension, a group of fourth features is obtained, wherein each fourth feature in the group of fourth features corresponds to a key frame in the group of key frames; For each fourth feature in the set of fourth features, extracting a fifth feature about a region corresponding to the object to be removed from the first feature; The processed preliminary restoration features are obtained by splicing the fifth features extracted from each fourth feature in the group of fourth features, and adding the splicing result to the preliminary restoration features of the group of key frames.

9. The method of claim 8, wherein: For each fourth feature in the set of fourth features, the step of extracting a fifth feature about an area corresponding to the object to be removed from the first feature comprises: Performing feature block tiling on a fourth feature in the group of fourth features and the first feature respectively; Calculate a similarity matrix for the tiled fourth feature and the tiled first feature; By normalizing the similarity matrix, a weight matrix is ​​obtained; A fifth feature about the area corresponding to the object to be removed is obtained according to the first feature and the weight matrix.

10. The method of claim 6, wherein: The step of updating the second feature based on the processed feature comprises: By dividing the processed features of the set of key frames in a time dimension, a set of sixth features is obtained, wherein each sixth feature in the set of sixth features corresponds to a key frame in the set of key frames; The second feature is updated by processing the set of sixth features using a neural network formed by at least one gated recurrent unit (GRU) module connected in series.

11. The method of claim 5, wherein: The step of extracting a third feature from the processed features based on semantic relevance comprises: By dividing the processed features of the group of key frames in the time dimension, a group of seventh features is obtained, wherein each seventh feature in the group of seventh features corresponds to a key frame in the group of key frames; Obtaining a set of eighth features by performing feature compression on each seventh feature in the set of seventh features in the spatial dimension; The third feature is obtained by performing feature compression on the splicing result of the set of eighth features in the time dimension.

12. The method of claim 11, wherein: The step of obtaining a set of eighth features by performing feature compression on each seventh feature in the set of seventh features in the spatial dimension includes: performing the following operation on each seventh feature to obtain an eighth feature: Performing feature block tiling on a seventh feature; Obtaining a ninth feature corresponding to the seventh feature by the following operations: for each feature block in the seventh feature, calculating a similarity matrix between the feature block and each feature block in the seventh feature, and fusing the feature blocks based on the similarity matrix; By performing full connection on the feature blocks of the ninth feature, a tenth feature corresponding to the one seventh feature is obtained; By rearranging the tenth feature, an eighth feature is obtained.

13. The method of claim 11, wherein: The step of obtaining the third feature by performing feature compression on the splicing result of the set of eighth features in the time dimension includes: Performing feature block tiling on the splicing result of the set of eighth features; The eleventh feature corresponding to the splicing result of the set of eighth features is obtained by the following operations: for each feature block in the splicing result of the set of eighth features, a similarity matrix between the feature block and each feature block in the splicing result of the set of eighth features is calculated, and the feature blocks are merged based on the similarity matrix; By performing full connection on the feature blocks of the eleventh feature, a twelfth feature corresponding to the concatenation result of the set of eighth features is obtained; The third feature is obtained by rearranging the twelfth feature.

14. The method of claim 1, wherein: The step of repairing the at least one non-key frame based on the repaired at least one key frame includes: repairing each non-key frame in the at least one non-key frame by the following operations: Obtaining at least one aligned first key frame by aligning at least one first key frame related to the current non-key frame to the current non-key frame; and The current non-key frame is repaired based on the at least one aligned first key frame.

15. The method of claim 14, wherein: The step of obtaining at least one aligned first key frame includes: performing the following operations on each first key frame to obtain an aligned first key frame: Based on the motion vector information of the current non-key frame relative to a first key frame, an aligned first key frame is obtained.

16. A method performed by an electronic device, comprising: Determine a target repair area in the first video; Using a first neural network, repairing a target repair area in a key frame of the first video; displaying a second video containing the repaired keyframes; Receive a continue repair instruction input by a user; According to the continue repair instruction, using a second neural network, repairing a target repair area in a non-key frame of the first video based on the repaired key frame; Show the first video after restoration.

17. The method of claim 16, further comprising: After displaying the second video, receiving a re-restoration instruction input by a user; Re-determining a target restoration area in the first video according to the re-restoration instruction; The re-determined target repair area in the key frame of the first video is repaired using a first neural network.

18. The method of claim 16, further comprising: After displaying the second video, receiving a setting instruction input by a user for setting other target repair areas; According to the setting instruction, determining the other target repair areas in the repaired key frame; The other target repair areas in the repaired key frame are repaired using the first neural network.

19. An electronic device comprising: at least one processor; as well as at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 18.

20. A computer-readable storage medium storing instructions, wherein: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 18.

Images