Video frame insertion method and device, equipment and storage medium

Abstract

The invention discloses a video frame insertion method and device, equipment and a storage medium. The video frame insertion method comprises the following steps: performing target feature extraction on a video potential vector to obtain a foreground feature; the video potential vector is a vector representation of a video to be subjected to frame insertion in a potential space; performing background texture extraction on the video potential vector to obtain background features; based on a time attention layer with a sliding window, carrying out feature fusion on the noise potential vector, the foreground feature and the background feature to obtain a fusion feature; the noise potential vector is a vector representation of noise in a potential space; and performing normalization processing on the fusion features to obtain a complete video vector after frame insertion. According to the technical scheme of the embodiment of the invention, high-quality video frame insertion can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video frame insertion method, device, equipment and storage medium. Background Art

[0002] During live video transmission, if the network conditions are unstable, the video frame rate is often reduced to adapt to bandwidth limitations. However, low frame rate videos have obvious differences in viewing experience, especially when the action changes quickly, the video will appear stuck and blurry.

[0003] In order to improve the viewing quality of live video recordings, it is particularly important to increase the video frame rate in subsequent processing. The interpolation methods commonly used in the industry often rely on traditional image processing technology. When these methods process complex scenes and fast movements, they are prone to problems such as blurred edges and discontinuous motion trajectories, resulting in poor video viewing. Summary of the invention

[0004] The present invention provides a video frame insertion method, device, equipment and storage medium to solve the problems of blurred edges, discontinuous motion trajectory and the like generated when performing video frame insertion based on traditional image processing technology.

[0005] According to one aspect of the present invention, a video frame insertion method is provided, comprising:

[0006] Extracting target features from a video latent vector to obtain foreground features; the video latent vector is a vector representation of the video to be inserted in the latent space;

[0007] Extracting background texture from the video potential vector to obtain background features;

[0008] Based on a temporal attention layer with a sliding window, the noise latent vector, the foreground feature and the background feature are feature-fused to obtain a fused feature; the noise latent vector is a vector representation of the noise in the latent space;

[0009] The fused features are normalized to obtain a complete video vector after frame insertion.

[0010] According to another aspect of the present invention, a video frame insertion device is provided, comprising:

[0011] A foreground feature acquisition module is used to extract target features from a video latent vector to obtain a foreground feature; the video latent vector is a vector representation of the video to be inserted in a latent space;

[0012] A background feature acquisition module, used to extract background texture from the video potential vector to obtain background features;

[0013] A fusion feature acquisition module, used for fusing the noise latent vector, the foreground feature and the background feature based on a temporal attention layer with a sliding window to obtain a fusion feature; the noise latent vector is a vector representation of noise in a latent space;

[0014] The complete video determination module is used to normalize the fusion features to obtain a complete video vector after interpolation.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video frame insertion method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video frame insertion method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the video frame insertion method of any embodiment of the present disclosure is implemented.

[0021] The technical solution of the embodiment of the present invention performs target feature extraction on the video latent vector to obtain foreground features, and performs background texture extraction on the video latent vector to obtain background features. Furthermore, based on the temporal attention layer with a sliding window, the noise latent vector, the foreground features and the background features are feature fused to obtain fused features. Finally, the fused features are normalized to obtain a complete video vector after interpolation. By using the Diffusion Transformer method to model the temporal relationship between video frames, efficient and high-quality video interpolation is achieved, thereby improving the video viewing experience.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a video frame insertion method provided according to Embodiment 1 of the present invention;

[0025] Figure 2a is a flowchart of a video frame insertion method provided according to Embodiment 2 of the present invention;

[0026] Figure 2b is a principle diagram of video frame insertion provided according to Embodiment 2 of the present invention;

[0027] Figure 2c is a flow chart of a foreground feature determination method provided in Embodiment 2 of the present invention;

[0028] Figure 2d is a flow chart of a method for determining background features according to Embodiment 2 of the present invention;

[0029] Figure 2e is a flow chart of a feature fusion method provided according to Embodiment 2 of the present invention;

[0030] Figure 3 is a structural schematic diagram of a video frame insertion device provided according to Embodiment 3 of the present invention;

[0031] Figure 4 It is a structural schematic diagram of an electronic device for implementing the video frame insertion method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 1 A flowchart of a video interpolation method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a Diffusion Transformer method is used for video interpolation. The method can be performed by a video interpolation device. The video interpolation device can be implemented in the form of hardware and / or software. The video interpolation device can be configured in various general-purpose computing devices. Figure 1 As shown, the method includes:

[0036] S110, extracting target features from the video latent vector to obtain foreground features; the video latent vector is a vector representation of the video frame to be inserted in the latent space.

[0037] The video to be inserted is a video with a frame rate lower than a set threshold. For example, the video to be inserted is a video stored during the live broadcast with a reduced frame rate due to unstable network conditions. Due to the low frame rate, the video to be inserted will be stuck and blurred during playback, affecting the user's viewing experience. Therefore, the video to be inserted needs to be inserted.

[0038] The video frame of the video to be inserted is used as the input of deep learning models such as convolutional neural networks. After network convolution and pooling operations, the video vector of the video to be inserted in the pixel space can be obtained. The DeepWalk algorithm can also be used to achieve video vectorization by learning the node vector representation in the graph.

[0039] The latent space is a multidimensional space that is convenient for data feature mining. Before inserting frames into the video to be inserted, the video vector can be converted into the latent space to obtain the video latent vector, so as to extract more video features and improve the quality of video insertion.

[0040] In the embodiment of the present invention, target feature extraction is performed on the video latent vector to obtain foreground features as a basis for subsequent video frame insertion. Specifically, target detection can be performed on the video latent vector first to identify the target boundary vector, and then target control can be performed on the video latent vector based on the target boundary vector to obtain the foreground features.

[0041] S120, extracting background texture from the video latent vector to obtain background features.

[0042] In the embodiment of the present invention, each frame of the video includes not only the target but also the background. When performing video interpolation, the foreground and the background need to be combined. After extracting the foreground features in the video latent vector, background texture extraction is performed on the video latent vector to obtain background features. Specifically, background texture extraction can be performed through a self-attention mechanism.

[0043] In a specific example, the video latent vector is simultaneously input into five self-attention layers with different window sizes, and the outputs of multiple self-attention layers are jointly input into the fully connected layer to obtain the background features of the fully connected layer output.

[0044] S130. Based on a temporal attention layer with a sliding window, the noise latent vector, foreground features, and background features are fused to obtain fused features; the noise latent vector is a vector representation of noise in the latent space.

[0045] In an embodiment of the present invention, after obtaining the foreground features and background features, the noise latent vector, the foreground features and the background features are fused based on the temporal attention layer with a sliding window to obtain a fused feature. The noise latent vector is a vector obtained by converting the original noise vector into a latent space, and the noise can be a random Gaussian noise. Specifically, the noise latent vector, the foreground features and the background features can be input into the temporal attention layer with a sliding window, and then the output of the temporal self-attention layer is input into the self-attention layer, and the output of the self-attention layer is further input into the fully connected layer to obtain a fused feature.

[0046] S140: Normalize the fused features to obtain a complete video vector after frame insertion.

[0047] In the embodiment of the present invention, the fused features are normalized to obtain a complete video vector after interpolation. Specifically, the fused features can be passed through a normalization module consisting of a Layer Normalization layer and a 3-layer Multi-Layer Perceptron (MLP) to obtain a complete video vector after interpolation.

[0048] The technical solution of the embodiment of the present invention performs target feature extraction on the video latent vector to obtain foreground features, and performs background texture extraction on the video latent vector to obtain background features. Furthermore, based on the temporal attention layer with a sliding window, the noise latent vector, the foreground features and the background features are feature fused to obtain fused features. Finally, the fused features are normalized to obtain a complete video vector after interpolation. By using the Diffusion Transformer method to model the temporal relationship between video frames, efficient and high-quality video interpolation is achieved, thereby improving the video viewing experience.

[0049] Embodiment 2

[0050] Figure 2a This is a flow chart of a video interpolation method provided in Embodiment 2 of the present invention. This embodiment further refines the above embodiment and provides specific steps of extracting target features from a video latent vector to obtain foreground features, extracting background texture from a video latent vector to obtain background features, and fusing noise latent vectors, foreground features, and background features based on a temporal attention layer with a sliding window to obtain fused features. Figure 2a As shown, the method includes:

[0051] S210 , embedding the original video vector of the video frame to be inserted in the pixel space to obtain a video latent vector in the latent space.

[0052] In the embodiment of the present invention, the principle of video frame insertion is as follows: Figure 2b As shown, first, in order to extract more video features and improve the quality of video frame insertion, the original video vector of the video to be inserted in the pixel space is embedded to obtain the video latent vector in the latent space. For example, the original video vector is X1∈R H×W×3×t , where H is the length of each video frame, W is the width of each video frame, 3 is the number of channels, and t is the number of video frames.

[0053] In a specific example, the original video vector X1∈R is trained by a variational autoencoder (VAE). H×W×3×t Perform video embedding and obtain the video latent vector X2∈R in the latent space 32 ×32×4096×t , where the number of channels changes from 3 to 4096 in the latent space.

[0054] S220, embedding the original noise vector in the pixel space into a video to obtain a noise latent vector in the latent space.

[0055] In the embodiment of the present invention, after embedding the video features to obtain the video latent vector, the original noise vector in the pixel space is also subjected to noise embedding to obtain the noise latent vector in the latent space. For example, the original noise vector is Z1∈R 32×32×4×t , where the first 32 is the length of each video frame, the second is the width of each video frame, 4 is the number of channels, and t is the number of video frames.

[0056] In a specific example, the original video vector Z1∈R is obtained by a noise embedding module, for example, the Contrastive Language-Image Pre-training (CLIP) model. 32 ×32×4×t Noise embedding is performed to obtain the noise potential vector Z2∈R in the latent space 32×32×4×t .

[0057] S230: Perform target detection on the video potential vector to obtain a target boundary vector.

[0058] In the embodiment of the present invention, in order to extract the foreground features in the video latent vector, the video latent vector is first subjected to target detection to obtain the target boundary vector. For example, the video latent vector can be passed through a target detection module, such as Yolov10, to obtain the target boundary vector X3∈R H×W×3×t×B , where B represents the bounding box coordinate information of all targets in each frame.

[0059] S240: Based on the target boundary vector, target control is performed on the video potential vector to obtain a foreground feature.

[0060] In the embodiment of the present invention, based on the target boundary vector, the video latent vector is subjected to target control to obtain the foreground feature. Specifically, the target boundary vector and the video latent vector are used as inputs of the target control module to obtain the foreground feature output by the target control module.

[0061] In a specific example, X3∈R H×W×3×t×B B (bounding box coordinate information of all objects in each frame) and the video potential vector X2∈R 32×32×4096×t Together as the input of the target control module, the foreground feature X4∈R output by the target control module is obtained 32×32×4096×t .

[0062] Optionally, based on the target boundary vector, the video potential vector is targeted and controlled to obtain foreground features, including:

[0063] Input the video latent vector into the first controlled self-attention layer to obtain a first output, and sum the first output and the video latent vector to obtain a second output;

[0064] Input the second output and the target bounding box coordinate information contained in the target boundary vector into the second control self-attention layer to obtain a third output;

[0065] The third output is input to the control fully connected layer to obtain a fourth output, and the fourth output and the second output are summed to obtain a fifth output;

[0066] The fifth output is input into the cross attention layer to obtain the sixth output, and the sixth output and the fifth output are summed to obtain the foreground feature.

[0067] The goal of target control is to maintain the overall coordination of the target in the video changes and solve the problem of consistency of the target motion trajectory. Its main components include self-attention layer, cross-attention layer and full connection layer. In this optional embodiment, a specific method for performing target control on the video potential vector based on the target boundary vector to obtain the foreground feature is provided, as follows: Figure 2c As shown in Figure 1, the video latent vector X2 is first input into the first control self-attention layer to obtain the first output V1, and the first output V1 and the video latent vector X2 are summed to obtain the second output V2. Furthermore, the second output V2 and the target boundary vector X3∈R H×W×3×t×B The target bounding box coordinate information B contained in is input to the second control self-attention layer to obtain the third output V3. Further, the third output V3 is input to the control fully connected layer to obtain the fourth output V4, and the fourth output V4 and the second output V2 are summed to obtain the fifth output V5. Finally, the fifth output V5 is input to the cross attention layer to obtain the sixth output V6, and the sixth output V6 and the fifth output V5 are summed to obtain the foreground feature X4.

[0068] The calculation formula of the second output V2 is as follows:

[0069] V2=X2+V1=X2+SelfAttn(X2)

[0070] Among them, SelfAttn represents the self-attention layer, which can enhance the feature interaction effect between video latent vectors.

[0071] The calculation formula of the fifth output V5 is as follows:

[0072] V5=V4+V2=TS(V3)+V2=TS[SelfAttn(V2,B)]+V2

[0073] Among them, TS represents the fully connected layer, which is used to improve the token screening operation of the target. The calculation formula of the foreground feature X4 is as follows:

[0074] X4=V6+V5=CrossAttn(V5)+V5

[0075] Among them, CrossAttn represents the cross attention layer, which can enhance the feature interaction effect between multiple targets.

[0076] S250, inputting the video latent vector into multiple background self-attention layers respectively, inputting the output results of each background self-attention layer into the background fully connected layer together, and obtaining the background features output by the background fully connected layer; the window sizes of the multiple background self-attention layers are different.

[0077] In the embodiment of the present invention, in order to enhance the edge details in the interpolated video, after obtaining the foreground features, it is necessary to continue to determine the background features, input the video potential vectors into multiple background self-attention layers respectively, and input the output results of each background self-attention layer into the background fully connected layer to obtain the background features output by the background fully connected layer. The window sizes of the multiple background self-attention layers are different.

[0078] In a specific example, Figure 2d As shown in Figure 1, the background texture extraction module consists of five self-attention layers with different window sizes, denoted as {Attn1, Attn2, Attn3, Attn4, Attn5} and a fully connected layer. The window sizes of the self-attention layers are {4, 8, 16, 32, 64}, and the dimensions are {4096, 2048, 1024, 512, 256}. After the video latent vector X2 passes through these five self-attention layers, five intermediate vectors are obtained. Finally, the five intermediate vectors are input into the fully connected layer to obtain a background feature X5∈R with a dimension of 4096. 32×32×4096×t , which can enhance the edge details in the interpolated video and improve the problem of unclear target edges.

[0079] S260, based on the temporal attention layer with a sliding window, the noise latent vector, the foreground feature and the background feature are feature fused to obtain a fused feature; the noise latent vector is a vector representation of the noise in the latent space.

[0080] Optionally, based on a temporal attention layer with a sliding window, the noise latent vector, foreground features, and background features are fused to obtain fused features, including:

[0081] Input the noise potential vector to the first fused self-attention layer to obtain a first noise vector, and input the first noise vector to the first fused fully connected layer to obtain a second noise vector;

[0082] The second noise vector, foreground features, and background features are input into the temporal attention layer with a sliding window to obtain the attention score of each frame in the fused feature.

[0083] The attention score of each frame is input into the second fused self-attention layer, and the output of the second fused self-attention layer is input into the second fused fully connected layer to obtain the fused features output by the second fused fully connected layer.

[0084] In this optional embodiment, after obtaining the foreground features and background features, a specific method is provided to fuse the noise latent vector, the foreground features and the background features based on a temporal attention layer with a sliding window to obtain the fused features, as shown in the following example: Figure 2e As shown: First, the noise potential vector Z2 is input into the first fusion self-attention layer to obtain the first noise vector, and the first noise vector is input into the first fusion fully connected layer Z 21 , get the second noise vector Z 22 , and then the second noise vector Z 22 , foreground feature X4 and background feature X5 are input to the temporal attention layer with sliding window to obtain the attention score of each frame in the fusion feature. Finally, the attention score of each frame is input to the second fusion self-attention layer, and the output of the second fusion self-attention layer is input to the second fusion fully connected layer to obtain the fusion feature X6∈R containing different motion trends output by the second fusion fully connected layer 32×32×4096×t .

[0085] Optionally, the second noise vector, foreground features, and background features are input into a temporal attention layer with a sliding window to obtain an attention score for each frame in the fused features, including:

[0086] The second noise vector, the foreground feature and the background feature are spliced ​​in the channel dimension to obtain a video splicing feature;

[0087] Extract one frame in the video splicing feature as the current frame in turn, and calculate the attention score of each sliding window associated with the current frame;

[0088] Perform a weighted sum of the attention scores of each sliding window associated with the current frame to obtain the attention score of the current frame, and return to execute the operation of extracting one frame in the video splicing feature as the current frame in turn until the attention score of each frame in the fusion feature is calculated;

[0089] The weight of the weighted sum is determined based on the number of windows associated with the current frame, the frame sequence number, and the frame index value at the middle position of each sliding window.

[0090] In this optional embodiment, a specific method is provided for inputting the second noise vector, the foreground feature and the background feature into a temporal attention layer with a sliding window to obtain the attention score of each frame in the fusion feature: first, the second noise vector Z 22, foreground feature X4 and background feature X5 are spliced ​​in the channel dimension to obtain the video splicing feature V. One frame is extracted from the video splicing feature V in turn as the current frame, and the attention score of each sliding window associated with the current frame is calculated. The specific calculation formula is as follows:

[0091]

[0092] in, represents the attention score of the j-th sliding window in the i-th frame, SelfAttn represents the self-attention operation, and V represents the video splicing feature.

[0093] Furthermore, the attention scores of each sliding window associated with the current frame are weighted and summed to obtain the attention score of the current frame, and the operation of extracting one frame in the video splicing feature as the current frame is returned to execute until the attention score of each frame in the fusion feature is calculated;

[0094] The weight of the weighted sum is determined based on the number of windows associated with the current frame, the frame sequence number, and the frame index value at the middle position of each sliding window.

[0095] Since each frame contains the attention scores of multiple windows, in order to achieve smooth time transition, the attention score of each frame is calculated using the following weight formula:

[0096]

[0097] in, represents the attention score of the i-th frame, represents the attention score of the jth sliding window in the i-th frame, U represents the number of windows associated with the i-th frame, and c j Indicates the frame index value at the middle position of the jth window associated with the i-th frame, 6|ic j |6 represents the distance between the i-th frame and the j-th sliding window. If the area of ​​the j-th sliding window belonging to the i-th frame exceeds the set ratio, then 6|ic j |6 has a value of 0.

[0098] S270: Normalize the fused features to obtain a complete video vector after frame insertion.

[0099] In the embodiment of the present invention, after obtaining the fusion feature of the latent space, the fusion feature is normalized to obtain a complete video vector after interpolation. 32×32×4096×t After the normalization module consisting of the LayerNormalization layer and the 3-layer Multi-Layer Perceptron (MLP), the final inserted complete video vector X6∈RH×W×3×t .

[0100] The technical solution of the embodiment of the present invention performs target detection on a video latent vector to obtain a target boundary vector, and based on the target boundary vector, performs target control on the video latent vector to obtain a foreground feature, inputs the video latent vector into multiple background self-attention layers respectively, and inputs the output results of each background self-attention layer into the background fully connected layer to obtain the background features output by the background fully connected layer, and based on the time attention layer with a sliding window, performs feature fusion on the noise latent vector, the foreground feature and the background feature to obtain the fused feature, and finally normalizes the fused feature to obtain a complete video vector after interpolation. The video interpolation solution based on the Diffusion Transformer method can adapt to video content of different scenes and motion speeds, and improve the video frame rate while ensuring the video quality.

[0101] Embodiment 3

[0102] Figure 3 FIG. 1 is a schematic diagram of the structure of a video frame insertion device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0103] The foreground feature acquisition module 310 is used to extract target features from the video latent vector to obtain foreground features; the video latent vector is a vector representation of the video to be inserted in the latent space;

[0104] A background feature acquisition module 320, used to extract background texture from the video latent vector to obtain background features;

[0105] A fusion feature acquisition module 330 is used to perform feature fusion on the noise latent vector, the foreground feature and the background feature based on a temporal attention layer with a sliding window to obtain a fusion feature; the noise latent vector is a vector representation of noise in a latent space;

[0106] The complete video determination module 340 is used to normalize the fused features to obtain a complete video vector after interpolation.

[0107] The technical solution of the embodiment of the present invention performs target feature extraction on the video latent vector to obtain foreground features, and performs background texture extraction on the video latent vector to obtain background features. Furthermore, based on the temporal attention layer with a sliding window, the noise latent vector, the foreground features and the background features are feature fused to obtain fused features. Finally, the fused features are normalized to obtain a complete video vector after interpolation. By using the Diffusion Transformer method to model the temporal relationship between video frames, efficient and high-quality video interpolation is achieved, thereby improving the video viewing experience.

[0108] Optionally, the fusion feature acquisition module 330 includes:

[0109] A second noise vector determination unit is used to input the noise potential vector into the first fused self-attention layer to obtain a first noise vector, and input the first noise vector into the first fused fully connected layer to obtain a second noise vector;

[0110] a frame attention score determination unit, configured to input the second noise vector, the foreground feature, and the background feature into a temporal attention layer with a sliding window to obtain an attention score for each frame in the fused feature;

[0111] A fusion feature acquisition unit is used to input the attention score of each frame into a second fusion self-attention layer, and input the output of the second fusion self-attention layer into a second fusion fully connected layer to obtain the fusion feature output by the second fusion fully connected layer.

[0112] Optionally, a frame attention score determination unit is used to:

[0113] splicing the second noise vector, the foreground feature and the background feature in a channel dimension to obtain a video splicing feature;

[0114] Extracting one frame in the video splicing feature in turn as the current frame, and calculating the attention score of each sliding window associated with the current frame;

[0115] Performing a weighted summation on the attention scores of each sliding window associated with the current frame to obtain the attention score of the current frame, and returning to execute the operation of sequentially extracting one frame in the video splicing feature as the current frame until the attention score of each frame in the fusion feature is calculated;

[0116] The weight of the weighted sum is determined based on the number of windows associated with the current frame, the frame sequence number, and the frame index value at the middle position of each sliding window.

[0117] Optionally, the foreground feature acquisition module 310 includes:

[0118] A boundary vector determination unit, configured to perform target detection on the video potential vector to obtain a target boundary vector;

[0119] The foreground feature acquisition unit is used to perform target control on the video potential vector based on the target boundary vector to obtain the foreground feature.

[0120] Optionally, the foreground feature acquisition unit is specifically used to:

[0121] Inputting the video latent vector into a first control self-attention layer to obtain a first output, and summing the first output and the video latent vector to obtain a second output;

[0122] Inputting the second output and the target bounding box coordinate information contained in the target bounding vector into a second control self-attention layer to obtain a third output;

[0123] Input the third output to the control fully connected layer to obtain a fourth output, and sum the fourth output and the second output to obtain a fifth output;

[0124] The fifth output is input into the cross attention layer to obtain a sixth output, and the sixth output and the fifth output are summed to obtain the foreground feature.

[0125] Optionally, the background feature acquisition module 320 is specifically used for:

[0126] Inputting the video latent vectors into multiple background self-attention layers respectively, and inputting the output results of each background self-attention layer into the background fully connected layer together to obtain the background features output by the background fully connected layer;

[0127] The window sizes of the multiple background self-attention layers are different.

[0128] Optionally, the video frame insertion device further includes:

[0129] A video latent vector determination module is used to perform video embedding on the original video vector of the video to be inserted in the pixel space before extracting the target feature of the video latent vector to obtain the foreground feature, so as to obtain the video latent vector in the latent space;

[0130] The noise latent vector determination module is used to embed the original noise vector in the pixel space into the video to obtain the noise latent vector in the latent space.

[0131] The video frame insertion device provided in the embodiment of the present invention can execute the video frame insertion method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0132] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0133] Embodiment 4

[0134] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0135] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, applicators, blade applicators, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0136] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0137] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0138] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a video interpolation method.

[0139] In some embodiments, the video interpolation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video interpolation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video interpolation method in any other appropriate manner (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or application.

[0142] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein may be implemented in a computing system that includes a backend component (e.g., as a data applier), or a computing system that includes a middleware component (e.g., an application applier), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0145] A computing system may include a client and an application server. The client and the application server are generally remote from each other and usually interact through a communication network. The relationship between the client and the application server is generated by computer programs running on corresponding computers and having a client-application server relationship with each other. The application server may be a cloud application server, also known as a cloud computing application server or a cloud host, which is a host product in a cloud computing application system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS applications.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0147] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video frame insertion method, It is characterized in that include: Extract target features from the video latent vector to obtain foreground features; The video latent vector is a vector representation of the video frame to be inserted in the latent space; Extracting background texture from the video potential vector to obtain background features; Based on a temporal attention layer with a sliding window, the noise latent vector, the foreground feature and the background feature are subjected to feature fusion to obtain a fused feature; The noise latent vector is a vector representation of noise in the latent space; The fused features are normalized to obtain a complete video vector after frame insertion.

2. The method according to claim 1, It is characterized in that Based on the temporal attention layer with a sliding window, the noise potential vector, the foreground feature and the background feature are fused to obtain fused features, including: Inputting the noise potential vector into a first fused self-attention layer to obtain a first noise vector, and inputting the first noise vector into a first fused fully connected layer to obtain a second noise vector; Inputting the second noise vector, the foreground feature and the background feature into a temporal attention layer with a sliding window to obtain an attention score for each frame in the fused feature; The attention score of each frame is input into the second fused self-attention layer, and the output of the second fused self-attention layer is input into the second fused fully connected layer to obtain the fused features output by the second fused fully connected layer.

3. The method according to claim 2, It is characterized in that The second noise vector, the foreground feature and the background feature are inputted into a temporal attention layer with a sliding window to obtain an attention score of each frame in the fused feature, including: splicing the second noise vector, the foreground feature and the background feature in a channel dimension to obtain a video splicing feature; Extracting one frame in the video splicing feature in turn as the current frame, and calculating the attention score of each sliding window associated with the current frame; Performing a weighted summation on the attention scores of each sliding window associated with the current frame to obtain the attention score of the current frame, and returning to execute the operation of sequentially extracting one frame in the video splicing feature as the current frame until the attention score of each frame in the fusion feature is calculated; The weight of the weighted sum is determined based on the number of windows associated with the current frame, the frame sequence number, and the frame index value at the middle position of each sliding window.

4. The method according to claim 1, It is characterized in that Target features are extracted from the video potential vector to obtain foreground features, including: Performing target detection on the video potential vector to obtain a target boundary vector; Based on the target boundary vector, the video potential vector is subjected to target control to obtain the foreground feature.

5. The method according to claim 4, It is characterized in that Based on the target boundary vector, the video potential vector is subjected to target control to obtain the foreground feature, including: Inputting the video latent vector into a first control self-attention layer to obtain a first output, and summing the first output and the video latent vector to obtain a second output; Inputting the second output and the target bounding box coordinate information contained in the target bounding vector into a second control self-attention layer to obtain a third output; Input the third output to the control fully connected layer to obtain a fourth output, and sum the fourth output and the second output to obtain a fifth output; The fifth output is input into the cross attention layer to obtain a sixth output, and the sixth output and the fifth output are summed to obtain the foreground feature.

6. The method according to claim 1, It is characterized in that Performing background texture extraction on the video potential vector to obtain background features includes: Inputting the video latent vectors into multiple background self-attention layers respectively, and inputting the output results of each background self-attention layer into the background fully connected layer together to obtain the background features output by the background fully connected layer; The window sizes of the multiple background self-attention layers are different.

7. The method according to claim 1, It is characterized in that Before extracting target features from the video latent vector and obtaining foreground features, the following steps are also included: Embedding the original video vector of the video frame to be inserted in the pixel space to obtain a video latent vector in the latent space; The original noise vector in the pixel space is embedded in the video to obtain the noise latent vector in the latent space.

8. A video frame insertion device, It is characterized in that include: A foreground feature acquisition module is used to extract target features from the video potential vector to obtain foreground features; The video latent vector is a vector representation of the video frame to be inserted in the latent space; A background feature acquisition module, used to extract background texture from the video potential vector to obtain background features; A fusion feature acquisition module, used for fusing the noise potential vector, the foreground feature and the background feature based on a temporal attention layer with a sliding window to obtain a fusion feature; The noise latent vector is a vector representation of noise in the latent space; The complete video determination module is used to normalize the fusion features to obtain a complete video vector after interpolation.

9. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video frame insertion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video frame insertion method according to any one of claims 1 to 7 when executed.

Images