Video processing method and apparatus, storage medium, and device

Abstract

The present disclosure provides a video processing method and device, a storage medium and equipment, relates to the technical field of video processing, can acquire a plurality of continuous video frames in a video, determine the score of a first video frame in the plurality of continuous video frames based on the plurality of continuous video frames, the score is positively correlated with the probability of the first video frame being reconstructable, and by deleting the video frame with a score greater than or equal to a preset threshold from the video frame, not only the data amount of the video can be compressed and the transmission speed of the video can be improved, but also the deleted video frame can be reconstructed through the frame interpolation technology, and the frame interpolation quality is improved.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, storage medium, and device. Background Technology

[0002] In video compression and transmission, to reduce data volume and improve encoding efficiency, a combination of frame extraction and frame interpolation is typically used. Specifically, during the encoding stage, partial video frames are extracted to reduce the amount of data to be encoded, thus achieving compression. During the decoding stage, frame interpolation is used to reconstruct the missing frames, restoring the original frame rate and visual smoothness of the video. However, frame interpolation often results in a decrease in frame quality. Summary of the Invention

[0003] The purpose of this disclosure is to provide a video processing method, apparatus, storage medium, and device to improve frame interpolation quality.

[0004] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0005] According to a first aspect of this disclosure, a video processing method is provided, comprising: acquiring a plurality of consecutive video frames in a video; determining a score of a first video frame among the plurality of consecutive video frames based on the plurality of consecutive video frames, wherein the score is positively correlated with the probability that the first video frame can be reconstructed; and deleting the first video frame from the video in response to the score being greater than or equal to a preset threshold.

[0006] In some exemplary embodiments of this disclosure, determining the score of the first video frame among the plurality of consecutive video frames based on the plurality of consecutive video frames includes: performing a splicing process on the plurality of consecutive video frames in the channel dimension to obtain a multi-channel tensor; and determining the score of the first video frame based on the multi-channel tensor.

[0007] In some exemplary embodiments of this disclosure, determining the score of the first video frame based on the multi-channel tensor includes: inputting the multi-channel tensor into a preset evaluation model, and outputting the score of the first video frame through the evaluation model.

[0008] In some exemplary embodiments of this disclosure, the plurality of consecutive video frames includes three consecutive video frames, wherein the first video frame refers to the middle frame of the three consecutive video frames.

[0009] In some exemplary embodiments of this disclosure, the evaluation model includes a first network and a second network. The first network is used to extract image features from the plurality of consecutive video frames, and the second network is used to predict the score of a first video frame among the plurality of consecutive video frames based on the image features. The evaluation model is trained by a target base model. The target base model includes the first network, the second network, and a third network. The third network is used to predict the optical flow between adjacent video frames based on the image features extracted by the first network. The parameters of the first network and the second network are adjusted based on the difference between the score predicted by the second network and the actual score, and the difference between the optical flow predicted by the third network and the actual optical flow.

[0010] In some exemplary embodiments of this disclosure, the true score is determined based on the difference between the predicted video frame and the true video frame, wherein the predicted video frame is obtained by predicting the true video frame based on the frame preceding and following the true video frame.

[0011] In some exemplary embodiments of this disclosure, the real optical flow is predicted when predicting the real video frames.

[0012] According to a second aspect of this disclosure, a video processing apparatus is provided, comprising: an acquisition module for acquiring a plurality of consecutive video frames in a video; a determination module for determining a score of a first video frame among the plurality of consecutive video frames based on the plurality of consecutive video frames, wherein the score is positively correlated with the probability that the first video frame can be reconstructed; and a processing module for deleting the first video frame from the video in response to the score being greater than or equal to a preset threshold.

[0013] In some implementations, a determining module is used to stitch together the plurality of consecutive video frames in the channel dimension to obtain a multi-channel tensor; and to determine the score of the first video frame based on the multi-channel tensor.

[0014] In some implementations, a determining module is used to input the multi-channel tensor into a preset evaluation model and output a score for the first video frame through the evaluation model.

[0015] In some implementations, the plurality of consecutive video frames includes three consecutive video frames, wherein the first video frame refers to the middle frame of the three consecutive video frames.

[0016] In some implementations, the evaluation model includes a first network and a second network. The first network is used to extract image features from the plurality of consecutive video frames, and the second network is used to predict a score for a first video frame among the plurality of consecutive video frames based on the image features. The evaluation model is trained using a target base model. The target base model includes the first network, the second network, and a third network. The third network is used to predict optical flow between adjacent video frames based on the image features extracted by the first network. The parameters of the first network and the second network are adjusted based on the difference between the score predicted by the second network and the actual score, and the difference between the optical flow predicted by the third network and the actual optical flow.

[0017] In some implementations, the true score is determined based on the difference between the predicted video frame and the true video frame, wherein the predicted video frame is obtained by predicting the true video frame based on the frame preceding and following the true video frame.

[0018] In some implementations, the true optical flow is predicted when predicting the true video frames.

[0019] According to a third aspect of this disclosure, an electronic device is provided, including a processor and a memory, the memory being used to store executable instructions of the processor; wherein the processor is configured to perform the method of the first aspect described above by executing the executable instructions.

[0020] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect described above.

[0021] The video processing method, apparatus, storage medium, and device provided in this disclosure can acquire multiple consecutive video frames in a video. Based on the multiple consecutive video frames, a score of the first video frame among the multiple consecutive video frames is determined. The score is positively correlated with the probability that the first video frame can be reconstructed. By deleting video frames with scores greater than or equal to a preset threshold from the video frames, not only can the amount of video data be compressed and the video transmission speed be improved, but the deleted video frames can also be reconstructed through frame interpolation technology, thereby improving the quality of frame interpolation.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A flowchart of a video processing method according to an embodiment of this disclosure is shown;

[0025] Figure 2 This is a schematic diagram of a scoring method provided in an embodiment of this disclosure;

[0026] Figure 3 This is a schematic diagram of another scoring method provided in an embodiment of this disclosure;

[0027] Figure 4 A flowchart illustrating a method for determining a score for a video frame according to an embodiment of this disclosure is shown;

[0028] Figure 5 This is a schematic diagram of a model structure provided in an embodiment of this disclosure;

[0029] Figure 6 This is a schematic diagram of a basic model provided in an embodiment of this disclosure;

[0030] Figure 7 This diagram illustrates a video processing apparatus according to an embodiment of the present disclosure;

[0031] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0033] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] Frame extraction methods in the video encoding stage mainly include fixed-ratio frame extraction and dynamic frame extraction based on inter-frame differences. Fixed-ratio frame extraction is simple and efficient, but it cannot adapt to dynamic changes in video content, which can easily lead to the accidental deletion of keyframes and affect video continuity. Dynamic frame extraction based on inter-frame differences, motion information, etc., while having a certain degree of content adaptability, still has limitations. These limitations mainly lie in the fact that it does not consider the predictability of the extracted video frames and is not designed in conjunction with the subsequent frame interpolation process, which can easily lead to a decrease in reconstruction quality. It is difficult to balance compression efficiency and frame interpolation quality, thus limiting the performance of video processing.

[0035] To address the shortcomings of related technologies, this disclosure provides a video processing method, apparatus, storage medium, and device. By scoring the reconstructability of video frames contained in a video and deleting video frames with scores greater than or equal to a preset threshold, it is possible not only to reduce the amount of video data and improve the speed of video transmission, but also to ensure that the deleted video frames can be reconstructed, thereby improving the quality of frame interpolation.

[0036] The following examples illustrate the solutions provided in this disclosure.

[0037] Figure 1 A flowchart of a video processing method according to an embodiment of this disclosure is shown. This method can be executored by an electronic device. The electronic device can be understood as any device with video processing capabilities, such as a laptop, tablet, mobile phone, desktop computer, cloud server, distributed computing node, etc., but is not limited to the devices listed herein. Figure 1 As shown, in some embodiments, the video processing method provided in this disclosure may include the following steps.

[0038] In step S101, multiple consecutive video frames are acquired from the video.

[0039] In this embodiment of the disclosure, the video can be any type of video, such as short videos, long videos (e.g., surveillance videos, TV series, movies, documentaries, variety shows), etc.

[0040] In some implementations, the number of video frames included in the plurality of consecutive video frames referred to in this disclosure can be arbitrary, and this disclosure does not specifically limit this. However, for ease of understanding, the plurality of consecutive video frames can be exemplarily understood as three consecutive video frames in this disclosure.

[0041] In step S103, based on multiple consecutive video frames, the score of the first video frame among the multiple consecutive video frames is determined, and the score of the first video frame is positively correlated with the probability that the first video frame can be reconstructed.

[0042] In this embodiment of the disclosure, the first video frame can be understood as any one of a plurality of consecutive video frames. For example, when the plurality of consecutive video frames includes three consecutive video frames, the first video frame can be understood as the first frame (also called the preceding frame), the second frame (also called the middle frame, intermediate frame, or middle one frame), or the third frame (also called the following frame) of the three consecutive video frames. For ease of understanding, in this embodiment of the disclosure, the first video frame can be understood as the middle frame of three consecutive video frames.

[0043] In the embodiments of this disclosure, there are various methods for determining the score of the first video frame among multiple consecutive video frames. Several exemplary methods are illustrated below. However, it should be understood that other methods may also be used to determine the score of the first video frame in other embodiments of this disclosure.

[0044] Method 1: In some exemplary implementations, video frames other than the first video frame from multiple consecutive video frames can be input into a preset frame interpolation network. The frame interpolation network predicts the first video frame from the multiple consecutive video frames. The video frames predicted by the frame interpolation network are compared with the first video frame, and the similarity between the predicted video frames and the first video frame is determined as the score of the first video frame. The frame interpolation network can be, for example, TOFlow, IFRNet, or EMA-VFI. TOFlow is an optical flow-based video frame interpolation algorithm whose model structure includes three parts: optical flow estimation, optical flow transformation, and image processing. In video frame interpolation tasks, the number of input frames is usually 3. The optical flow estimation module uses a pre-trained SpyNet (Spatial Pyramid Network) as its core to obtain optical flow information between adjacent frames; the optical flow transformation module aligns the input frames to the reference frames based on the optical flow information; the image processing module uses a ResNet (Residual Network) structure to process the transformed image into the final frame interpolation result. IFRNet is an efficient encoder-decoder based network used for fast synthesis of intermediate frames. EMA-VFI addresses the characteristics of video frame interpolation tasks by utilizing an inter-frame attention mechanism to simultaneously extract motion and appearance information. It employs a hybrid CNN (Convolutional Neural Network) and Transformer architecture to balance interpolation performance and computational load. Through the inter-frame attention mechanism, EMA-VFI can explicitly extract motion information and the appearance information that has interacted between two frames simultaneously, and then perform frame interpolation based on these two pieces of information. For the frame interpolation methods and principles based on TOFlow, IFRNet, and EMA-VFI, please refer to the relevant technical documentation; they will not be elaborated upon here.

[0045] For example, Figure 2 This is a schematic diagram of a scoring method provided in an embodiment of this disclosure, such as... Figure 2 As shown, video frames I1, I2, and I3 are three consecutive video frames. I2 is the intermediate frame. In some examples, I1 and I3 can be input into a pre-defined frame interpolation network, which predicts I2 to obtain I21. The similarity D1 between I21 and I2 is calculated and determined as the score of I2. A higher score indicates that I2 is easier to reconstruct, while a lower score indicates that I2 is more difficult to reconstruct.

[0046] Method 2 involves stitching together multiple consecutive video frames along the channel dimension to obtain a multi-channel tensor, and then determining the score of the first video frame based on this multi-channel tensor. The multi-channel tensor obtained by stitching together multiple consecutive video frames can also be understood as a real multi-channel tensor obtained by stitching together actual video frames.

[0047] In some examples, determining the score of the first video frame based on a multi-channel tensor can be understood as follows: All video frames except the first one from multiple consecutive video frames are input into a pre-defined interpolation network. The interpolation network predicts the first video frame, and the predicted video frame is concatenated with the input video frame in the channel dimension to obtain the predicted multi-channel tensor. The similarity between the predicted multi-channel tensor and the actual multi-channel tensor is used to determine the score of the first video frame. The method for concatenating video frames in the channel dimension can be found in relevant technologies and will not be elaborated here. Similarly, the method for calculating the similarity between two multi-channel tensors can also be found in relevant technologies and will not be elaborated here.

[0048] For example, Figure 3 This is a schematic diagram of another scoring method provided in an embodiment of this disclosure, such as... Figure 3 As shown, video frames I1, I2, and I3 are three consecutive video frames, with I2 being the intermediate frame. In some implementations, I1, I2, and I3 can be concatenated along the channel dimension to obtain a multi-channel tensor Z1. Video frames I1 and I3 are input into a preset frame interpolation network, which predicts I2 to obtain I21. By concatenating I1, I21, and I3 along the channel dimension, a multi-channel tensor Z2 is obtained. The similarity D2 between Z1 and Z2 can then be used as the score for I2. The higher the similarity between Z1 and Z2, the greater the probability that I2 can be reconstructed, meaning I2 is easier to reconstruct; conversely, the lower the similarity between Z1 and Z2, the lower the probability that I2 can be reconstructed, meaning I2 is less likely to be reconstructed.

[0049] By stitching together multiple consecutive video frames along the channel dimension, the contextual information of multiple consecutive video frames can be fully integrated, which helps to accurately assess the reconstructability of video frames in multiple consecutive video frames.

[0050] Of course, the above are just examples and not the only limitations.

[0051] In step S105, in response to the score of the first video frame being greater than or equal to a preset threshold, the first video frame is deleted from the video.

[0052] Specifically, when the score of the first video frame is greater than or equal to a preset threshold, the probability of the first video frame being reconstructed is relatively high, and the first video frame can be deleted from the video. When the score of the first video frame is less than the preset threshold, the probability of the first video frame being reconstructed is relatively low, and the first video frame needs to be retained in the video.

[0053] according to Figures 1-3In this embodiment, multiple consecutive video frames can be acquired from a video. Based on these multiple consecutive video frames, a score for the first video frame is determined. This score is positively correlated with the probability that the first video frame can be reconstructed. By deleting video frames with scores greater than or equal to a preset threshold from the video frames, not only can the amount of video data be compressed and the video transmission speed be improved, but the deleted video frames can also be reconstructed using frame interpolation technology, thereby improving the quality of frame interpolation.

[0054] Figure 4 A flowchart illustrating a method for determining a score for a video frame according to an embodiment of this disclosure is shown. Figure 4 As shown, in some implementations, multiple consecutive video frames can be stitched together along the channel dimension to obtain a multi-channel tensor, and the score of the first video frame among the multiple consecutive video frames can be determined based on the multi-channel tensor. The method for determining the score of the first video frame among multiple consecutive video frames based on the multi-channel tensor may include the following steps:

[0055] In step S401, the multi-channel tensor is input into a preset evaluation model, and the score of the first video frame is output through the evaluation model.

[0056] For example, in some implementations, the evaluation model described in this disclosure may include a first network and a second network. The first network is used to extract image features (e.g., feature maps, but not limited to feature maps) from multiple consecutive video frames based on a multi-channel tensor obtained by stitching together multiple consecutive video frames. These image features include temporal information. The second network is used to predict a score for a first video frame among the multiple consecutive video frames based on the image features extracted by the first network. This score is positively correlated with the probability that the first video frame can be reconstructed. A higher score indicates a higher probability that the first video frame can be reconstructed; a lower score indicates a lower probability that the first video frame can be reconstructed.

[0057] In some implementations, the first network may include downsampling and convolution. Downsampling is used to downsample the input multichannel tensor. Convolution is used to convolve the downsampling result to obtain image features containing temporal information.

[0058] In some implementations, the second network includes an average pooling layer, at least one fully connected layer, and an activation function. The average pooling layer is used to compress image features to a fixed size. The fully connected layer is used to perform non-linear transformations and dimensionality adjustments on the compressed image features, and the activation function is used to output a score for the first video frame.

[0059] For example, Figure 5 This is a schematic diagram of a model structure provided in an embodiment of this disclosure, such as... Figure 5As shown, in some examples, the first network of the scoring model may include a downsampling network layer, two convolutional layers with a stride of 2, and eight convolutional layers with a stride of 1. The downsampling network layer performs downsampling operations on the input multi-channel tensor. The downsampling operation can be, for example, bilinear interpolation downsampling, but is not limited to bilinear interpolation downsampling. The convolutional layers with a stride of 2 are used for downsampling and expanding the receptive field. The convolutional layers with a stride of 1 are used to extract deep semantic features. The second network includes an average pooling layer, a fully connected layer FC1, a fully connected layer FC2, and a sigmoid activation function. The average pooling layer is used to compress the features extracted by the two convolutional layers with a stride of 2 and the features extracted by the eight convolutional layers with a stride of 1 into a fixed size. FC1 and FC2 are used to perform non-linear transformations and dimensionality adjustments on the compressed features, and the sigmoid activation function is used to output the score of the first video frame. Of course, Figure 5 This is merely an illustrative example of the evaluation model, not the only one.

[0060] In some implementations, the evaluation model described in the embodiments of this disclosure can be obtained by training a target base model.

[0061] In some implementations, the target base model may include a third network, as well as the first and second networks referred to in the evaluation model above. The third network is used to predict the optical flow between adjacent video frames based on image features extracted by the first network, such as the optical flow between the first video frame and the preceding video frame, and the optical flow between the first video frame and the following video frame.

[0062] The training method for the target base model will be explained below with examples.

[0063] Example, Figure 6 This is a schematic diagram of a basic model provided in an embodiment of this disclosure. Figure 6 As shown, in some implementations, the first network of the target base model may include downsampling network layers, two convolutional layers with a stride of 2, and eight convolutional layers with a stride of 1. The second network includes average pooling layers, fully connected layers FC1 and FC2, and a sigmoid activation function. The third network includes transposed convolutional layers and upsampling layers. See also Figure 6 In some implementations, the training method for the target base model may include the following steps.

[0064] In step S601, training data pairs are constructed.

[0065] Taking three video frames as an example, in some examples, three consecutive video frames can be extracted from a preset video dataset and denoted as the previous frame. medium frame , the next frame The three video frames are concatenated along the channel dimension to form a 9-channel input tensor. .

[0066] In step S603, the true score and true optical flow are obtained.

[0067] The preceding and following frames are input into a pre-defined interpolation network (such as RIFE (Real-Time Intermediate Stream Estimation)) to generate predicted intermediate frames. And calculate the predicted frames Compared to real medium frames Image quality metrics (such as PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), or LPIPS (Learned Perceptual Image Patch Similarity)) are used as mid-frames after normalizing the image quality metric values. Real ratings Simultaneously, the optical flow predicted by the frame interpolation network is extracted as the real optical flow. During the training phase, the real optical flow serves as a supervisory signal to guide the target base model in learning more time-aware feature representations. In other words, in some examples, the true score can be determined based on the difference between the predicted video frame and the real video frame, and the predicted video frame can be obtained by predicting the real video frame based on the frame preceding and following it. The real optical flow can be predicted when predicting the real video frame.

[0068] In step S605, feature extraction and dimensionality reduction encoding are performed.

[0069] The input tensor X is fed into the target base model, and first undergoes bilinear downsampling to the original value. Figure 1 The size is reduced to 1 / 4, and then further reduced to 1 / 16 through two convolutional layers with a stride of 2, expanding the number of channels to 240. Subsequently, deep semantic feature extraction is performed through eight convolutional layers with a stride of 1.

[0070] In step S607, the second network outputs the predicted score.

[0071] The features obtained after convolutional processing by two convolutional layers with a stride of 2, and the features obtained after convolutional processing by eight convolutional layers with a stride of 1, are fed into the second network, and then sequentially pass through global average pooling and two fully connected layers. Figure 6 (FC1 and FC2 in the frame), output by the sigmoid activation function. rating .

[0072] In step S609, the third network outputs the predicted optical flow.

[0073] The features obtained after convolution processing through two convolutional layers with a stride of 2, and the features obtained after convolution processing through eight convolutional layers with a stride of 1, are fed into the third network. The optical flow from the previous frame to the middle frame and from the next frame to the middle frame are predicted through transposed convolutional layers and upsampling layers.

[0074] In step S611, the scoring loss is calculated.

[0075] The regression loss between the predicted score and the true score is calculated using the mean squared error (MSELoss). The loss function takes the following form:

[0076] (1)

[0077] in, To predict the score of the mid-frame, This represents the true score for the mid-frame. N is the number of samples; in this example, N is 3.

[0078] In step S613, optical flow loss is calculated.

[0079] The difference between the predicted optical flow and the actual optical flow is measured using the Endpoint Error (EPE) method, and the optical flow loss is calculated. The expression for calculating optical flow loss is as follows:

[0080] (2)

[0081] in, This represents the vector of optical flow predicted by the third network. This represents the vector of the actual optical flow. N is the number of samples; in this example, N is 3.

[0082] In step S615, the total loss function is calculated.

[0083] The total loss function is obtained by weighted summation of the scoring loss and optical flow loss. The expression is as follows:

[0084] (3)

[0085] in, and is the weighting coefficient, used to balance the contributions of the two losses during training. and The value can be set as needed.

[0086] In step S617, backpropagation and parameter update are performed.

[0087] Using standard stochastic gradient descent (SGD) or Adam (Adaptive Moment Estimation) optimizers, based on the total loss... Backpropagation and weight updates are performed to adjust the parameters of the first and second networks. In the total loss... Upon convergence, training stops, and the final parameters of the first and second networks are obtained. In other words, in some examples, the parameters of the first and second networks can be adjusted based on the difference between the score predicted by the second network and the actual score, and the difference between the optical flow predicted by the third network and the actual optical flow. The methods for backpropagation and weight updates based on standard stochastic gradient descent (SGD) or Adam (Adaptive Moment Estimation) optimizers can be found in related techniques and will not be elaborated upon here.

[0088] After training is completed, the use of the third network can be turned off or the trained model can be pruned to remove the third network from the trained model, thus obtaining the scoring model as described in the embodiments of this disclosure.

[0089] By using a third network to assist the training of the first and second networks, the first network can extract more time-aware feature representations, thereby improving the accuracy of the second network's scoring.

[0090] It should be noted that when the number of video frames included in the multiple consecutive video frames mentioned in this embodiment is greater than 3, the multiple consecutive video frames can be divided into multiple groups of consecutive video frames, with each group including three video frames. In this case, an evaluation model can be used to evaluate the score of the intermediate frame for each group.

[0091] Furthermore, although the above embodiments are illustrated using the example of determining the score of the middle frame of three consecutive video frames, it is understood that in other embodiments, the scoring model can also be trained to determine the score of the preceding or following frame other than the middle frame in three consecutive video frames. The training method is similar and will not be described in detail here.

[0092] according to Figure 5 and Figure 6 In this example, by introducing a scoring model to rate the reconstructability of video frames, the decision on whether to retain video frames can be aided, avoiding accidental deletion of keyframes and improving the quality of frame interpolation. By training the first and second networks in the third network-assisted scoring model, the semantic awareness capability of the first network can be enhanced, and the scoring accuracy of the second network can be improved.

[0093] Figure 7 A schematic diagram of a video processing apparatus according to an embodiment of the present disclosure is shown. Figure 7 As shown, in some embodiments, the video processing apparatus 700 may include:

[0094] The acquisition module 701 is used to acquire multiple consecutive video frames in a video.

[0095] The determining module 702 is used to determine a score of the first video frame among the plurality of consecutive video frames based on the plurality of consecutive video frames, wherein the score is positively correlated with the probability that the first video frame can be reconstructed.

[0096] Processing module 703 is used to delete the first video frame from the video in response to the score being greater than or equal to a preset threshold.

[0097] In some implementations, the determining module 702 is used to perform splicing processing on the plurality of consecutive video frames in the channel dimension to obtain a multi-channel tensor; and to determine the score of the first video frame based on the multi-channel tensor.

[0098] In some implementations, the determining module 702 is used to input the multi-channel tensor into a preset evaluation model and output a score for the first video frame through the evaluation model.

[0099] In some implementations, the plurality of consecutive video frames includes three consecutive video frames, wherein the first video frame refers to the middle frame of the three consecutive video frames.

[0100] In some implementations, the evaluation model includes a first network and a second network. The first network is used to extract image features from the plurality of consecutive video frames, and the second network is used to predict a score for a first video frame among the plurality of consecutive video frames based on the image features. The evaluation model is trained using a target base model. The target base model includes the first network, the second network, and a third network. The third network is used to predict optical flow between adjacent video frames based on the image features extracted by the first network. The parameters of the first network and the second network are adjusted based on the difference between the score predicted by the second network and the actual score, and the difference between the optical flow predicted by the third network and the actual optical flow.

[0101] In some implementations, the true score is determined based on the difference between the predicted video frame and the true video frame, wherein the predicted video frame is obtained by predicting the true video frame based on the frame preceding and following the true video frame.

[0102] In some implementations, the true optical flow is predicted when predicting the true video frames.

[0103] Figure 7 The execution method and beneficial effects of the video processing device in the embodiment can be found in [reference needed]. Figures 1-6 Any of the embodiments described herein will not be repeated here.

[0104] In some embodiments, this disclosure also provides an electronic device including a processor and a memory, the memory being used to store executable instructions of the processor; wherein the processor is configured to perform the method in any of the above method embodiments by executing the executable instructions.

[0105] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Referring below... Figure 8 To describe an electronic device 800 according to this embodiment of the present invention. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0106] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810 (included in one or more processors), at least one storage unit 820 (included in one or more memories), and a bus 830 connecting different system components (including storage unit 820 and processing unit 810).

[0107] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention.

[0108] Storage unit 820 may include readable media in the form of volatile storage units, such as random access memory (RAM) 821 and / or cache memory 822, and may further include read-only memory (ROM) 823.

[0109] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0110] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0111] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0113] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0114] A program product for implementing the above-described method according to embodiments of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0117] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0118] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0119] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0120] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0121] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A video processing method, characterized in that, include: To capture multiple consecutive video frames from a video; Based on the plurality of consecutive video frames, a score is determined for the first video frame among the plurality of consecutive video frames, and the score is positively correlated with the probability that the first video frame can be reconstructed; The score of the first video frame is determined by a preset evaluation model, which includes a first network and a second network. The first network is used to extract image features from the plurality of consecutive video frames, and the second network is used to predict the score of the first video frame among the plurality of consecutive video frames based on the image features. The evaluation model is obtained by training the target base model; The target base model includes the first network, the second network, and the third network, wherein the third network is used to predict the optical flow between adjacent video frames based on the image features extracted by the first network. The parameters of the first network and the second network are adjusted based on the difference between the score predicted by the second network and the actual score, and the difference between the optical flow predicted by the third network and the actual optical flow. In response to the score being greater than or equal to a preset threshold, the first video frame is deleted from the video.

2. The method according to claim 1, characterized in that, Determining the score of the first video frame among the plurality of consecutive video frames based on the plurality of consecutive video frames includes: The multiple consecutive video frames are stitched together along the channel dimension to obtain a multi-channel tensor. The score of the first video frame is determined based on the multi-channel tensor.

3. The method according to claim 2, characterized in that, Determining the score of the first video frame based on the multi-channel tensor includes: The multi-channel tensor is input into a preset evaluation model, and the score of the first video frame is output through the evaluation model.

4. The method according to any one of claims 1-3, characterized in that, The plurality of consecutive video frames includes three consecutive video frames, wherein the first video frame refers to the middle frame of the three consecutive video frames.

5. The method according to claim 1, characterized in that, The true score is determined based on the difference between the predicted video frame and the true video frame. The predicted video frame is obtained by predicting the true video frame based on the frame before and the frame after the true video frame.

6. The method according to claim 5, characterized in that, The true optical flow is predicted when predicting the true video frames.

7. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire multiple consecutive video frames in a video. A determination module is used to determine the score of a first video frame among a plurality of consecutive video frames based on the plurality of consecutive video frames. The score is positively correlated with the probability that the first video frame can be reconstructed. The score of the first video frame is determined by a preset evaluation model. The evaluation model includes a first network and a second network. The first network is used to extract image features from the plurality of consecutive video frames, and the second network is used to predict the score of the first video frame among the plurality of consecutive video frames based on the image features. The evaluation model is trained by a target base model. The target base model includes the first network, the second network, and a third network. The third network is used to predict the optical flow between adjacent video frames based on the image features extracted by the first network. The parameters of the first network and the second network are adjusted based on the difference between the score predicted by the second network and the actual score, and the difference between the optical flow predicted by the third network and the actual optical flow. A processing module is configured to delete the first video frame from the video in response to the score being greater than or equal to a preset threshold.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method described in any one of claims 1 to 6.

Images