Video frame acquisition method and device, electronic equipment and storage medium

Abstract

The invention provides a video frame acquisition method, and the method comprises the steps: obtaining video frame prediction information which comprises an input video frame and time information corresponding to a to-be-acquired target video frame; acquiring a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model; wherein when the preset video frame acquisition model acquires the target video frame, the feature channel information is superposed; the feature channel information is used for representing a time relationship between the target video frame and the input video frame; the feature channel information is determined according to time information set for the input video frame and time information corresponding to the target video frame to be acquired. According to the method, the target video frame can be obtained at any moment through a single model.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer vision, and in particular to a method, device, electronic device and storage medium for acquiring a video frame. Background Art

[0002] At present, deep learning algorithms have achieved fruitful results in the field of video interpolation and video prediction. In particular, pixel synthesis based on optical flow is the mainstream algorithm in the field of video interpolation and prediction. The algorithm is mainly divided into two steps: the first step is to estimate the optical flow between the input frame and the target frame; the second step is to use the optical flow to guide the synthesis of the intermediate frame.

[0003] Video interpolation refers to inputting consecutive frames and synthesizing any intermediate frames that do not exist between the consecutive frames. Video prediction refers to inputting two consecutive frames and predicting any frames before and after the two consecutive frames.

[0004] In the related art, the video interpolation and prediction methods based on neural networks usually use different models to achieve the acquisition of video interpolation and the prediction of video frames. No single model can support both the acquisition of video interpolation and the prediction of video frames at any time. In other words, no model can achieve the acquisition of target video frames at any time. Summary of the invention

[0005] In view of this, the present application provides a video frame acquisition method, device, electronic device and storage medium, which can achieve target video frame acquisition at any time through a single model.

[0006] To solve the above technical problems, the technical solution of this application is implemented as follows:

[0007] In one embodiment, a video frame acquisition method is provided, the method comprising:

[0008] Acquire video frame prediction information, wherein the video frame prediction information includes: time information corresponding to the input video frame and the target video frame to be acquired;

[0009] Acquire a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model;

[0010] Among them, the preset video frame acquisition model superimposes feature channel information when acquiring the target video frame; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined based on the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

[0011] The input video frame includes: a first video frame and a second video frame; the feature channel information is determined according to the time information set for the input video frame and the time information corresponding to the target video frame to be acquired, including:

[0012] The characteristic channel information is determined as:

[0013] 1<t, 0<t<1, or t<0;

[0014] Among them, 0 is the time information set for the first video frame, 1 is the time information set for the second video frame, and t is the time information corresponding to the target video frame to be obtained.

[0015] The input video frame includes: a first video frame and a second video frame; the step of acquiring a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model includes:

[0016] The preset video frame acquisition model predicts the bidirectional optical flow between the first video frame and the second video frame based on the first video frame and the second video frame;

[0017] Obtaining a first optical flow from the first video frame to the target video frame and a second optical flow from the second video frame to the target video frame by using a linear weighted method based on the bidirectional optical flow;

[0018] Perform forward mapping based on the first optical flow, the second optical flow, the first video frame, the second video frame, and the CNN features of the first video frame and the CNN features of the second video frame to obtain a mapped first mapped video frame and a second mapped video frame;

[0019] Video synthesis is performed based on the first mapped video frame, the second mapped video frame, the CNN features of the first mapped video frame, the CNN features of the second mapped video frame and the feature channel information to obtain and output the target video frame.

[0020] The bidirectional optical flow includes a forward optical flow and a reverse optical flow; the forward optical flow is the optical flow from the first video frame to the second video frame; the reverse optical flow is the optical flow from the second video frame to the first video frame;

[0021] The step of obtaining a first optical flow from the first video frame to the target video frame and a second optical flow from the second video frame to the target video frame by using a linear weighted method based on the bidirectional optical flow includes:

[0022] Determine the first optical flow as the product of the forward optical flow and t;

[0023] The second optical flow is determined as the product of the reverse optical flow and 1-t.

[0024] The acquisition of the preset video frame acquisition model includes:

[0025] Establish an initial video frame acquisition model;

[0026] Acquire training samples; the training samples include multiple groups of triplets, each of which is three consecutive video frames; any one of the frames is used as a target video frame, and the other two frames are used as input video frames;

[0027] The initial video frame acquisition model is trained using the training sample to obtain a preset video frame acquisition model.

[0028] Wherein, after acquiring the video frame prediction information and before acquiring the target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model, the method further includes:

[0029] Determine whether t is less than -1 or greater than 2. If so, use an iterative method to determine the target video frame; otherwise, execute the method of obtaining the target video frame corresponding to the video frame prediction information based on the preset video frame acquisition model.

[0030] In another embodiment, a video frame acquisition device is provided, the device comprising:

[0031] A storage unit, used for storing a preset video frame acquisition model;

[0032] A first acquisition unit is used to acquire video frame prediction information, wherein the video frame prediction information includes: time information corresponding to the input video frame and the target video frame to be acquired;

[0033] A second acquisition unit is used to acquire a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model; wherein, when acquiring the target video frame, the preset video frame acquisition model superimposes feature channel information; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined based on the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

[0034] in,

[0035] The input video frame includes: a first video frame and a second video frame; the feature channel information is:

[0036] 1<t, 0<t<1, or t<0;

[0037] Among them, 0 is the time information set for the first video frame, 1 is the time information set for the second video frame, and t is the time information corresponding to the target video frame to be obtained.

[0038] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a video frame acquisition method when executing the program.

[0039] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, a video frame acquisition method is implemented.

[0040] As can be seen from the above technical solution, in the above embodiment, by superimposing feature channel information in the established preset video frame acquisition model, wherein the feature channel information is used to characterize the temporal relationship between the target video frame and the input video frame, a video interpolation frame or a predicted frame at any time corresponding to the input video frame is acquired. This solution can achieve the acquisition of the target video frame at any time through a single model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a process for obtaining a preset video frame acquisition model in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a preset video frame acquisition model structure in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a process of obtaining a target video frame by a preset video frame acquisition model in an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of a video frame acquisition process in an embodiment of the present application;

[0046] Figure 5 This is another schematic diagram of a video frame acquisition process in an embodiment of the present application;

[0047] Figure 6 This is a schematic diagram of the structure of a video frame acquisition device in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe the order or precedence of the objects. It should be understood that the data used in this way can be interchangeable where appropriate, so that the implementation of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0052] In the related art, the video interpolation and prediction methods based on neural networks usually implement the acquisition of video interpolation frames and the prediction of video frames through different models.

[0053] Such as video prediction based on forward-warping, and video interpolation and video prediction based on backward-warping.

[0054] In the related art, for a network model for video interpolation, if you want to perform video frame prediction, you need to modify the input and output pairs during the training phase; however, although these implementation solutions are relatively convenient when building the model, it is impossible to train a single model to support both video interpolation and video frame prediction tasks at the same time.

[0055] Currently, many solutions support video insertion at any time, but there is no solution that supports video prediction at any time.

[0056] Based on the above technical problems, a video frame acquisition method is proposed in an embodiment of the present application, which acquires a video interpolation frame or a predicted frame at any time corresponding to the input video frame by superimposing feature channel information in an established preset video frame acquisition model, wherein the feature channel information is used to characterize the temporal relationship between the target video frame and the input video frame. This solution can achieve the acquisition of the target video frame at any time through a single model.

[0057] The following is a detailed description of the process of acquiring any video frame in accordance with an embodiment of the present application in conjunction with the accompanying drawings. Before acquiring any video frame, the acquisition of a preset video frame acquisition model is first described.

[0058] In the embodiment of the present application, frame-0, frame-1, and frame-t are used to uniformly represent the input video frame and the target frame; wherein the input video frame includes two frames, namely the first video frame and the second video frame; frame-0 represents the first video frame with time information of 0, frame-1 represents the second video frame with time information of 1, and frame-t represents the target video frame with time information of t. In the preset video frame acquisition model and in the process of determining the time relationship, the time information of the first video frame and the second video frame are determined in this default manner, so that the time information t is a relative time information relative to the time information 0 and the time information 1.

[0059] When t is less than 0, it means predicting the video frame before frame-0. When t is greater than 0 and less than 1, it means predicting the interpolated frame between frame-0 and frame-1. When t is greater than 1, it means predicting the video frame after frame-1.

[0060] See also Figure 1 , Figure 1 This is a flow chart of obtaining a preset video frame acquisition model in an embodiment of the present application. The specific steps are:

[0061] Step 101, establishing an initial video frame acquisition model.

[0062] Step 102, obtaining training samples; the training samples include multiple groups of triplets, each triplet is three consecutive video frames; any one of the frames is used as a target video frame, and the other two frames are used as input video frames.

[0063] If video frame 1, video frame 2 and video frame 3 are three consecutive video frames, video frame 1 and video frame 3 can be selected as input frames, and video frame 2 can be selected as the target video frame. In this case, the target video frame is the video interpolation frame.

[0064] If video frame 2 and video frame 3 can be selected as input frames, video frame 1 is used as the target video frame, and the target video frame at this time is the predicted video frame.

[0065] When step 101 and step 102 are implemented specifically, the execution order thereof is not limited.

[0066] Step 103: Use the training samples to train the initial video frame acquisition model to obtain a preset video frame acquisition model.

[0067] During model training, the model can be trained using the gradient descent method. The target video frame output by the initial video frame acquisition model is compared with the corresponding target video frame in the sample. When the preset conditions are met, the training of the initial video frame acquisition model is terminated to obtain the preset video frame acquisition model.

[0068] See also Figure 2 , Figure 2 A schematic diagram of the preset video frame acquisition model structure in an embodiment of the present application. Figure 2 It includes: input module, optical flow acquisition module, linear weighting module, CNN feature acquisition module, mapping module, synthesis module and output module.

[0069] Combine the following Figure 2 and Figure 3 The preset video frame acquisition model is described in detail.

[0070] See also Figure 3 , Figure 3 This is a flow chart of the preset video frame acquisition model acquiring the target video frame in the embodiment of the present application. The specific steps are:

[0071] Step 301 : predicting a bidirectional optical flow between a first video frame and a second video frame.

[0072] The bidirectional optical flow includes forward optical flow and reverse optical flow; the forward optical flow is the optical flow from the first video frame to the second video frame; the reverse optical flow is the optical flow from the second video frame to the first video frame;

[0073] This step is done by Figure 2 The optical flow acquisition module in is implemented; the optical flow acquisition module can be implemented by any optical flow model in the existing implementation, and this is not limited in the embodiments of the present application.

[0074] The first video frame, the second video frame, and the time information t used later are all input through the input module.

[0075] Step 302: A first optical flow from the first video frame to the target video frame and a second optical flow from the second video frame to the target video frame are obtained by using a linear weighted method based on the bidirectional optical flow.

[0076] The specific implementation of using the linear weighted method to obtain the first optical flow and the second optical flow in this step is:

[0077] The first optical flow is determined as the product of the forward optical flow and t, which can be expressed as:

[0078] flow 0→t =t*flow 0→1 ;

[0079] Among them, flow 0→t represents the first optical flow, t is the time information of the target video frame, flow 0→1 represents the forward optical flow;

[0080] The second optical flow is determined as the product of the reverse optical flow and 1-t, which can be expressed as:

[0081] flow 1→t =(1-t)*flow 1→0 ;

[0082] Among them, flow 1→t represents the second optical flow; t is the time information of the target video frame, flow 1→0 Represents reverse optical flow.

[0083] This step is done by Figure 2 The linear weighting module in is implemented; the linear weighting module extends the linear scaling method and obtains the optical flow from the input frame to the target frame by the bidirectional flow between frames in a linear weighted manner.

[0084] Step 303, forward mapping is performed based on the first optical flow, the second optical flow, the first video frame, the second video frame, and the CNN features of the first video frame and the CNN features of the second video frame to obtain the mapped first mapped video frame and the second mapped video frame.

[0085] In the present application, the Figure 2 The CNN feature acquisition module in obtains the CNN features of each video frame. In this step, the CNN features of the first video frame and the CNN features of the second video frame are obtained respectively.

[0086] This step is done by Figure 2 The mapping module in the embodiment of the present application is implemented by forward mapping. Compared with backward mapping, forward mapping can more easily obtain the required optical flow at any time.

[0087] Step 304: Perform video synthesis based on the first mapped video frame, the second mapped video frame, the CNN features of the first mapped video frame, the CNN features of the second mapped video frame, and the feature channel information to obtain and output a target video frame.

[0088] In this step, Figure 2The CNN feature acquisition module in the image processing module obtains the CNN features of the first mapped video frame and the CNN features of the second mapped video frame respectively;

[0089] pass Figure 2 The feature channel information acquisition module in the image processing unit obtains the feature channel information; since video interpolation and video prediction are at different points in the pixel synthesis stage, simply training these two tasks at the same time may cause conflicts, resulting in poor final convergence. Therefore, the present application adds feature channel information of a special feature channel to the input of the pixel synthesis network, which is used to indicate the relationship between the target video frame and the input video frame: front, middle, and back. This allows the training of the video frame acquisition model to converge better and achieve better generalization effects.

[0090] pass Figure 2 The synthesis module in the target video frame is realized by Figure 2 The output module in outputs the synthesized target image frame.

[0091] There is no limitation on the specific synthesis network implemented by the synthesis module in the embodiments of the present application.

[0092] The following is a detailed description of the video frame acquisition process implemented in the embodiment of the present application.

[0093] See also Figure 4 , Figure 4 This is a schematic diagram of a video frame acquisition process in an embodiment of the present application. The specific steps are:

[0094] Step 401 : obtaining video frame prediction information, where the video frame prediction information includes: time information corresponding to an input frame and a target video frame to be obtained.

[0095] Step 402, based on the preset video frame acquisition model, acquires the target video frame corresponding to the video frame prediction information; wherein, the preset video frame acquisition model superimposes the feature channel information when acquiring the target video frame; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined according to the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

[0096] The preset video frame acquisition model will set the time information of the input video frame; since the input video frame includes two frames, namely the first video frame and the second video frame, according to the embodiment of the present application, the unified setting of the time information of the video frame can be: setting the time information of the first video frame to 0, and the time information of the second video frame to 1.

[0097] In the specific implementation, the sequence of input video frames needs to be marked; the first video frame and the second video frame are identified by marking, or it is also possible not to mark and directly default which video frame is used as the first video frame and which video frame is used as the second video frame according to the input sequence.

[0098] In one example, the specific process of determining the characteristic channel information according to the time information set for the input video frame and the time information corresponding to the target video frame to be acquired is as follows:

[0099] The determined characteristic channel information is:

[0100] 1<t, 0<t<1, or t<0;

[0101] Among them, when 1<t, it is determined to be predicting the next frame, when 0<t<1, it is determined to be predicting the middle frame, and when t<0, it is determined to be predicting the previous frame; 0 is the time information set for the first video frame, 1 is the time information set for the second video frame, and t is the time information corresponding to the target video frame to be obtained.

[0102] It can also be seen here that t is the time information corresponding to the target video frame that actually needs to be obtained, and the specific value of t is not limited. Therefore, the technical solution provided in the embodiment of the present application can obtain a video interpolation frame at any time, or a predicted frame at any time through a preset video frame acquisition model.

[0103] In the embodiment of the present application, feature channel information is superimposed in the established preset video frame acquisition model, wherein the feature channel information is used to characterize the temporal relationship between the target video frame and the input video frame, so as to obtain a video interpolation frame or a predicted frame at any time corresponding to the input video frame. This solution can achieve the acquisition of the target video frame at any time through a single model.

[0104] See also Figure 5 , Figure 5 This is another video frame acquisition process diagram in the embodiment of the present application. The specific steps are:

[0105] Step 501: Obtain video frame prediction information, wherein the video frame prediction information includes: a first video frame, a second video frame, and time information t corresponding to a target video frame to be obtained.

[0106] Step 502 : Determine, based on the time information set for the first video frame and the second video frame, that the time information of the first video frame is 0, and that the time information of the second video frame is 1.

[0107] Step 503, determine whether t is less than -1, or greater than 2, if yes, execute step 504; otherwise, execute step 505.

[0108] Step 504: Determine the target video frame using an iterative method. This process ends.

[0109] Although the preset video frame acquisition model in the embodiment of the present application can calculate the optical flow from the input video frame to the frame at any time, in order to ensure the final synthesis effect, the time information should be limited to between -1 and 2 as much as possible. For frames beyond this time range, an iterative method can be used for synthesis, that is, first predicting the video frame that is closer, and then predicting the farther video frame based on the closer video frame, until the target frame corresponding to the time information t is predicted. The reason for not directly calculating the optical flow from the input video frame to the target frame that is particularly far away (the time distance is greater than 2) is that between two adjacent very far video frames, due to occlusion or exceeding the video range, there are a large number of "disappearing" pixels between each other, making it almost impossible to accurately calculate the optical flow.

[0110] Use the iterative method to determine the target video frame, specifically:

[0111] If t is less than -1, -1 is input as the time information of the target video frame into the preset video frame acquisition model to obtain an intermediate target video frame with time information of -1; then the intermediate target video frame with time information of -1 is used as the input video frame, and -2 is input as the time information of the target video frame into the preset video frame acquisition model to obtain an intermediate video frame with time information of -2, until an intermediate target video frame with time information greater than t and closest to t is obtained, and the intermediate target video frame with time information greater than t and closest to t is used as the input video frame, and t is input as the time information of the target video frame into the video frame acquisition model to obtain a target video frame with time information t;

[0112] If t is greater than 2, 2 is input as the time information of the target video frame into the preset video frame acquisition model to obtain an intermediate target video frame with time information of 2; then the intermediate target video frame with time information of 2 is used as the input video frame, and 3 is input as the time information of the target video frame into the preset video frame acquisition model to obtain an intermediate video frame with time information of 3, until an intermediate target video frame with time information less than t and closest to t is obtained, and the intermediate target video frame with time information less than t and closest to t is used as the input video frame, and t is input as the time information of the target video frame into the video frame acquisition model to obtain the target video frame with time information t.

[0113] Step 505: Acquire a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model.

[0114] Among them, the preset video frame acquisition model superimposes feature channel information when acquiring the target video frame: 1<t, 0<t<1, or t<0; the feature channel information is used to characterize the time relationship between the target video frame and the first video frame and the second video frame; the feature channel information is determined according to the time information set for the first video frame and the second video frame, and the time information corresponding to the target video frame to be acquired.

[0115] In the embodiment of the present application, the target video frame with time information t less than 2 and greater than -1 is directly obtained by using the established preset video frame acquisition model. If t is greater than 2, the intermediate target video frame with time information 2 is first obtained, and the target video frame with time information t is iteratively obtained in sequence; if t is less than -1, the intermediate target video with time information -1 is first obtained, and the target video frame with time information t is iteratively obtained in sequence. Therefore, the present application can achieve the acquisition of target video frames at any time through a single model.

[0116] Since the technical solution provided by this application supports video prediction at any time, we can improve the video frame rate by prediction. Video prediction only relies on past video frames and does not need to wait for the arrival of the next frame, which can greatly reduce the algorithm delay.

[0117] The technical solution provided by this application can also serve many downstream applications based on video interpolation and prediction algorithms, including video compression, video enhancement, perspective synthesis, etc. In particular, in the concept of the metaverse, the interaction of real-time high-definition video content becomes increasingly important. Under limited bandwidth conditions, transmitting a video stream with a lower frame rate and using video interpolation or prediction technology on the client to convert the low frame rate video into a high frame rate video are both better video frame acquisition solutions.

[0118] It can also be applied to some interesting and previously difficult-to-achieve scenarios, such as SuperSlowMotion generation containing previous and next frames, frame rate improvement based on video prediction, etc.

[0119] Since only one model needs to be trained in the embodiment of the present application, in scenarios where both interpolation and prediction are required, the use of the technical solution provided by the present application can greatly reduce deployment overhead.

[0120] It can also be used to alleviate the transmission pressure of video data in the Metaverse project, that is, to transmit low-frame-rate videos and restore high-frame-rate videos through interpolation or prediction technology.

[0121] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0122] Based on the same inventive concept, the present application also provides a video frame acquisition device. Figure 6 , Figure 6 This is a schematic diagram of the structure of a video frame acquisition device in an embodiment of the present application. The device includes:

[0123] The storage unit 601 is used to store a preset video frame acquisition model;

[0124] A first acquisition unit 602 is used to acquire video frame prediction information, wherein the video frame prediction information includes: time information corresponding to the input video frame and the target video frame to be acquired;

[0125] The second acquisition unit 603 is used to acquire the target video frame corresponding to the video frame prediction information based on the preset video frame acquisition model; wherein, the preset video frame acquisition model superimposes the feature channel information when acquiring the target video frame; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined based on the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

[0126] In another embodiment,

[0127] The input video frames include: a first video frame and a second video frame; the feature channel information is:

[0128] 1<t, 0<t<1, or t<0;

[0129] Among them, 0 is the time information set for the first video frame, 1 is the time information set for the second video frame, and t is the time information corresponding to the target video frame to be obtained.

[0130] In another embodiment, the input video frame includes: a first video frame and a second video frame;

[0131] The preset video frame acquisition model predicts the bidirectional optical flow between the first video frame and the second video frame; based on the bidirectional optical flow, a linear weighted method is used to obtain the first optical flow from the first video frame to the target video frame, and the second optical flow from the second video frame to the target video frame; forward mapping is performed based on the first optical flow, the second optical flow, the first video frame, the second video frame, and the CNN features of the first video frame and the CNN features of the second video frame to obtain the mapped first mapping video frame and the second mapping video frame; video synthesis is performed based on the first mapping video frame, the second mapping video frame, the CNN features of the first mapping video frame, the CNN features of the second mapping video frame and the feature channel information to obtain the target video frame and output it.

[0132] In another embodiment, the bidirectional optical flow includes a forward optical flow and a reverse optical flow; the forward optical flow is an optical flow from the first video frame to the second video frame; the reverse optical flow is an optical flow from the second video frame to the first video frame;

[0133] The preset video frame acquisition model uses a linear weighted method based on bidirectional optical flow to obtain a first optical flow from a first video frame to a target video frame, and a second optical flow from a second video frame to a target video frame, including: determining the first optical flow as the product of the forward optical flow and t; determining the second optical flow as the product of the reverse optical flow and 1-t.

[0134] In another embodiment, the apparatus further comprises:

[0135] The model training unit 604 is used to establish an initial video frame acquisition model; obtain training samples; the training samples include multiple groups of triplets, each triplet is three consecutive video frames; any one of the frames is used as a target video frame, and the other two frames are used as input video frames; use the training samples to train the initial video frame acquisition model to obtain a preset video frame acquisition model.

[0136] In another embodiment, the apparatus further comprises:

[0137] The determination unit 605 is used to determine whether t is less than -1 or greater than 2 when the first acquisition unit acquires the video frame prediction information. If so, the second acquisition unit 603 is triggered to use an iterative method to determine the target video frame; otherwise, the second acquisition unit 603 is triggered to acquire the target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model.

[0138] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the video frame acquisition method is implemented when the processor executes the program.

[0139] In another embodiment, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the video frame acquisition method is implemented.

[0140] Figure 7 Schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the electronic device may include: a processor (Processor) 710, a communication interface (Communications Interface) 720, a memory (Memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the following method:

[0141] Acquire video frame prediction information, wherein the video frame prediction information includes: time information corresponding to the input video frame and the target video frame to be acquired;

[0142] Acquire a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model;

[0143] Among them, the preset video frame acquisition model superimposes feature channel information when acquiring the target video frame; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined based on the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

[0144] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order of the standards in different figures. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims disclosed in this application may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in this application.

[0149] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.

Claims

1. A video frame acquisition method, It is characterized in that The method comprises: Acquire video frame prediction information, wherein the video frame prediction information includes: time information corresponding to the input video frame and the target video frame to be acquired; Acquire a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model; Among them, the preset video frame acquisition model superimposes feature channel information when acquiring the target video frame; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined based on the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

2. The method according to claim 1, It is characterized in that The input video frame includes: a first video frame and a second video frame; the feature channel information is determined according to time information set for the input video frame and time information corresponding to the target video frame to be acquired, including: The characteristic channel information is determined as: 1<t, 0<t<1, or t<0; Among them, 0 is the time information set for the first video frame, 1 is the time information set for the second video frame, and t is the time information corresponding to the target video frame to be obtained.

3. The method according to claim 1, It is characterized in that The input video frame includes: a first video frame and a second video frame; and obtaining a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model includes: The preset video frame acquisition model predicts the bidirectional optical flow between the first video frame and the second video frame based on the first video frame and the second video frame; Obtaining a first optical flow from the first video frame to the target video frame and a second optical flow from the second video frame to the target video frame by using a linear weighted method based on the bidirectional optical flow; Perform forward mapping based on the first optical flow, the second optical flow, the first video frame, the second video frame, and the CNN features of the first video frame and the CNN features of the second video frame to obtain a mapped first mapped video frame and a second mapped video frame; Video synthesis is performed based on the first mapped video frame, the second mapped video frame, the CNN features of the first mapped video frame, the CNN features of the second mapped video frame and the feature channel information to obtain and output the target video frame.

4. The method according to claim 3, It is characterized in that The bidirectional optical flow includes a forward optical flow and a reverse optical flow; the forward optical flow is the optical flow from the first video frame to the second video frame; the reverse optical flow is the optical flow from the second video frame to the first video frame; The step of obtaining a first optical flow from the first video frame to the target video frame and a second optical flow from the second video frame to the target video frame by using a linear weighted method based on the bidirectional optical flow includes: Determine the first optical flow as the product of the forward optical flow and t; The second optical flow is determined as the product of the reverse optical flow and 1-t.

5. The method according to any one of claims 1 to 4, It is characterized in that The acquisition of the preset video frame acquisition model includes: Establish an initial video frame acquisition model; Acquire training samples; the training samples include multiple groups of triplets, each of which is three consecutive video frames; any one of the frames is used as a target video frame, and the other two frames are used as input video frames; The initial video frame acquisition model is trained using the training sample to obtain a preset video frame acquisition model.

6. The method according to any one of claims 2 to 4, It is characterized in that After acquiring the video frame prediction information and before acquiring the target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model, the method further includes: Determine whether t is less than -1 or greater than 2. If so, use an iterative method to determine the target video frame; otherwise, execute the method of obtaining the target video frame corresponding to the video frame prediction information based on the preset video frame acquisition model.

7. A video frame acquisition device, It is characterized in that The device comprises: A storage unit, used for storing a preset video frame acquisition model; A first acquisition unit is used to acquire video frame prediction information, wherein the video frame prediction information includes: time information corresponding to the input video frame and the target video frame to be acquired; A second acquisition unit is used to acquire a target video frame corresponding to the video frame prediction information based on a preset video frame acquisition model; wherein, when acquiring the target video frame, the preset video frame acquisition model superimposes feature channel information; the feature channel information is used to characterize the time relationship between the target video frame and the input video frame; the feature channel information is determined based on the time information set for the input video frame and the time information corresponding to the target video frame to be acquired.

8. The device according to claim 7, It is characterized in that The input video frame includes: a first video frame and a second video frame; the feature channel information is: 1<t, 0<t<1, or t<0; Among them, 0 is the time information set for the first video frame, 1 is the time information set for the second video frame, and t is the time information corresponding to the target video frame to be obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images